Spring Batch With MongoDB Example

In this example, we will build a simple Batch application using Spring Batch with MongoDB in Java Spring Boot.

Spring Batch is an open source, lightweight framework which is designed for use in developing robust batch applications. Batch applications are usually required by systems that need to process large volume of data on daily basis.

Spring Batch is not designed for use as a scheduling framework. However, it can be used in combination with a scheduling framework such as Quartz, Control-M, etc.

Spring Boot provides spring-boot-starter-batch dependency.

This sample batch application that we will build in this example will do the following:

  1. Read data from a CSV file.
  2. Transform the data.
  3. Write the final results to a MongoDB database.

Complete the following steps to build our sample batch application:

Create a Spring Boot Application

  1. Go to Spring Initializr at https://start.spring.io and create a Spring Boot application with details as follows:
    • Project: Choose Gradle Project or Maven Project.
    • Language: Java
    • Spring Boot: Latest stable version of Spring Boot is selected by default. So leave it as is.
    • Project Metadata: Provide group name in the Group field. The group name is the id of the project. In Artifact field, provide the name of your project. In the package field, provide package name for your project. Next, select your preferred version of Java that is installed on your computer and is available on your local environment.
    • Dependencies: Add dependencies for Spring Web, Spring Boot DevTools, Spring Batch, and Spring Data MongoDB.

    Refer to the image below for example:

    Spring Batch with MongoDB example

  2. Click the GENERATE button and save/download the project zip bundle.
  3. Extract the project to your preferred working directory.
  4. Import the project in your preferred Java development IDE such as Eclipse or IntelliJ IDEA.

The following listing shows Spring Batch and Spring Data MongoDB dependencies for Gradle and Maven project:

With Gradle

build.gradle

implementation 'org.springframework.boot:spring-boot-starter-batch'
implementation 'org.springframework.boot:spring-boot-starter-data-mongodb'

With Maven

pom.xml

Add Application Configurations

Open the application.properties file and copy the configurations listed below. Do not forget to update the configuration values to make them relevant to your project:

src/main/resources/application.properties 

# Server port
server.port = 8080

# Mongo Configuration
spring.data.mongodb.host = localhost
spring.data.mongodb.port = 27017
spring.data.mongodb.database = spring_batch_test
spring.data.mongodb.username = test
spring.data.mongodb.password = 12345678

#disabled job run at startup
spring.batch.job.enabled=false

Create a CSV file

The application will read data from a CSV file, so let's create a user-sample-data.csv file within the src/main/resources folder of the project and add the following values to it:

src/main/resources/user-sample-data.csv 

"[email protected]", "John", "William", "90001234555"
"[email protected]", "Jay", "Watson", "90001234556"
"[email protected]", "Ryan", "Watson", "90001234557"
"[email protected]", "Jake", "Watson", "90001234558"

Create a Model class

Create a UserDetail.java Java class to represent the data of the above CSV file. The following code shows how to do so:

src/main/java/com/example/model/UserDetail.java 

package com.example.model;

public class UserDetail {
    private String email;
    private String firstName;
    private String lastName;
    private String mobileNumber;

    public String getEmail() {
        return email;
    }

    public void setEmail(String email) {
        this.email = email;
    }

    public String getFirstName() {
        return firstName;
    }

    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }

    public String getLastName() {
        return lastName;
    }

    public void setLastName(String lastName) {
        this.lastName = lastName;
    }

    public String getMobileNumber() {
        return mobileNumber;
    }

    public void setMobileNumber(String mobileNumber) {
        this.mobileNumber = mobileNumber;
    }

}

Create a Domain class

After reading the CSV file, the application will save the file data in a MongoDB database. To perform the insert operation, let's create a User.java domain class and annotate it with @Document(collection = "table_name"). The @Document annotation identifies a class as being a domain object that we want to persist to the database. The attributes of the domain class are mapped to the columns of the database table. The following code shows how to do so:

src/main/java/com/example/domain/User.java 

package com.example.domain;

import java.time.LocalDateTime;
import org.springframework.data.annotation.CreatedDate;
import org.springframework.data.annotation.Id;
import org.springframework.data.annotation.LastModifiedDate;
import org.springframework.data.mongodb.core.mapping.Document;

@Document(collection = "user")
public class User {
    @Id
    private String id;
    private String email;
    private String firstName;
    private String lastName;
    private String mobileNumber;
    @CreatedDate
    private LocalDateTime createdDate;
    @LastModifiedDate
    private LocalDateTime lastModifieDate;

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getEmail() {
        return email;
    }

    public void setEmail(String email) {
        this.email = email;
    }

    public String getFirstName() {
        return firstName;
    }

    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }

    public String getLastName() {
        return lastName;
    }

    public void setLastName(String lastName) {
        this.lastName = lastName;
    }

    public String getMobileNumber() {
        return mobileNumber;
    }

    public void setMobileNumber(String mobileNumber) {
        this.mobileNumber = mobileNumber;
    }

    public LocalDateTime getCreatedDate() {
        return createdDate;
    }

    public void setCreatedDate(LocalDateTime createdDate) {
        this.createdDate = createdDate;
    }

    public LocalDateTime getLastModifieDate() {
        return lastModifieDate;
    }

    public void setLastModifieDate(LocalDateTime lastModifieDate) {
        this.lastModifieDate = lastModifieDate;
    }

}

Create a Processor class

Create a UserItemProcessor.java Java class and implement the ItemProcessor<I, O> interface of the Spring Batch framework.

The ItemProcessor<I, O> interface takes two arguments, Input type and Output type. Both doesn't need to be of the same type. You may provide input of one type and return output of some other type after it has been read.

The ItemProcessor<I, O> has a public method that takes an argument of object with data that is read. Inside this method is where the read data must be transformed. The following code shows how to do so:

src/main/java/com/example/processor/UserItemProcessor.java 

package com.example.processor;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.item.ItemProcessor;
import com.example.domain.User;
import com.example.model.UserDetail;


public class UserItemProcessor implements ItemProcessor<UserDetail, User> {
    private static final Logger log = LoggerFactory.getLogger(UserItemProcessor.class);

    @Override
    public User process(UserDetail item) throws Exception {

        log.info("processing user data.....{}", item);

        User transformedUser = new User();
        transformedUser.setEmail(item.getEmail());
        transformedUser.setFirstName(item.getFirstName());
        transformedUser.setLastName(item.getLastName());
        transformedUser.setMobileNumber(item.getMobileNumber());
        return transformedUser;
    }

}

Create a Job Notification Listener class

Create a JobCompletionNotificationListener.java Java class and extend the class with the JobExecutionListenerSupport class from the Spring Batch framework. The JobExecutionListenerSupport class has callback methods which can be called before starting or after completion of a job.

Let's override the afterJob method of JobExecutionListenerSupport class which will help the application to notify when the job completes.

src/main/java/com/example/listener/JobCompletionNotificationListener.java 

package com.example.listener;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.listener.JobExecutionListenerSupport;
import org.springframework.stereotype.Component;
@ Component
public class JobCompletionNotificationListener extends JobExecutionListenerSupport {
  
    private static final Logger log =
            LoggerFactory.getLogger(JobCompletionNotificationListener.class);

    @Override
    public void afterJob(JobExecution jobExecution) {
        if (jobExecution.getStatus() == BatchStatus.COMPLETED) {
            log.info("!!! JOB FINISHED! Time to verify the results");

        }
    }
}

Create a BatchConfiguration class

Create a BatchConfiguration.java batch configuration Java class and annotate it with @Configuration and @EnableBatchProcessing annotations. The @EnableBatchProcessing enables Spring Batch features and provide a base configuration for setting up batch jobs in an @Configuration class. The following code shows how to do so:

src/main/java/com/example/config/BatchConfiguration.java 

package com.example.config;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.data.MongoItemWriter;
import org.springframework.batch.item.data.builder.MongoItemWriterBuilder;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.builder.FlatFileItemReaderBuilder;
import org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.data.mongodb.core.MongoTemplate;
import com.example.domain.User;
import com.example.listener.JobCompletionNotificationListener;
import com.example.model.UserDetail;
import com.example.processor.UserItemProcessor;

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {
    @Autowired
    public JobBuilderFactory jobBuilderFactory;
    @Autowired
    public StepBuilderFactory stepBuilderFactory;


    @Bean
    public FlatFileItemReader<UserDetail> reader() {
        return new FlatFileItemReaderBuilder<UserDetail>().name("userItemReader")
                .resource(new ClassPathResource("user-sample-data.csv")).delimited()
                .names(new String[] {"email", "firstName", "lastName", "mobileNumber"})
                .fieldSetMapper(new BeanWrapperFieldSetMapper<UserDetail>() {
                    {
                        setTargetType(UserDetail.class);
                    }
                }).build();
    }

    @Bean
    public MongoItemWriter<User> writer(MongoTemplate mongoTemplate) {
        return new MongoItemWriterBuilder<User>().template(mongoTemplate).collection("user")
                .build();
    }


    @Bean
    public UserItemProcessor processor() {
        return new UserItemProcessor();
    }


    @Bean
    public Step step1(FlatFileItemReader<UserDetail> itemReader, MongoItemWriter<User> itemWriter)
            throws Exception {

        return this.stepBuilderFactory.get("step1").<UserDetail, User>chunk(5).reader(itemReader)
                .processor(processor()).writer(itemWriter).build();
    }

    @Bean
    public Job updateUserJob(JobCompletionNotificationListener listener, Step step1)
            throws Exception {

        return this.jobBuilderFactory.get("updateUserJob").incrementer(new RunIdIncrementer())
                .listener(listener).start(step1).build();
    }

}

Update Main Application class

Annotate the main Application Class with @EnableMongoAuditing and exclude DataSourceAutoConfiguration.class as shown below:

src/main/java/com/example/BatchMongodbSampleApplication.java 

package com.example.batch.mongodb;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
import org.springframework.data.mongodb.config.EnableMongoAuditing;

@EnableMongoAuditing
@SpringBootApplication(exclude = {DataSourceAutoConfiguration.class})
public class BatchMongodbSampleApplication {

    public static void main(String[] args) {
        SpringApplication.run(BatchMongodbSampleApplication.class, args);
    }

}

Run the Application

Run your batch application. The application should read the CSV file and save the data to the MongoDB database.

Summary

Congratulations! you have learned how to build a Spring Batch application with MongoDB and Spring Boot.



About us
Privacy Policy
Cookies Policy
Terms of use
Helping
Contact us
Facebook Linkedin Youtube Instagram Twitter