Spring Batch With MongoDB Example

In this example, we will build a simple Batch application using Spring Batch with MongoDB in Java Spring Boot.

Spring Batch is an open source, lightweight framework which is designed for use in developing robust batch applications. Batch applications are usually required by systems that need to process large volume of data on daily basis.

Spring Batch is not designed for use as a scheduling framework. However, it can be used in combination with a scheduling framework such as Quartz, Control-M, etc.

This sample batch application reads data from a CSV file, transforms the data, and writes this data to a MongoDB table called user:

1. Visit Spring Initializr website at https://start.spring.io.
2. Create a Spring Boot application with details as follows:
• Project: Choose the project type (Maven or Gradle).
• Language: Set the language to Java.
• Spring Boot: Specify the Spring Boot version. The default selection is the latest stable version of Spring Boot, so you can leave it unchanged.
• Project Metadata: Enter a Group and Artifact name for your project. The group name is the id of the project. Artifact is the name of your project. Add any necessary project metadata (description, package name, etc.)
• Choose between packaging as a JAR (Java Archive) or a WAR (Web Application Archive) depends on how you plan to deploy your Spring Boot application. Choose JAR packaging if you want a standalone executable JAR file and WAR packaging if you intend to deploy your application to a Java EE application server or servlet container. When you package your Spring Boot application as a JAR using JAR packaging, it includes an embedded web server, such as Tomcat, by default. This means that you don't need to separately deploy your application to an external Tomcat server. Instead, you can run the JAR file directly, and the embedded Tomcat server will start and serve your application.
• Select the Java version based on the compatibility requirements of your project. Consider the specific needs of your project, any compatibility requirements, and the Java version supported by your target deployment environment when making these choices.
  
3. Add project dependencies:
   
• Click on the "Add Dependencies" button.
• Choose the following dependencies: Spring Batch, and Spring Data MongoDB.

Here's an example:




4. Generate the project:
• Click on the "Generate" button.
• Spring Initializr will generate a zip file containing your Spring Boot project.
  
5. Download and extract the generated project:
• Download the zip file generated by Spring Initializr.
• Extract the contents of the zip file to a directory on your local machine.
  
6. Import the project into your IDE:
• Open your preferred IDE (IntelliJ IDEA, Eclipse, or Spring Tool Suite).
• Import the extracted project as a Maven or Gradle project, depending on the build system you chose in Spring Initializr.
  
7. Add Configurations:

Open the src/main/resources/application.properties file in your Eclipse editor and add the following configuration lines to the file:

# Server port
server.port = 8080

# Mongo Configuration
spring.data.mongodb.host = localhost
spring.data.mongodb.port = 27017
spring.data.mongodb.database = database_name
spring.data.mongodb.username = your-user-name
spring.data.mongodb.password = your-password

#disabled job run at startup
spring.batch.job.enabled=true

8. Create a Model Class:

Create a class named "UserDetail" to represent the data of the CSV file:

package com.example.model;

public class UserDetail {
    private String email;
    private String firstName;
    private String lastName;
    private String mobileNumber;

    public String getEmail() {
        return email;
    }

    public void setEmail(String email) {
        this.email = email;
    }

    public String getFirstName() {
        return firstName;
    }

    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }

    public String getLastName() {
        return lastName;
    }

    public void setLastName(String lastName) {
        this.lastName = lastName;
    }

    public String getMobileNumber() {
        return mobileNumber;
    }

    public void setMobileNumber(String mobileNumber) {
        this.mobileNumber = mobileNumber;
    }

}



9. Create Entity:

Create a User class that represents the user entity. This is the table to which data will be written to:

package com.example.entity;

import java.time.LocalDateTime;
import org.springframework.data.annotation.CreatedDate;
import org.springframework.data.annotation.Id;
import org.springframework.data.annotation.LastModifiedDate;
import org.springframework.data.mongodb.core.mapping.Document;

@Document(collection = "user")
public class User {
    @Id
    private String id;
    private String email;
    private String firstName;
    private String lastName;
    private String mobileNumber;
    @CreatedDate
    private LocalDateTime createdDate;
    @LastModifiedDate
    private LocalDateTime lastModifieDate;

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getEmail() {
        return email;
    }

    public void setEmail(String email) {
        this.email = email;
    }

    public String getFirstName() {
        return firstName;
    }

    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }

    public String getLastName() {
        return lastName;
    }

    public void setLastName(String lastName) {
        this.lastName = lastName;
    }

    public String getMobileNumber() {
        return mobileNumber;
    }

    public void setMobileNumber(String mobileNumber) {
        this.mobileNumber = mobileNumber;
    }

    public LocalDateTime getCreatedDate() {
        return createdDate;
    }

    public void setCreatedDate(LocalDateTime createdDate) {
        this.createdDate = createdDate;
    }

    public LocalDateTime getLastModifieDate() {
        return lastModifieDate;
    }

    public void setLastModifieDate(LocalDateTime lastModifieDate) {
        this.lastModifieDate = lastModifieDate;
    }

}



10. Create a Processor:

A processor class is used to transform data. A processor class should implement the ItemProcessor interface of the Spring Batch framework. The ItemProcessor interface takes two arguments, I (Input) type and O (Output) type. Both doesn't need to be of the same type. You may provide input of one type and return output of some other type after it has been read. The ItemProcessor has a public method that takes an argument of object with data that is read. This method is where the read data must be transformed.

Create a class processor called "UserItemProcessor":

package com.example.processor;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.item.ItemProcessor;
import com.example.entity.User;
import com.example.model.UserDetail;


public class UserItemProcessor implements ItemProcessor<UserDetail, User> {
    private static final Logger log = LoggerFactory.getLogger(UserItemProcessor.class);

    @Override
    public User process(UserDetail item) throws Exception {

        log.info("processing user data.....{}", item);

        User transformedUser = new User();
        transformedUser.setEmail(item.getEmail());
        transformedUser.setFirstName(item.getFirstName());
        transformedUser.setLastName(item.getLastName());
        transformedUser.setMobileNumber(item.getMobileNumber());
        return transformedUser;
    }

}



11. Create a Job Notification Listener:

A job notification listener is created by creating a class that extends JobExecutionListenerSupport class from the Spring Batch framework. It contains callback methods that can be invoked before the start or after the completion of a job. To print a notification after the job has completed, override the "afterJob" method of the "JobExecutionListenerSupport" class.

Create a class named "JobCompletionNotificationListener":

package com.example.listener;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.listener.JobExecutionListenerSupport;
import org.springframework.stereotype.Component;
@ Component
public class JobCompletionNotificationListener extends JobExecutionListenerSupport {

    private static final Logger log =
            LoggerFactory.getLogger(JobCompletionNotificationListener.class);

    @Override
    public void afterJob(JobExecution jobExecution) {
        if (jobExecution.getStatus() == BatchStatus.COMPLETED) {
            log.info("!!! JOB FINISHED! Time to verify the results");

        }
    }
}



12. Create Batch Configurations:

Create a BatchConfiguration.java batch configuration Java class and annotate it with @Configuration and @EnableBatchProcessing annotations. The @EnableBatchProcessing enables Spring Batch features and provide a base configuration for setting up batch jobs in an @Configuration class:

package com.example.config;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.data.MongoItemWriter;
import org.springframework.batch.item.data.builder.MongoItemWriterBuilder;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.builder.FlatFileItemReaderBuilder;
import org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.data.mongodb.core.MongoTemplate;
import com.example.entity.User;
import com.example.listener.JobCompletionNotificationListener;
import com.example.model.UserDetail;
import com.example.processor.UserItemProcessor;

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {
    @Autowired
    public JobBuilderFactory jobBuilderFactory;
    @Autowired
    public StepBuilderFactory stepBuilderFactory;


    @Bean
    public FlatFileItemReader<UserDetail> reader() {
        return new FlatFileItemReaderBuilder<UserDetail>().name("userItemReader")
                .resource(new ClassPathResource("user-sample-data.csv")).delimited()
                .names(new String[] {"email", "firstName", "lastName", "mobileNumber"})
                .fieldSetMapper(new BeanWrapperFieldSetMapper<UserDetail>() {
                    {
                        setTargetType(UserDetail.class);
                    }
                }).build();
    }

    @Bean
    public MongoItemWriter<User> writer(MongoTemplate mongoTemplate) {
        return new MongoItemWriterBuilder<User>().template(mongoTemplate).collection("user")
                .build();
    }


    @Bean
    public UserItemProcessor processor() {
        return new UserItemProcessor();
    }


    @Bean
    public Step step1(FlatFileItemReader<UserDetail> itemReader, MongoItemWriter<User> itemWriter)
            throws Exception {

        return this.stepBuilderFactory.get("step1").<UserDetail, User>chunk(5).reader(itemReader)
                .processor(processor()).writer(itemWriter).build();
    }

    @Bean
    public Job updateUserJob(JobCompletionNotificationListener listener, Step step1)
            throws Exception {

        return this.jobBuilderFactory.get("updateUserJob").incrementer(new RunIdIncrementer())
                .listener(listener).start(step1).build();
    }

}



13. Enable Jpa Auditing:

Annotate the main class with @EnableMongoAuditing annotation. It is used to enable auditing in JPA entities. Auditing allows automatic population of specific fields such as createdDate and lastModifieDate in JPA entities based on the current date and time. It is commonly used to keep track of when an entity was created or last modified. The main class should look like this:

package com.example.batch.mongodb;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
import org.springframework.data.mongodb.config.EnableMongoAuditing;

@EnableMongoAuditing
@SpringBootApplication(exclude = {DataSourceAutoConfiguration.class})
public class BatchMongodbSampleApplication {

    public static void main(String[] args) {
        SpringApplication.run(BatchMongodbSampleApplication.class, args);
    }
}



14. Run and Test your Application:

Use your IDE's build tools (Maven or Gradle) to build your project and resolve dependencies. Once the build is successful, run the main class of your application. You should see logs indicating that the application has started.

The application should read the CSV file and save the data to the MongoDB database.