Last updated: Sep 13, 2023

Spring Batch With MySQL Example

In this example, we will build a simple Batch application using Spring Batch with MySQL in Java Spring Boot.

Spring Batch is an open source, lightweight framework which is designed for use in developing robust batch applications. Batch applications are usually required by systems that need to process large volume of data on daily basis.

Spring Batch is not designed for use as a scheduling framework. However, it can be used in combination with a scheduling framework such as Quartz, Control-M, etc.

This sample batch application reads data from a MySQL database table called user, transforms the data, and writes this data to another table called profile:

Visit Spring Initializr website at https://start.spring.io.
Create a Spring Boot application with details as follows:

Project: Choose the project type (Maven or Gradle).
Language: Set the language to Java.
Spring Boot: Specify the Spring Boot version. The default selection is the latest stable version of Spring Boot, so you can leave it unchanged.
Project Metadata: Enter a Group and Artifact name for your project. The group name is the id of the project. Artifact is the name of your project. Add any necessary project metadata (description, package name, etc.)
Choose between packaging as a JAR (Java Archive) or a WAR (Web Application Archive) depends on how you plan to deploy your Spring Boot application. Choose JAR packaging if you want a standalone executable JAR file and WAR packaging if you intend to deploy your application to a Java EE application server or servlet container. When you package your Spring Boot application as a JAR using JAR packaging, it includes an embedded web server, such as Tomcat, by default. This means that you don't need to separately deploy your application to an external Tomcat server. Instead, you can run the JAR file directly, and the embedded Tomcat server will start and serve your application.
Select the Java version based on the compatibility requirements of your project. Consider the specific needs of your project, any compatibility requirements, and the Java version supported by your target deployment environment when making these choices.

Add project dependencies:

Click on the "Add Dependencies" button.
Choose the following dependencies: Spring Web, Spring Batch, MySQL Driver, Spring Data JPA, Lombok, and Spring Boot DevTools.

Here's an example:

Generate the project:

Click on the "Generate" button.
Spring Initializr will generate a zip file containing your Spring Boot project.

Download and extract the generated project:

Download the zip file generated by Spring Initializr.
Extract the contents of the zip file to a directory on your local machine.

Import the project into your IDE:

Open your preferred IDE (IntelliJ IDEA, Eclipse, or Spring Tool Suite).
Import the extracted project as a Maven or Gradle project, depending on the build system you chose in Spring Initializr.

Add Configurations:

Open the src/main/resources/application.properties file in your Eclipse editor and add the following configuration lines to the file:

# Server port
server.port = 8080

#mysql database connection
spring.datasource.url = jdbc:mysql://localhost:3306/database_name
spring.datasource.username = your-username
spring.datasource.password = your-password
spring.datasource.timeBetweenEvictionRunsMillis = 60000
spring.datasource.maxIdle = 1

#hibernate
spring.jpa.properties.hibernate.dialect = org.hibernate.dialect.MySQLDialect
spring.jpa.generate-ddl=true
spring.jpa.hibernate.ddl-auto=update

#disabled job run at application startup
spring.batch.job.enabled=false

spring.batch.jdbc.initialize-schema = ALWAYS

Here's an explanation of the above configurations:

server.port=8080: This configuration line is used to specify the port number on which the server will listen for incoming requests. In this case, it sets the server port to 8080.

spring.datasource.url: This line specifies the URL for connecting to the MySQL database. It includes the host (localhost), port (3306), and the name of the database (database_name) you want to connect to.

spring.datasource.username: Your MySQL database username.

spring.datasource.password: Your MySQL database password.

spring.datasource.timeBetweenEvictionRunsMillis: This property defines the time (in milliseconds) between eviction runs for idle connections in the database connection pool. It helps manage and clean up idle connections.

spring.datasource.maxIdle: The value of this represents the maximum number of idle connections allowed in the database connection pool.

spring.jpa.properties.hibernate.dialect: Specifies the Hibernate dialect for MySQL databases. It tells Hibernate how to generate SQL statements specific to MySQL.

spring.jpa.generate-ddl: If set to true, Hibernate will generate and execute the database schema DDL (Data Definition Language) statements to create or update database tables based on your JPA entity classes.

spring.jpa.hibernate.ddl-auto: This property determines the behavior of Hibernate's schema generation. "update" means that Hibernate will update the schema according to the entity classes, but it won't drop and recreate tables.

spring.batch.job.enabled: This property controls whether Spring Batch jobs should run automatically at application startup. If set to false, Spring Batch jobs will not be triggered automatically when the application starts.

spring.batch.jdbc.initialize-schema: This property determines when Spring Batch should initialize its database schema. Setting it to "ALWAYS" means that Spring Batch will always attempt to create its schema on startup.

Create Entities:

Create a User class that represents the user entity. This is the table from which data will be read:

package com.example.entity;

import java.util.Date;
import org.springframework.data.annotation.CreatedDate;
import org.springframework.data.annotation.LastModifiedDate;
import org.springframework.data.jpa.domain.support.AuditingEntityListener;
import jakarta.persistence.Entity;
import jakarta.persistence.EntityListeners;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

@Builder
@Data
@NoArgsConstructor
@AllArgsConstructor
@EntityListeners(AuditingEntityListener.class)
@Entity
@Table(name = "user")
public class User {
  
  @Id
  @GeneratedValue(strategy = GenerationType.IDENTITY)
  private Integer id;
  private String email;
  private boolean emailVerified;
  private String firstName;
  private String lastName;
  private String status;
  @CreatedDate
  private Date createdDate;
  @LastModifiedDate
  private Date lastModifiedDate;
  
}

Create a Profile class that represents the profile entity. This is the table to which data will be updated:

package com.example.entity;

import java.util.Date;
import org.springframework.data.annotation.CreatedDate;
import org.springframework.data.annotation.LastModifiedDate;
import org.springframework.data.jpa.domain.support.AuditingEntityListener;
import jakarta.persistence.Entity;
import jakarta.persistence.EntityListeners;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

@Builder
@Data
@NoArgsConstructor
@AllArgsConstructor
@EntityListeners(AuditingEntityListener.class)
@Entity
@Table(name = "profile")
public class Profile {

  @Id
  @GeneratedValue(strategy = GenerationType.IDENTITY)
  private Integer id;
  private String fullName;
  private Integer userId;
  @CreatedDate
  private Date createdDate;
  @LastModifiedDate
  private Date lastModifieDate;

}

Create Repositories:

Create a UserRepository interface that represents a repository responsible for handling data access operations related to users:

package com.example.repository;

import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
import org.springframework.data.repository.PagingAndSortingRepository;
import org.springframework.stereotype.Repository;
import com.example.entity.User;

@Repository
public interface UserRepository extends PagingAndSortingRepository<User, Long> {

  Page<User> findByStatusAndEmailVerified(String status, boolean emailVerified, Pageable pageable);

}

Create a ProfileRepository interface that represents a repository responsible for handling data access operations related to user profiles:

package com.example.repository;

import org.springframework.data.repository.CrudRepository;
import org.springframework.stereotype.Repository;
import com.example.entity.Profile;

@Repository
public interface ProfileRepository extends CrudRepository<Profile, Long> {

}

Create a Processor:

A processor class is used to transform data. A processor class should implement the ItemProcessor interface of the Spring Batch framework. The ItemProcessor interface takes two arguments, I (Input) type and O (Output) type. Both doesn't need to be of the same type. You may provide input of one type and return output of some other type after it has been read. The ItemProcessor has a public method that takes an argument of object with data that is read. This method is where the read data must be transformed.

Create a class processor called "ProfileItemProcessor":

package com.example.batch.processor;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.item.ItemProcessor;
import com.example.entity.Profile;
import com.example.entity.User;

public class ProfileItemProcessor implements ItemProcessor<User, Profile> {
  private static final Logger LOGGER = LoggerFactory.getLogger(ProfileItemProcessor.class);
  
  @Override
  public Profile process(User user) throws Exception {

    LOGGER.info("Processing user data.....{}", user);
            
    Profile profile = new Profile();
    profile.setUserId(user.getId());
    profile.setFullName(user.getFirstName() + " " + user.getLastName());
    return profile;
  }
}

Create a Job Notification Listener:

A job notification listener is created by creating a class that extends JobExecutionListenerSupport class from the Spring Batch framework. It contains callback methods that can be invoked before the start or after the completion of a job. To print a notification after the job has completed, override the "afterJob" method of the "JobExecutionListenerSupport" class.

Create a class named "JobCompletionNotificationListener":

package com.example.batch.listener;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobExecutionListener;
import org.springframework.stereotype.Component;

@Component
public class JobCompletionNotificationListener implements JobExecutionListener {

  private static final Logger LOGGER =
      LoggerFactory.getLogger(JobCompletionNotificationListener.class);

  @Override
  public void afterJob(JobExecution jobExecution) {
    if (jobExecution.getStatus() == BatchStatus.COMPLETED) {
      LOGGER.info("Job Finished! Time to verify the results.");

    } else {
      LOGGER.info("Job execution status : {}", jobExecution.getStatus());
    }
  }

}

Create Batch Configurations:

Create a class named "BatchConfig" and annotate it with @Configuration annotation:

package com.example.batch.config;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.job.builder.JobBuilder;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.core.repository.JobRepository;
import org.springframework.batch.core.step.builder.StepBuilder;
import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.data.RepositoryItemReader;
import org.springframework.batch.item.data.RepositoryItemWriter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Lazy;
import org.springframework.data.domain.Sort.Direction;
import org.springframework.orm.jpa.JpaTransactionManager;
import org.springframework.transaction.PlatformTransactionManager;
import com.example.batch.listener.JobCompletionNotificationListener;
import com.example.batch.processor.ProfileItemProcessor;
import com.example.entity.Profile;
import com.example.entity.User;
import com.example.repository.ProfileRepository;
import com.example.repository.UserRepository;

@Configuration
public class BatchConfig {

  @Autowired
  @Lazy
  private UserRepository userRepository;

  @Autowired
  @Lazy
  private ProfileRepository profileRepository;


  @Bean
  public RepositoryItemReader<User> reader() {
    RepositoryItemReader<User> reader = new RepositoryItemReader<>();
    reader.setRepository(userRepository);
    reader.setMethodName("findByStatusAndEmailVerified");

    List<Object> queryMethodArguments = new ArrayList<>();
    // for status
    queryMethodArguments.add("APPROVED");
    // for emailVerified
    queryMethodArguments.add(true);

    reader.setArguments(queryMethodArguments);
    reader.setPageSize(100);
    Map<String, Direction> sorts = new HashMap<>();
    sorts.put("id", Direction.ASC);
    reader.setSort(sorts);

    return reader;
  }

  @Bean
  public RepositoryItemWriter<Profile> writer() {
    RepositoryItemWriter<Profile> writer = new RepositoryItemWriter<>();
    writer.setRepository(profileRepository);
    writer.setMethodName("save");
    return writer;
  }


  @Bean
  public ProfileItemProcessor processor() {
    return new ProfileItemProcessor();
  }


  @Bean
  public Step step1(JobRepository jobRepository, ItemReader<User> itemReader,
      ItemWriter<Profile> itemWriter, PlatformTransactionManager transactionManager)
      throws Exception {

    int chunkSize = 5;
    return new StepBuilder("step1", jobRepository)
        .<User, Profile>chunk(chunkSize, transactionManager).reader(itemReader)
        .processor(processor()).writer(itemWriter).build();
  }

  @Bean
  public Job profileUpdateJob(JobCompletionNotificationListener listener,
      JobRepository jobRepository, Step step1) throws Exception {

    return new JobBuilder("profileUpdateJob", jobRepository).incrementer(new RunIdIncrementer())
        .listener(listener).start(step1).build();
  }

  @Bean
  public PlatformTransactionManager transactionManager() {
    return new JpaTransactionManager();
  }

}

Create a Web Controller:

Create a class named "BatchController" with a POST method to run the batch:

package com.example.controller;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.JobParametersBuilder;
import org.springframework.batch.core.JobParametersInvalidException;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.batch.core.repository.JobExecutionAlreadyRunningException;
import org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException;
import org.springframework.batch.core.repository.JobRestartException;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping(path = "/batch")
public class BatchController {
  @Autowired
  private JobLauncher jobLauncher;
  @Autowired
  private Job job;

  @PostMapping(path = "/start")
  public ResponseEntity<?> startBatch() {
    JobExecution jobExecution = null;
    try {
      JobParameters jobParameters = new JobParametersBuilder()
          .addLong("uniqueId", System.currentTimeMillis()).toJobParameters();

      jobExecution = jobLauncher.run(job, jobParameters);

    } catch (JobExecutionAlreadyRunningException | JobRestartException
        | JobInstanceAlreadyCompleteException | JobParametersInvalidException e) {
      e.printStackTrace();
    }
    return ResponseEntity.ok(jobExecution.getStatus());
  }

}

Enable Jpa Auditing:

Annotate the main class with @EnableJpaAuditing annotation. It is used to enable auditing in JPA entities. Auditing allows automatic population of specific fields such as createdDate and lastModifieDate in JPA entities based on the current date and time. It is commonly used to keep track of when an entity was created or last modified. The main class should look like this:

package com.example;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.data.jpa.repository.config.EnableJpaAuditing;

@EnableJpaAuditing
@SpringBootApplication
public class ExampleSpringBatchMysqlApplication {

  public static void main(String[] args) {
    SpringApplication.run(ExampleSpringBatchMysqlApplication.class, args);
  }

}

Run and Test your Application:

Use your IDE's build tools (Maven or Gradle) to build your project and resolve dependencies. Once the build is successful, run the main class of your application. The Spring Boot application will start and deploy on an embedded web server (Tomcat) automatically. You should see logs indicating that the application has started.

Use API testing tools (example: Postman) to test your application's endpoints: