Download Multiple Files from Amazon S3 Bucket in Spring Boot

Follow these steps to download multiple files from an Amazon S3 bucket and zip them for download using a REST API in Spring Boot:

1. Create a Spring Boot Project:

We assume you have a Spring Boot project set up and ready. If not, you can create one using Spring Initializr or your preferred approach.

2. Add Amazon SDK Dependencies:

Add the AWS Java SDK For Amazon S3 dependency to your Spring Boot project:

For Maven:

<dependency>
    <groupid>com.amazonaws</groupid>
    <artifactid>aws-java-sdk-s3</artifactid>
    <version>1.12.556</version>
</dependency>

For Gradle:

implementation group: 'com.amazonaws', name: 'aws-java-sdk-s3', version: '1.12.556'

3. Add Configurations:

To access S3 bucket from a Spring Boot project, you'll need to configure your project with AWS credentials. You can do this by providing your AWS access key and secret key, which can be set in your application.properties or application.yml file or loaded from environment variables. For example, in your application.properties file:

server.port=8080

aws.access-key = your-access-key
aws.access-secret-key = your-access-secret-key
aws.region = us-east-1

4. Create an AmazonS3Client:

Create a bean for the AmazonS3 client in your Spring Boot application configuration class. You can use the @Configuration annotation to create a configuration class:

package com.example.app.config;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;

@Configuration
public class AmazonS3Config {

  private String awsAccessKey;
  private String awsAccessSecretKey;
  private String awsRegion;

  public AmazonS3Config(@Value(value = "${aws.access-key}") String awsAccessKey,
      @Value(value = "${aws.access-secret-key}") String awsAccessSecretKey,
      @Value(value = "${aws.region}") String awsRegion) {
    this.awsAccessKey = awsAccessKey;
    this.awsAccessSecretKey = awsAccessSecretKey;
    this.awsRegion = awsRegion;
  }

  public AWSStaticCredentialsProvider getAwsCredentialsProvider() {
    BasicAWSCredentials awsCred =
        new BasicAWSCredentials(this.awsAccessKey, this.awsAccessSecretKey);
    return new AWSStaticCredentialsProvider(awsCred);
  }

  @Bean
  public AmazonS3 getAmazonS3Client() {
    return AmazonS3ClientBuilder.standard().withRegion(this.awsRegion)
        .withCredentials(getAwsCredentialsProvider()).build();
  }

}

5. Configure Asynchronous Processing:

It is recommended that you explicitly configure the TaskExecutor if the file to be downloaded is large and will take more than a minute to download. Here is the complete code for configuring the TaskExecutor with a request timeout of 3600000 milliseconds (60 minutes):

package com.example.app.config;

import java.util.concurrent.Callable;
import org.springframework.aop.interceptor.AsyncUncaughtExceptionHandler;
import org.springframework.aop.interceptor.SimpleAsyncUncaughtExceptionHandler;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.task.AsyncTaskExecutor;
import org.springframework.scheduling.annotation.AsyncConfigurer;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import org.springframework.web.context.request.NativeWebRequest;
import org.springframework.web.context.request.async.CallableProcessingInterceptor;
import org.springframework.web.context.request.async.TimeoutCallableProcessingInterceptor;
import org.springframework.web.servlet.config.annotation.AsyncSupportConfigurer;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;

@Configuration
@EnableAsync
@EnableScheduling
public class AsyncConfig implements AsyncConfigurer {

  @Override
  @Bean(name = "taskExecutor")
  public AsyncTaskExecutor getAsyncExecutor() {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    executor.setCorePoolSize(10);
    executor.setMaxPoolSize(15);
    executor.setQueueCapacity(50);
    return executor;
  }

  @Override
  public AsyncUncaughtExceptionHandler getAsyncUncaughtExceptionHandler() {
    return new SimpleAsyncUncaughtExceptionHandler();
  }

  @Bean
  public WebMvcConfigurer webMvcConfigurerConfigurer(AsyncTaskExecutor taskExecutor,
      CallableProcessingInterceptor callableProcessingInterceptor) {
    return new WebMvcConfigurer() {
      @Override
      public void configureAsyncSupport(AsyncSupportConfigurer configurer) {
        configurer.setDefaultTimeout(3600000).setTaskExecutor(taskExecutor);
        configurer.registerCallableInterceptors(callableProcessingInterceptor);
        WebMvcConfigurer.super.configureAsyncSupport(configurer);
      }
    };
  }

  @Bean
  public CallableProcessingInterceptor callableProcessingInterceptor() {
    return new TimeoutCallableProcessingInterceptor() {
      @Override
      public <T> Object handleTimeout(NativeWebRequest request, Callable<T> task) throws Exception {
        return super.handleTimeout(request, task);
      }
    };
  }

}

This configuration class is used to enable and configure asynchronous processing in a Spring Boot application. It defines an AsyncTaskExecutor, sets up default timeouts, and provides exception handling for asynchronous tasks. Additionally, it configures Spring MVC to support asynchronous requests and handles timeouts for callable tasks.

6. Create a Service:

Create a service interface named S3MultipleFilesDownloadService with a method for downloading multiple files from an S3 bucket:

package com.example.app.service;

import java.util.List;
import org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody;
import jakarta.servlet.http.HttpServletResponse;

public interface S3MultipleFilesDownloadService {

  StreamingResponseBody downloadMultipleFilesFromS3(HttpServletResponse response,
      List<String> fileIds);

}

7. Create a Service Implementation:

Create an implementation class named S3MultipleFilesDownloadServiceImpl that implements the S3MultipleFilesDownloadService interface and handles the business logic:

package com.example.app.service.impl;

import java.io.IOException;
import java.io.InputStream;
import java.util.Arrays;
import java.util.List;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.S3Object;
import com.example.app.service.S3MultipleFilesDownloadService;
import jakarta.servlet.http.HttpServletResponse;

@Service
public class S3MultipleFilesDownloadServiceImpl implements S3MultipleFilesDownloadService {

  private String bucketName = "my-test-bucket";

  private String s3FolderName = "/myfolder/images/";

  @Autowired
  private AmazonS3 s3Client;

  @Override
  public StreamingResponseBody downloadMultipleFilesFromS3(HttpServletResponse response,
      List<String> fileIds) {
    // get file names by fileIds from your database
    List<String> filenameList = Arrays.asList("file1.pdf", "file2.png", "file3.pdf", "file4.mp4");

    int BUFFER_SIZE = 1024;

    StreamingResponseBody streamResponseBody = out -> {

      final ZipOutputStream zipOutputStream = new ZipOutputStream(response.getOutputStream());
      ZipEntry zipEntry = null;
      InputStream inputStream = null;
      S3Object s3Object = null;

      try {
        for (String filename : filenameList) {
          // file location in S3
          String fileLocationKey = s3FolderName + filename;

          /* Retrieve file as object from S3 */
          s3Object = s3Client.getObject(new GetObjectRequest(bucketName, fileLocationKey));

          zipEntry = new ZipEntry(filename);

          inputStream = s3Object.getObjectContent();

          zipOutputStream.putNextEntry(zipEntry);
          byte[] bytes = new byte[BUFFER_SIZE];
          int length;
          while ((length = inputStream.read(bytes)) >= 0) {
            zipOutputStream.write(bytes, 0, length);
          }

        }
        // set zip size in response
        response.setContentLength((int) (zipEntry != null ? zipEntry.getSize() : 0));
      } catch (IOException e) {
        e.printStackTrace();
      } finally {
        if (inputStream != null) {
          inputStream.close();
        }
        if (zipOutputStream != null) {
          zipOutputStream.close();
        }
      }

    };

    response.setContentType("application/zip");
    response.setHeader("Content-Disposition", "attachment; filename=example.zip");
    response.addHeader("Pragma", "no-cache");
    response.addHeader("Expires", "0");

    return streamResponseBody;
  }

}

8. Create a Web Controller:

Create a controller with a REST API endpoint that downloads multiple files from S3 and allows us to download as a zipped file:

package com.example.app.controller;

import java.util.List;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBody;
import com.example.app.service.S3MultipleFilesDownloadService;
import jakarta.servlet.http.HttpServletResponse;

@RestController
@RequestMapping(path = "/files")
public class S3MultipleFilesDownloadController {

  @Autowired
  private S3MultipleFilesDownloadService s3MultipleFilesDownloadService;

  @GetMapping(value = "/{fileIds}/download")
  public ResponseEntity<StreamingResponseBody> downloadMultipleFilesAsZip(
      HttpServletResponse response,
      @PathVariable(name = "fileIds", required = true) List<String> fileIds) {

    return ResponseEntity
        .ok(s3MultipleFilesDownloadService.downloadMultipleFilesFromS3(response, fileIds));
  }

}