java - Using Spring Batch to process a very large database table

Question

I have written a small proof of concept Spring Boot / Batch application that uses JDBC to page through a very large database table (potentially billions of rows) and converts the rows read to XML before writing them out to an S3 bucket. Currently my PoC has a single job that commences on start up and will keep running until the table has been completely consumed. It runs against test data, which is only a few hundred rows. For my single job I am using:

1. JdbcPagingItemReader
2. A custom ItemProcessor to convert rows to XML
3. A custom ItemWriter to write XML to S3

I have a number of problems which I don't know how to address using Spring Batch. Having read the manual, nothing is jumping out at me as a solution for how I might progress. For the sake of argument lets say the table contains 1 billion rows of Person information and looks like this:

pk	name	creation_date	payload_data
1	Jim	2020-10-10	Some
2	Bob	2020-10-10	Data
3	Jon	2020-10-11	In
4	Sue	2020-10-14	JSON

question from:https://stackoverflow.com/questions/65920262/using-spring-batch-to-process-a-very-large-database-table

Answer 1

Waitting for answers