spring batch - FlatFileItemWriter should write output file named same as input file

Question

I have a spring batch job that reads files matching naming pattern from a directory, does some processing, and writes back status of processing per line in input file. The writer must produce output files with same name as the input files. To the MultiResourceItemReader I pass the pattern: "files-*.txt" and expect the FlatFileItemWriter to use the name of the input file. How do I specify this constraint in context xml file ?

Reader bean

<bean id="multiResourceReader" class="org.springframework.batch.item.file.MultiResourceItemReader" scope="step">
  <property name="resources" value="file:#{jobParameters['cwd']}/#{jobParameters['inputFolder']}/file-*.txt" />
  <property name="delegate" ref="itemReader" />
</bean>
<bean id="itemReader" class="org.springframework.batch.item.file.FlatFileItemReader" scope="step">

        <property name="lineMapper">
          <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">

            <property name="lineTokenizer">
              <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                <property name="delimiter" value="," />
                <property name="names" value="paramA,paramB" />
              </bean>
            </property>
            <property name="fieldSetMapper" >
              <bean class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
                <property name="targetType" value="a.b.c.d.MyObject" />
              </bean>
            </property>
          </bean>
        </property>
    </bean>

I tried this: add the input file path as a string property to the ExecutionContext's map in my implementation of FlatFileItemReader. In my FlatFileItemWriter implementation - override setResource and actually create a Resource object out of value from ExecutionContext. Is this alright ?

Answer 1

I think this solution can be produce wrong output because MultiResourceItemReader reads from first file a bunch of items and if this set doesn't match your completion-policy it reads from next resource; in this case you will have items from two different sources and itemReader bean is not synched to write to right output resource.
You can try to solve let a.b.c.d.MyObject implements org.springframework.batch.item.ResourceAware and write a custom multi-resource item writer and manage resource open/close when resource change (this solution is not trivial because requires state save for restartability, of course).

class MyItemWriter implements ItemWriter<MyObject>,ItemStream {
  private FlatFileItemWriter delegate;
  private String resourceName;

  public open(ExecutionContext e) {
    resourceName = e.getString("resourceName");
    delegate.open(e);
  }
  public close(ExecutionContext e) {
    e.putString("resourceName",resourceName);
    delegate.close(e);
  }
  public update(ExecutionContext e) {
    e.putString("resourceName",resourceName);
    delegate.update(e);
  }
  public void write(List<? extends MyObject> items) {
    for(final MyObject o : items) {
      String itemResource = o.getResource().toString();
      if(!itemResource.equals(resourceName)) {
        closeDelegate();
        openDelegate();
      }
      writeToDelegate(o);
    }
  }
}

Of course,this is not tested and it's just an idea.
Another way is quite different and simulate a loop using a decider in this way:

1. Create a bean (called ResourceHolder, for example) used to store resources from file:#{jobParameters['cwd']}/#{jobParameters['inputFolder']}/file-*.txt; this bean is not stored into execution context but filled with a JobExecutionListener at every job start because execution contexts should be stay small.
2. Use a decider to simulate a loop! from decider store into execution context the name of the current resource; also you need to store the index of current resource to move to next one or to stop your job
3. Read/process/write step use one single file,the one stored into execution context from decider

---

<batch:job id="job">
  <batch:decision decider="decider" id="decider">
    <batch:next on="CONTINUABLE" to="readWriteStep" />
    <batch:end on="FINISHED" />
  </batch:decision>
  <batch:step id="readWriteStep" reader="itemReader" write="itemWriter" next="decider" />
  <batch:listeners>
    <batch:listener class="ResourceHolderFiller">
      <property name="resources" value="file:#{jobParameters['cwd']}/#{jobParameters['inputFolder']}/file-*.txt" />
      <property name="resourceHolderBean" ref="resourceHolderBean" />
    </batch:listener>
  </batch:listeners>
</batch:job>
<bean id="resourceHolder" class="ResourceHolderBean" />
<bean id="decider" class="MyDecider">
  <property name="resourceHolderBean" ref="resourceHolderBean" />
</bean>

class ResourceHolderBean
{
  Resource[] resource;

  public void setResource(Resource[] resource) {...}
  public Resource[] getResource() {...}
}
class MyDecider implements JobExecutionDecider {
  ResourceHolderBean resourceBean;

  public void setResourceBean(ResourceHolderBean resBean){...}

  FlowExecutionStatus decide(JobExecution jobExecution, StepExecution stepExecution) {
    Integer resourceIndex = jobExecution.getExecutionContext().getInt("resourceIndex");
    if(resourceIndex == null)
    {
      resourceIndex = 0;
    }
    if(resourceIndex == resourceBean.getResources().size()) {
      return new FlowExecutionStatus(RepeatStatus.FINISHED.name());
    }
    else {
      jobExecution.getExecutionContext().putInt("resourceIndex",resourceIndex);
      // This is the key your reader/writer must use as inpu/output filename
      jobExecution.getExecutionContext().putString("resourceName",resourceBean.getResources()[resourceIndex].toString());
    }
  }
}

ResourceHolderFiller is a job listener that injects resources into ResourceHolderBean bean.

Hope that can help.