Spring Integration - SftpPersistentAcceptOnceFileListFilter filters out new file / modified file

Question

I have added SftpPersistentAcceptOnceFileListFilter on my Sftp Outbound Gateway to filter out files that I have processed.

<int-sftp:outbound-gateway id="gateway"
  expression="payload"
  request-channel="request"
  remote-directory="main"
  command-options="-R"
  command="ls"
  session-factory="sessionFactory"
  filter="sftpPersistentAcceptOnceFileListFilter"
  reply-channel="reply">
</int-sftp:outbound-gateway>

 <bean name="metadataStore" class="org.springframework.integration.metadata.PropertiesPersistingMetadataStore">
    <property name="baseDirectory" value="/temp"/>
</bean>

 <bean id="sftpPersistentAcceptOnceFileListFilter" class="org.springframework.integration.sftp.filters.SftpPersistentAcceptOnceFileListFilter">
    <constructor-arg name="store" ref="metadataStore"/>
    <constructor-arg name="prefix" value="files/"/>
 </bean>

For example I have this directory :

main
  --directoryA
    --file1.txt
    --file2.txt
    --file3.txt

During the first polling, everything seems working fine. Checked on the metadata store and the file1, file2, file3 were stored there. The next polling, payload returns empty since all the files are already in the store and they are all filtered out by the filter as expected. However when I tried to load another file, it is somehow filtered out and was not saved on the metadata store. The same happens when I upload a new version of file1.txt. I expected it will passed through the filter since the modified date changes but somehow it did not. Did I missed some configuration here?

main
  --directoryA
    --file1.txt
    --file2.txt
    --file3.txt
    **--file4.txt**

question from:https://stackoverflow.com/questions/66053141/spring-integration-sftppersistentacceptoncefilelistfilter-filters-out-new-file

Answer 1

I just tested it and it worked fine for me.

$ tree ~/main/
/.../main/
├── directoryA
│?? ├── newOne
│?? └── subdirectory1
│??     └── bar
└── directoryB
    └── subdirectory1
        └── foo

4 directories, 3 files

I added newOne after the second poll:

GenericMessage [payload=[FileInfo [isDirectory=true, isLink=false, Size=96, ModifiedTime=Fri Feb 05 10:30:20 EST 2021, Filename=directoryA, RemoteDirectory=main/, Permissions=drwxr-xr-x], FileInfo [isDirectory=true, isLink=false, Size=96, ModifiedTime=Mon Feb 01 10:09:57 EST 2021, Filename=directoryA/subdirectory1, RemoteDirectory=main/, Permissions=drwxr-xr-x], FileInfo [isDirectory=false, isLink=false, Size=0, ModifiedTime=Mon Feb 01 10:09:57 EST 2021, Filename=directoryA/subdirectory1/bar, RemoteDirectory=main/, Permissions=-rw-r--r--], FileInfo [isDirectory=true, isLink=false, Size=96, ModifiedTime=Mon Feb 01 10:09:35 EST 2021, Filename=directoryB, RemoteDirectory=main/, Permissions=drwxr-xr-x], FileInfo [isDirectory=true, isLink=false, Size=96, ModifiedTime=Mon Feb 01 10:09:49 EST 2021, Filename=directoryB/subdirectory1, RemoteDirectory=main/, Permissions=drwxr-xr-x], FileInfo [isDirectory=false, isLink=false, Size=0, ModifiedTime=Mon Feb 01 10:09:49 EST 2021, Filename=directoryB/subdirectory1/foo, RemoteDirectory=main/, Permissions=-rw-r--r--]], headers={file_remoteHostPort=localhost:22, id=94154291-1640-7e9c-25a4-707844517008, file_remoteDirectory=main/, timestamp=1612539050536}]
GenericMessage [payload=[], headers={file_remoteHostPort=localhost:22, id=931d9c2d-6e09-7435-dc61-caa95f6659b5, file_remoteDirectory=main/, timestamp=1612539060927}]
GenericMessage [payload=[FileInfo [isDirectory=true, isLink=false, Size=128, ModifiedTime=Fri Feb 05 10:31:06 EST 2021, Filename=directoryA, RemoteDirectory=main/, Permissions=drwxr-xr-x], FileInfo [isDirectory=false, isLink=false, Size=0, ModifiedTime=Fri Feb 05 10:31:06 EST 2021, Filename=directoryA/newOne, RemoteDirectory=main/, Permissions=-rw-r--r--]], headers={file_remoteHostPort=localhost:22, id=ebee7a69-c351-7ca7-010d-e90937d61dfa, file_remoteDirectory=main/, timestamp=1612539071380}]
GenericMessage [payload=[], headers={file_remoteHostPort=localhost:22, id=097dae97-16d6-b8c4-62bb-d5581a27824e, file_remoteDirectory=main/, timestamp=1612539081806}]

I did notice, however, that we only store the file name, not the subdirectory, so if you have the same file name in multiple places, it won't work as expected.

I opened an issue for that.

By the way, you should use a more robust metadata store for production (or at least filter.setFlushOnUpdate(true); on the filter; otherwise they won't be persisted until the app is shutdown cleanly).

EDIT

Oh, but I found this was not picked up...

$ touch ~/main/directoryA/subdirectory1/andAnother

Investigating.

EDIT2

The issue is we are filtering the subdirs; when I added newOne it worked because directoryA's timestamp changed. When I added andAnother, the subdir timestamp changed but directoryA's did not.

Here is a work around:

SftpPersistentAcceptOnceFileListFilter filter = new SftpPersistentAcceptOnceFileListFilter(store, "foo") {

    @Override
    public boolean accept(LsEntry file) {
        if (file.getAttrs().isDir()) {
            return true;
        }
        else {
            return super.accept(file);
        }
    }

};

GenericMessage [payload=[FileInfo [isDirectory=false, isLink=false, Size=0, ModifiedTime=Fri Feb 05 10:31:06 EST 2021, Filename=directoryA/newOne, RemoteDirectory=main/, Permissions=-rw-r--r--], FileInfo [isDirectory=false, isLink=false, Size=0, ModifiedTime=Mon Feb 01 10:09:57 EST 2021, Filename=directoryA/subdirectory1/bar, RemoteDirectory=main/, Permissions=-rw-r--r--], FileInfo [isDirectory=false, isLink=false, Size=0, ModifiedTime=Mon Feb 01 10:09:49 EST 2021, Filename=directoryB/subdirectory1/foo, RemoteDirectory=main/, Permissions=-rw-r--r--]], headers={file_remoteHostPort=localhost:22, id=50faae38-5965-8655-102c-b263398e435b, file_remoteDirectory=main/, timestamp=1612541209839}]
GenericMessage [payload=[], headers={file_remoteHostPort=localhost:22, id=36cedd79-80e1-0c73-ba09-205528dccfca, file_remoteDirectory=main/, timestamp=1612541220313}]
GenericMessage [payload=[], headers={file_remoteHostPort=localhost:22, id=6a0826b4-f960-8b5b-a14e-1c205ca4f357, file_remoteDirectory=main/, timestamp=1612541230803}]
GenericMessage [payload=[FileInfo [isDirectory=false, isLink=false, Size=0, ModifiedTime=Fri Feb 05 11:07:11 EST 2021, Filename=directoryA/subdirectory1/andAnother, RemoteDirectory=main/, Permissions=-rw-r--r--]], headers={file_remoteHostPort=localhost:22, id=dcd9b6a4-5710-421c-1864-17b166d47a5a, file_remoteDirectory=main/, timestamp=1612541241217}]
GenericMessage [payload=[], headers={file_remoteHostPort=localhost:22, id=3eb67995-9ca1-6b8a-d9bd-02cd6d9ea894, file_remoteDirectory=main/, timestamp=1612541251695}]