We are using an IMAP consumer processor in our nifi pipeline to read the email from office 365. We have been observing issues in the IMAP processor while consuming the email from office 365 email box. We have scheduled the processor to run every 30 seconds with 1 concurrent task. The fetch size is kept 20. Our mailbox has 1000s of mails but still it is only fetching 1 mail every 30 sec even though we have specified fetch size as 20.
Processor Conf
Processor Conf
Please let us know what could be the reason or What is the optimal configuration we can have for the processor. We expect to get around 10000 mails per day but we don't want to get throttling error with mail server.
We are also getting the below error time to time in the ConsumeIMAP processor. We are not able to find the root cause. Any help is appreciated.
ERROR [Timer-Driven Process Thread-33] o.a.nifi.processors.email.ConsumeIMAP ConsumeIMAP[id=c31e4176-842d-3464-b870-2460ee675eee] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: Failed to receive messages from Email server: [javax.mail.MessagingException - IOException while copying message: org.apache.nifi.processor.exception.ProcessException: Failed to receive messages from Email server: [javax.mail.MessagingException - IOException while copying message
org.apache.nifi.processor.exception.ProcessException: Failed to receive messages from Email server: [javax.mail.MessagingException - IOException while copying message
at org.apache.nifi.processors.email.AbstractEmailProcessor.fillMessageQueueIfNecessary(AbstractEmailProcessor.java:328)
at org.apache.nifi.processors.email.AbstractEmailProcessor.receiveMessage(AbstractEmailProcessor.java:381)
at org.apache.nifi.processors.email.AbstractEmailProcessor.onTrigger(AbstractEmailProcessor.java:195)
at org.apache.nifi.processors.email.ConsumeIMAP.onTrigger(ConsumeIMAP.java:32)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.mail.MessagingException: IOException while copying message
at javax.mail.internet.MimeMessage.(MimeMessage.java:254)
at org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage.(AbstractMailReceiver.java:557)
at org.springframework.integration.mail.AbstractMailReceiver$IntegrationMimeMessage.(AbstractMailReceiver.java:552)
at org.springframework.integration.mail.AbstractMailReceiver.postProcessFilteredMessages(AbstractMailReceiver.java:415)
at org.springframework.integration.mail.AbstractMailReceiver.receive(AbstractMailReceiver.java:342)
at org.apache.nifi.processors.email.AbstractEmailProcessor.fillMessageQueueIfNecessary(AbstractEmailProcessor.java:323)
... 15 common frames omitted
Caused by: com.sun.mail.util.FolderClosedIOException: null
at com.sun.mail.imap.IMAPInputStream.forceCheckExpunged(IMAPInputStream.java:107)
at com.sun.mail.imap.IMAPInputStream.fill(IMAPInputStream.java:158)
at com.sun.mail.imap.IMAPInputStream.read(IMAPInputStream.java:218)
at com.sun.mail.imap.IMAPInputStream.read(IMAPInputStream.java:244)
at com.sun.mail.imap.IMAPMessage.writeTo(IMAPMessage.java:849)
at javax.mail.internet.MimeMessage.(MimeMessage.java:245)
... 20 common frames omitted
与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…
