ActiveMQ - Network of Brokers - Messages dequeued without ACK

Question

General Topology:

• Star of remote brokers in field, running embedded ActiveMQ, with a static network connector / static queues back to AmazonMQ. Low power SBCs
• Local publisher on remote nodes, messages back to AmazonMQ in middle
• Cell connections sometimes unreliable, although when being watched / simulated failing, networkConnectors reconnect successfully and messages resent
• KahaDB on remote nodes as persistence store

I'm hitting a situation whereby after a random time, messages are being enqueued on the remote brokers, and the dequeue on the queues will indicate that the messages are being removed and theoretically hitting the central hub AmazonMQ, however the messages are never actually hitting AmazonMQ in the middle. ACKs are expected for all messages, messages aren't sent non persistent.

As far as I can tell, neither remote or hub broker throws no exceptions, disconnects etc and the connections appear to be up still at both ends. A restart of the remote broker embedded in the app seems to clear this (although the messages are forever lost).

Is there any scenario that could logically cause this?

question from:https://stackoverflow.com/questions/65901346/activemq-network-of-brokers-messages-dequeued-without-ack

Answer 1

Waitting for answers