Have you looked at Qualcomm's AllJoyn library? It is designed to work over Bluetooth or wifi, so may fit, though if you're connecting over 3G or wider range networks it won't work.
Given the variation and reliability of networks between two remote devices not on the same network, I'd question whether peer-to-peer is the best solution, and would venture to suggest considering using an application server in between, so you could then use Cloud to Device Messaging [deprecated] (perhaps in tandem with Google App Engine). i.e. to send a message the sender passes it to the server, and the server then passes it on to the recipient.
In theory all devices on the net have a unique IP address and can talk to each other, but it's rarely that simple as routers/firewalls are configured differently so you'd need to pay great attention to the ports you use, especially considering many inbound ports are blocked by default for security reasons.
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…