c++ - Software Mixing of Multiple Audio Streams From Network

Question

TLDR: How do I mix network streams when streams might be unreliable?

I am working on a project which will allow musicians to practice/perform together over the internet with low latency. (Think JackTrip but more user friendly).

I am receiving raw PCM audio data via UDP over the network. As these come in, I identify who the sender is and pipe that stream into that particular person's buffer. I have a separate buffer for each person participating.

Periodically I then take one "chunk" at a time from each buffer, mix them together and output to the speakers of the device.

The issue I'm having is knowing when to take that "chunk". Some clients go offline and their buffers run empty. I cannot just wait for their buffer to refill as this will slow down the other clients.

Thoughts I had were to look at all the buffers at a particular point in time and only process a chunk when at least half of the buffers are populated.

Any ideas?

Other details:

• My buffers are 128 samples big.
• I am using a sample rate of 44100
• I am using 16 bit signed integer pcm
• Streams are MONO
• I am using PortAudio and C++.
• I am using ASIO audio drivers.

question from:https://stackoverflow.com/questions/65903637/software-mixing-of-multiple-audio-streams-from-network

Answer 1

Waitting for answers