ffmpeg - Send sprop-parameter-sets inband rather than in SDP

Question

I am trying to use ffmpeg to stream an MP4 file over rtp. I am sending the stream to a SFU server that will broadcast the stream to users. The clients are expecting to receive an h264 video stream with profile-level-id 42e01f. The issue I'm having is that the video received by the clients does not decode properly (just a black screen). If I transcode the video before sending, then everything works correctly. If I dump the SDP that describes what ffmpeg is sending, there is a distinct difference between the transcoded and non-transcoded version.

For the non-transcoded version, my ffmpeg command looks like

ffmpeg  '-re' 
  '-v' 
  'info' 
  '-protocol_whitelist' 
  'pipe,tls,file,http,https,tcp,rtp' 
  '-i' 
  '-f' 'mp4' 
  'https://storage.googleapis.com/my_bucket/file' 
  '-map' 
  '0:v:0' 
  '-c:v' 
  'copy' 
  '-f' 
  'rtp' 
  '-sdp_file' 'out.sdp' 
  'rtp://142.93.14.110:40425?rtcpport=45155'

When I run this command, out.sdp contains the line

a=fmtp:96 packetization-mode=1; sprop-parameter-sets=Z0LAH9kAUAW7AWoCAgKAAAH0gABdwAeMGSQ=,aMuMsg==; profile-level-id=42C01F

However, if I change -c:v copy to -c:v libx264 -preset ultrafast, then the sdp line changes to a=fmtp:96 packetization-mode=1;. Given that there is no SDP exchange between ffmpeg and my SFU, I think the issue is that ffmpeg needs to be sending the sprops in-band rather than setting them in the sdp. Any help here would be amazing. The other possible issue is that the profile levels are sightly different.

Answer 1

A H.264 video stream always has metadata associated with it - called SPS and PPS. SPS/PPS can be in band or stored in the MP4 file or both. The sprop-parameter-sets represents SPS and PPS BASE64 encoded and comma separated. In your case a number of things can be wrong and I can only guess.

original MP4s don't have in band SPS/PPS (legal but may throw off your decode process)

original H.264 slightly out of spec

mismatch between inband SPS/PPS and MP4 SPS/PPS

incorrect implemented RTP/RTSP client of your SFU server

I assume that you want to avoid the transcode. I would first check if SPS/PPS are in band in your source MP4. If not - I would add SPS/PPS in band without transcoding.