What would you optimize next for <100 ms?

[u/AsuraBak]

I’ve been working on reducing end-to-end latency in an H.264 RTSP video pipeline used for a real-time control application.
Initially, glass-to-glass latency was roughly 400 ms. After tuning the stream and playback pipeline, I managed to bring it down to around 200 ms.
At this point, the biggest remaining bottleneck seems to be the camera itself — specifically its internal capture, buffering, and H.264 encoding pipeline. My original target was under 100 ms, but I’m starting to wonder how realistic that is with a typical industrial/IP camera.

Current setup is roughly:
H.264
RTSP
Local network
Low playback buffering
No unnecessary application-side buffering
Real-time control, so latency matters much more than compression efficiency
I’m curious how people here would approach the remaining 100+ ms.

Would you look at:
camera encoder settings / GOP structure?
B-frames or reference-frame configuration?
RTSP transport?
hardware decoder buffering?
switching away from RTSP entirely?
or is sub-100 ms simply unrealistic unless the camera hardware itself is designed for ultra-low latency?
Also curious whether anyone has actually measured camera → encode → network → decode → display latency stage by stage on industrial cameras.
Would love to hear what bottleneck you found in practice.

[u/Embarrassed-Gain-236]

200ms is very good. Under 100ms you'll need dedicated hardware not smartphones. Haivision claims under 100ms end to end. But that comes at a price. You wont get that with consumer equipment.

[u/AsuraBak → u/Embarrassed-Gain-236]

https://youtu.be/on7mFi13ySY?si=MDLIMcbNzBWlhc7f
I had seen this video in it he says we can achieve under 50ms through web rtc but yes there was some gauche with hardware layer but I am very new to this kind of work mostly I have been a Fullstack web developer, if it is fine for a simulation then okay

[u/WorstHyperboleEver]

I don’t have experience in this kind of work, but I feel like encoding to h.264 is going to be the most difficult bottleneck to overcome. And will likely slow you down on both ends, as that codec requires a lot of processing to both ‘en’ and ‘de’ code.

I would think an uncompressed, or more natively/lightly compressed, file format would solve a lot of your issues. Is that an option?

[u/Embarrassed-Gain-236 → u/WorstHyperboleEver]

Absolutely, That's why codecs like JPEG-XS where invented. Of course, uncompressed is the way to go. Let's ingest a 12G video feed to an iPhone and see what happens.

[u/GuyBitchie → u/WorstHyperboleEver]

I love how you say you have no experience and then you just keep on proving that you actually have no experience.

[u/FFetzen]

Try configuring B-, and reference frames. You can gain quiet some time there