Subject: [Mops] Low latency protocol comparision

From: "Leslie Daigle" <[email-masked]>
Date: Sat, 13 Jun 2020 12:58:25 -0400

I thought this was a pretty interesting analysis:

Low broadcast latency and protocols for implementation thereof:
https://www.elecard.com/page/low_broadcast_latency_and_protocols_for_implementation_thereof

Particularly telling is the chart at the end:  WebRTC and SRT don’t 
look very similar, in this particular regard.

Leslie.

-- 

-------------------------------------------------------------------
Leslie Daigle
Principal, ThinkingCat Enterprises
[email-masked]
-------------------------------------------------------------------

---

From: Alexandre GOUAILLARD <[email-masked]>
Date: Sun, 14 Jun 2020 00:34:03 +0200

There are multiple inaccurate statements in the linked documents. Most of
them that have been carried around for many years by other SRT vendors like
Wowza and already called out (here
<http://webrtcbydralex.com/index.php/2019/05/18/wowzas-marketing-at-work-again-webrtc/>
for example).

I'm just going to point to three of the most egregious (read "verifiably
wrong") statements, that by themselves undermine the credibility of the
entire document.

- "Maximum supported resolution: 720p, 30 frames per second with a bit rate
of up to 2 Mbps."
- There is no limit to the spatial and temporal resolution in webrtc/rtcweb
protocol.
Stadia for example send up to 4K@30fps.

"WebRTC is inferior to its colleagues in terms of the coding quality and
maximum amount of transmitted data.
- There is no limit to the bandwidth used in the webrtc/rtcweb protocol.
- There is a limit of 2Mbps per peer connection on the sending-side *only
in the google chrome implementation*, only by default (web application can
lift this limit), and only for webcams devices (screensharing has no
bandwidth limit). This is a chrome implementation detail, and not a
protocol limit.
-While VP8 and H264 are both mandatory to implement (original IETF
discussion
<https://mailarchive.ietf.org/arch/msg/rtcweb/juVsZKp-AmRpYLCgy-uwartuNrY/>)
anybody can add any codec to webrtc as long as an RTP payload specification
exists.
-- Google chrome, and firefox, both supports real-time VP9 mode 2 (4:4:4,
10 bits, HDR)
https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/modules/video_coding/codecs/vp9/vp9_impl.cc;l=985
-- Chrome has a real-time AV1 implementation in webrtc, which goes up to
high profile (4:4:4, 8~10bits, HDR).
https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/modules/video_coding/codecs/av1/
In terms of quality, only codecs at high profile (i.e. 12bits) can do
better, and none of those have a "real-time" version today.
In terms of coding efficiency, AV1 is still the best codec available in
production.

"WebRTC is not available in Safari and partially unavailable in Bowser and
Edge."
- webrtc is fully available in Edge, and Safari.
https://caniuse.com/#search=webrtc
official daily W3C results:
https://wpt.fyi/results/?label=master&label=experimental&aligned&q=webrtc
- Safari has added H.265 support in addition to the mandatory to implement
VP8 and H.264 in the latest safari tech preview.

I'm not going into the Discussion about webrtc's underlying RTC/RTCP
protocol and corresponding extension: NACK, RTX, FEC, RED, or Congestion
Control (REMB, Google-cc, BBR, NADA), as IETF members I'm sure you re all
aware of RFC3550 and all its derivatives. Many authors of those RFCs are
part of this group.

Maybe more overlooked is the most recent Draft for end-to-end encryption on
top of webrtc co-authored by google:
https://tools.ietf.org/html/draft-omara-sframe-00

I'm not saying that SRT is worse, or better than WebRTC. I'm just saying,
this comparison is not egregiously incorrect, and I'm a little bit
surprised that we are discussing marketing blogs that go in the face of so
many IETF specs.

If anybody is interested in going deep into a real  thorough comparison
between all those protocols, why not doing it during the IETF hackathon in
July, or in october (hopefully in BKK) just like we did group co-testing
with IETF's AVTCORE/AVEXT/QUIC/RTCWEB/RMCAT during the previous IETF
hackathons?

Regards,

On Sat, Jun 13, 2020 at 6:58 PM Leslie Daigle <[email-masked]>
wrote:

> Particularly telling is the chart at the end: WebRTC and SRT don’t look
> very similar, in this particular regard.
>
> Leslie.
>
> --
> ------------------------------
>
> Leslie Daigle
> Mops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mops
>

-- 
Alex. Gouaillard, PhD, PhD, MBA
------------------------------------------------------------------------------------
President - CoSMo Software Consulting, Singapore
------------------------------------------------------------------------------------
sg.linkedin.com/agouaillard

   -

---

From: "Leslie Daigle" <[email-masked]>
Date: Sat, 13 Jun 2020 18:41:32 -0400

Hi,

[You wrote:]
> with IETF's AVTCORE/AVEXT/QUIC/RTCWEB/RMCAT during the previous IETF

Volunteering? :->

If there’s interest in doing it, along the specific vector of video 
delivery, and willingness to write it up as an I-D afterwards, I think 
it would be great.

Leslie.

--

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Sat, 13 Jun 2020 23:05:43 +0000

Yes, it’s a bit marketing, but it’s also one of the few write ups that attempts to put the popular media transports side by side for people to understand their strengths and abilities.   The marketing blogs also give us, the IETF, insight into how the outside world of adopters and sellers of our work see things.

I really wish that all web documents and pages contained a posting date.   This one doesn’t gave it a date.  However, the page must be from at least mid 2019 as it mentions things that were released then.

I like your write up of the WebRTC inaccuracies.  It would be great if any SRT folks on this list could do something similar,  to to mention those working with CMAF and LL HLS.  There a lot of adopters out there who are working through what protocol best fits their particular needs - webRTC?   SRT?  Something that bright summer intern wrote while she was here last summer?

 Capturing something like this, and even better some sort of bake off as you suggest would be valuable to a lot of people.  Though it may have to wait to get critical mass of participants until next year.

Glenn

On Jun 13, 2020, at 3:34 PM, Alexandre GOUAILLARD <[email-masked]> wrote:

There are multiple inaccurate statements in the linked documents. Most of them that have been carried around for many years by other SRT vendors like Wowza and already called out (here<http://webrtcbydralex.com/index.php/2019/05/18/wowzas-marketing-at-work-again-webrtc/> for example).

I'm just going to point to three of the most egregious (read "verifiably wrong") statements, that by themselves undermine the credibility of the entire document.

- "Maximum supported resolution: 720p, 30 frames per second with a bit rate of up to 2 Mbps."
- There is no limit to the spatial and temporal resolution in webrtc/rtcweb protocol.
Stadia for example send up to 4K@30fps.

"WebRTC is inferior to its colleagues in terms of the coding quality and maximum amount of transmitted data.
- There is no limit to the bandwidth used in the webrtc/rtcweb protocol.
- There is a limit of 2Mbps per peer connection on the sending-side only in the google chrome implementation, only by default (web application can lift this limit), and only for webcams devices (screensharing has no bandwidth limit). This is a chrome implementation detail, and not a protocol limit.
-While VP8 and H264 are both mandatory to implement (original IETF discussion<https://mailarchive.ietf.org/arch/msg/rtcweb/juVsZKp-AmRpYLCgy-uwartuNrY/>) anybody can add any codec to webrtc as long as an RTP payload specification exists.
-- Google chrome, and firefox, both supports real-time VP9 mode 2 (4:4:4, 10 bits, HDR)
https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/modules/video_coding/codecs/vp9/vp9_impl.cc;l=985
-- Chrome has a real-time AV1 implementation in webrtc, which goes up to high profile (4:4:4, 8~10bits, HDR).
https://source.chromium.org/chromium/chromium/src/+/master:third_party/webrtc/modules/video_coding/codecs/av1/
In terms of quality, only codecs at high profile (i.e. 12bits) can do better, and none of those have a "real-time" version today.
In terms of coding efficiency, AV1 is still the best codec available in production.

"WebRTC is not available in Safari and partially unavailable in Bowser and Edge."
- webrtc is fully available in Edge, and Safari.
https://caniuse.com/#search=webrtc
official daily W3C results: https://wpt.fyi/results/?label=master&label=experimental&aligned&q=webrtc
- Safari has added H.265 support in addition to the mandatory to implement VP8 and H..264 in the latest safari tech preview.

I'm not going into the Discussion about webrtc's underlying RTC/RTCP protocol and corresponding extension: NACK, RTX, FEC, RED, or Congestion Control (REMB, Google-cc, BBR, NADA), as IETF members I'm sure you re all aware of RFC3550 and all its derivatives. Many authors of those RFCs are part of this group.

Maybe more overlooked is the most recent Draft for end-to-end encryption on top of webrtc co-authored by google:
https://tools.ietf.org/html/draft-omara-sframe-00

I'm not saying that SRT is worse, or better than WebRTC.. I'm just saying, this comparison is not egregiously incorrect, and I'm a little bit surprised that we are discussing marketing blogs that go in the face of so many IETF specs.

If anybody is interested in going deep into a real  thorough comparison between all those protocols, why not doing it during the IETF hackathon in July, or in october (hopefully in BKK) just like we did group co-testing with IETF's AVTCORE/AVEXT/QUIC/RTCWEB/RMCAT during the previous IETF hackathons?

Regards,

On Sat, Jun 13, 2020 at 6:58 PM Leslie Daigle <[email-masked]<mailto:[email-masked]>> wrote:

Particularly telling is the chart at the end: WebRTC and SRT don’t look very similar, in this particular regard.

Leslie.

--

________________________________

Leslie Daigle

[email-masked]<mailto:[email-masked]>
--
Mops mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/mops

--
Alex. Gouaillard, PhD, PhD, MBA
------------------------------------------------------------------------------------
President - CoSMo Software Consulting, Singapore
------------------------------------------------------------------------------------
sg.linkedin.com/agouaillard<http://sg.linkedin.com/agouaillard>

  *

---

From: Alexandre GOUAILLARD <[email-masked]>
Date: Sun, 14 Jun 2020 01:27:40 +0200

@Deen, agreed. I gave it a try though a couple of years back. Let me know
if that would go in the right direction and I can update it with everything
I learned in the mean time, and taking into account the updates to the
different protocols (e.g. new HLS informational draft by pantos):
http://webrtcbydralex.com/index.php/2018/05/15/streaming-protocols-and-ultra-low-latency-including-webrtc/

@Leslie, I volunteer every year, every hackathon :-)
However, I am more knowledgeable about webrtc than other protocols, so in
that regard, I am biased.
I think to be fair each protocol should have a
reasonable representative/champion that at least validates that the testing
protocol chosen is adequate, and garantees the accuracy and the fairness of
the result.

For webrtc, it was problematic as well, as each browser vendor did not want
someone else to benchmark their implementation, so we ended up with one
representant of each browser vendor at 104 I believe (pragues), to validate
its result, and did it again in montreal:
https://trac.ietf.org/trac/ietf/meeting/wiki/105hackathon/webrtc
if we manage to get the same setting, I think it could be very successful
in getting fair and accurate results.

I think we could handle the webrtc testing (as in writing open source tests
and providing results that would be deterministic and
independently verifiable by any and all).
- Maybe Veriskope, which is trying to push an RTMP 2.0, could be interested
in joining for RTMP.
- Magnus W for RTSP 2.0 ?
- Then we would need one volunteer for SRT,
- and another one for QUIC (maybe martin thompson can point us in the right
direction). QUIC is always the biggest table so it should not be too
difficult.
- Jonathan Lennox, chairman of AVTCORE is always around and could help,
especially on the congestion control, Error concealment, and bandwidth
adaptation part
- We could motivate EKR to comment on the security aspects.
- I'm not sure if we want to asses IPv6 and NAT traversal readiness (ICE)

What do you think?

Regards,

Dr. Alex

On Sun, Jun 14, 2020 at 1:05 AM Deen, Glenn (NBCUniversal) <
[email-masked]> wrote:

> Yes, it’s a bit marketing, but it’s also one of the few write ups that
> attempts to put the popular media transports side by side for people to
> understand their strengths and abilities.   The marketing blogs also give
> us, the IETF, insight into how the outside world of adopters and sellers of
> our work see things.
>
> I really wish that all web documents and pages contained a posting date.
> This one doesn’t gave it a date.  However, the page must be from at least
> mid 2019 as it mentions things that were released then.
>
> I like your write up of the WebRTC inaccuracies.  It would be great if any
> SRT folks on this list could do something similar,  to to mention those
> working with CMAF and LL HLS.  There a lot of adopters out there who are
> working through what protocol best fits their particular needs - webRTC?
> SRT?  Something that bright summer intern wrote while she was here last
> summer?
>
>  Capturing something like this, and even better some sort of bake off as
> you suggest would be valuable to a lot of people.  Though it may have to
> wait to get critical mass of participants until next year.
>
> Glenn
>
> in the google chrome implementation*, only by default (web application
> can lift this limit), and only for webcams devices (screensharing has no
> VP8 and H..264 in the latest safari tech preview.
>
> I'm not saying that SRT is worse, or better than WebRTC.. I'm just saying,

   -

---

From: Maxim Sharabayko <[email-masked]>
Date: Sun, 14 Jun 2020 20:05:14 +0000

Hi,

The idea of performing the comparison is very exciting! I would be glad to participate as a delegate from SRT.

@Glen
It would be great if any SRT folks on this list could do something similar,  to to mention those working with CMAF and LL HLS.

There are of course some inaccuracies in the description of SRT.
The receiver buffer latency of 120 ms is the default value, not the minimum. The minimum is 1 ms.
There is no limit of the number of packet retransmissions, although the configurable limit might be added soon.
HLS and DASH can be transmitted over SRT. And so on.

But in general even with these inaccuracies, the overview of the existing protocols on the market is quite good.

--
Regards,
Maxim Sharabayko
Senior Software Developer | Haivision

From: Mops <[email-masked]> on behalf of Alexandre GOUAILLARD <[email-masked]>
Date: Sunday, 14. June 2020 at 01:28
To: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Cc: Leslie Daigle <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [Mops] Low latency protocol comparision

@Deen, agreed. I gave it a try though a couple of years back. Let me know if that would go in the right direction and I can update it with everything I learned in the mean time, and taking into account the updates to the different protocols (e.g. new HLS informational draft by pantos):
http://webrtcbydralex.com/index.php/2018/05/15/streaming-protocols-and-ultra-low-latency-including-webrtc/

@Leslie, I volunteer every year, every hackathon :-)
However, I am more knowledgeable about webrtc than other protocols, so in that regard, I am biased.
I think to be fair each protocol should have a reasonable representative/champion that at least validates that the testing protocol chosen is adequate, and garantees the accuracy and the fairness of the result.

For webrtc, it was problematic as well, as each browser vendor did not want someone else to benchmark their implementation, so we ended up with one representant of each browser vendor at 104 I believe (pragues), to validate its result, and did it again in montreal:
https://trac.ietf.org/trac/ietf/meeting/wiki/105hackathon/webrtc
if we manage to get the same setting, I think it could be very successful in getting fair and accurate results.

I think we could handle the webrtc testing (as in writing open source tests and providing results that would be deterministic and independently verifiable by any and all).
- Maybe Veriskope, which is trying to push an RTMP 2.0, could be interested in joining for RTMP.
- Magnus W for RTSP 2.0 ?
- Then we would need one volunteer for SRT,
- and another one for QUIC (maybe martin thompson can point us in the right direction). QUIC is always the biggest table so it should not be too difficult.
- Jonathan Lennox, chairman of AVTCORE is always around and could help, especially on the congestion control, Error concealment, and bandwidth adaptation part
- We could motivate EKR to comment on the security aspects.
- I'm not sure if we want to asses IPv6 and NAT traversal readiness (ICE)

What do you think?

Regards,

Dr. Alex

On Sun, Jun 14, 2020 at 1:05 AM Deen, Glenn (NBCUniversal) <[email-masked]<mailto:[email-masked]>> wrote:
Yes, it’s a bit marketing, but it’s also one of the few write ups that attempts to put the popular media transports side by side for people to understand their strengths and abilities.   The marketing blogs also give us, the IETF, insight into how the outside world of adopters and sellers of our work see things.

Glenn

On Jun 13, 2020, at 3:34 PM, Alexandre GOUAILLARD <[email-masked]<mailto:[email-masked]>> wrote:
There are multiple inaccurate statements in the linked documents. Most of them that have been carried around for many years by other SRT vendors like Wowza and already called out (here<http://webrtcbydralex.com/index.php/2019/05/18/wowzas-marketing-at-work-again-webrtc/> for example).

Regards,

Leslie.

--

Leslie Daigle

·

·

---

From: Alexandre GOUAILLARD <[email-masked]>
Date: Wed, 24 Jun 2020 15:41:20 +0200

Dear all,

I went around to the long time Hackathon attending IETF media guys and it
looks like it's going to be too short notice to prepare something for
Madrid.
Between AV1 RTP payload and implementation, the SVC extensions, the
QuicTransport / WebTransport , and the End-To-End encryption for real-time
media, everybody that would be otherwise interested and capable to
contribute is real busy.

However, I gathered sufficient interest for BKK (whether it will be virtual
or not), given the usual rules those people abide by are followed. Since
IETF does not have a set of proper rules nor a formal test suite, those
follow the rules that were used at IMTC superop events, previous IETF
Hackathon, W3C 's web-platform tests and/or AOMedia 's testing working
group. The usual IETF rules about the usage of MUST or SHOULD apply.
- Test cases must be discussed and defined early on, to make sure the
"everything else being equal" statement applies. The definition of starting
point and end point of latency measurement for example varies a lot
depending on the study.
- Test content (if any) must be defined and made publicly available to be
able to independently reproduce results.
- Test suite should be written in advance. The same test suite should be
used by all. The test code must be made publicly available to be able to
independently reproduce results.
- If the need for specific test bed appears, e.g. controlled network,
programmatic packet loss / jitter / bandwidth variation, we should speak
with the IETF network guys, and/or Charles E. to have that set up (in the
case of a real meeting), or for a common open source test infrastructure
like e.g. KITE to be set up and made available.

We usually consider it fair to provide 3 months to everyone after the first
comparative results are out, to try to optimize their implementations or
address bugs found in the process, so that the results are representative
of the capacity of a protocol, and not the deficiency of a given
implementation. I don't know if it makes sense to do this here, e.g. to
have temporary results in BKK, but give untill IETF 110 in Pragues to
freeze the data, since SRT is not an IETF protocol with fixed versions and
revisions at this stage.

If this sounds reasonable to at least one of the chair, and from maxim or
another SRT representative, I'm happy to facilitate for IETF 109 the set
up, bring enough IETF people and open source media server people to the
table to have critical mass of data and review, and to facilitate the
writing of a subsequent ID, in the same spirit we published a comparative
study of open-source media webrtc media servers:
https://www.semanticscholar.org/paper/Comparative-Study-of-WebRTC-Open-Source-SFUs-for-Andre-Breton/10b877cc13ffc9a1cabbf415cba73b22a1dd5442

Best regards,

Dr. Alex.

On Sun, Jun 14, 2020 at 10:05 PM Maxim Sharabayko <
[email-masked]> wrote:

> *It would be great if any SRT folks on this list could do something
> similar,  to to mention those working with CMAF and LL HLS.*
>
>
>
> *From: *Mops <[email-masked]> on behalf of Alexandre GOUAILLARD <
> [email-masked]>
> *Date: *Sunday, 14. June 2020 at 01:28
> *To: *"Deen, Glenn (NBCUniversal)" <[email-masked]>
> *Cc: *Leslie Daigle <[email-masked]>, "[email-masked]" <
> [email-masked]>
> *Subject: *Re: [Mops] Low latency protocol comparision
>
>
>
> For webrtc, it was problematic as well, as each browser vendor did not
> want someone else to benchmark their implementation, so we ended up with
> one representant of each browser vendor at 104 I believe (pragues), to
> validate its result, and did it again in montreal:
>
> I think we could handle the webrtc testing (as in writing open source
> tests and providing results that would be deterministic and
> - and another one for QUIC (maybe martin thompson can point us in the
> right direction). QUIC is always the biggest table so it should not be too
> difficult.
>

   -

---

From: "Leslie Daigle" <[email-masked]>
Date: Wed, 24 Jun 2020 18:16:14 -0400

Hi,

Speaking as only one of the chairs of this group, I have to say I like 
how this is shaping up.

My only question is whether there is work to be done even at the IETF 
108 hackathon, in order to establish the test cases and test suite?

Happy to hear from the other co-chair of this group :^) and any other 
group participants.

Leslie.

On 24 Jun 2020, at 9:41, Alexandre GOUAILLARD wrote:

>> The idea of performing the comparison is very exciting! I would be 
>> glad to
>> participate as a delegate from SRT.
>>
>>
>>
>> @Glen
>>
>> There is no limit of the number of packet retransmissions, although 
>> the
>> configurable limit might be added soon.
>>
>> But in general even with these inaccuracies, the overview of the 
>> existing
>> protocols on the market is quite good.
>>
>>
>>
>> --
>>
>> Regards,
>>
>> Maxim Sharabayko
>>
>> On Jun 13, 2020, at 3:34 PM, Alexandre GOUAILLARD 
>> <[email-masked]>
>> wrote:
>>
>> - "Maximum supported resolution: 720p, 30 frames per second with a 
>> bit
>> rate of up to 2 Mbps."
>> Maybe more overlooked is the most recent Draft for end-to-end 
>> encryption
>> on top of webrtc co-authored by google:
>>

--

---

From: Kyle Rose <[email-masked]>
Date: Wed, 24 Jun 2020 18:37:40 -0400

On Wed, Jun 24, 2020 at 6:16 PM Leslie Daigle <[email-masked]>
wrote:

> Hi,
>
> Speaking as only one of the chairs of this group, I have to say I like how
> this is shaping up.
>
> My only question is whether there is work to be done even at the IETF 108
> hackathon, in order to establish the test cases and test suite?
>

Heh...  As the other co-chair, Leslie and I are in alignment on this. If
there's prep work to be done at the 108 hackathon, great! If not, BKK (or,
likely, the virtual replacement for it) sounds like a plan. Even if we
don't do anything at the 108 hackathon, there's probably a bit of prep
we'll want to do prior to the 109 hackathon, including coordinating with
Charles on any special network setup we might need should the meeting be
held in person.

Kyle