Subject: [Sframe] using sframe with existing media networks

From: "Pascal Buhler (pabuhler)" <[email-masked]>
Date: Fri, 12 Jun 2020 08:03:15 +0000

Hi,

I had a couple of questions relating to the draft posted by Emad and sframe usage in general, not sure if it is best to post here, on github or a different forum.

First I make the assumption that although sframe is a valid concept/standard on its own and that webrtc is not explicitly mentioned, I take it that it is in the context of webrtc and the insertable streams api that this was originally developed.

I am looking at this from the prospective of a webrtc gateway, so there is no guarantee that there is a webrtc client at both ends.

I see that the encrypted frame will be packetized with a generic RTP packetizer and use a generic RTP header extension, but no references to what these are or how one would signal/configure a client to use the generic packetize instead of a codec specific one. This is maybe not directly relayed to the sframe standard but relevant when using sframe in conjunction with other standards, for example SDP.

Secondly I am interested in the Partial Decoding section, this seams to require that an entire frame is encoded as a single blob but there are many codecs that can encode parts of a frame as independent units, for example a single NAL that covers only a slice of a frame in H.264 or a single OBU that contains a tile in AV1. In these cases the units could still span many packets. Is it possible for sframe to work in these cases if one considers each of these unit to be a frame in the sframe concept? I am not familiar enough with the insertable streams api in webrtc to know how that works in the case of slices and tiles.

Thanks

pascal

---

From: Justin Uberti <[email-masked]>
Date: Fri, 12 Jun 2020 09:32:51 -0700

On Fri, Jun 12, 2020 at 1:03 AM Pascal Buhler (pabuhler) <pabuhler=
[email-masked]> wrote:

> Hi,
>
>
>
> I had a couple of questions relating to the draft posted by Emad and
> sframe usage in general, not sure if it is best to post here, on github or
> a different forum.
>
>
>
> First I make the assumption that although sframe is a valid
> concept/standard on its own and that webrtc is not explicitly mentioned, I
> take it that it is in the context of webrtc and the insertable streams api
> that this was originally developed.
>
>
>
> I am looking at this from the prospective of a webrtc gateway, so there is
> no guarantee that there is a webrtc client at both ends.
>
>
>
> I see that the encrypted frame will be packetized with a generic RTP
> packetizer and use a generic RTP header extension, but no references to
> what these are or how one would signal/configure a client to use the
> generic packetize instead of a codec specific one. This is maybe not
> directly relayed to the sframe standard but relevant when using sframe in
> conjunction with other standards, for example SDP.
>

Yes, some form of signaling is definitely needed, perhaps some new option
in a m= section negotiating the use of a generic packetizer.

>
>
> Secondly I am interested in the Partial Decoding section, this seams to
> require that an entire frame is encoded as a single blob but there are many
> codecs that can encode parts of a frame as independent units, for example a
> single NAL that covers only a slice of a frame in H.264 or a single OBU
> that contains a tile in AV1. In these cases the units could still span many
> packets. Is it possible for sframe to work in these cases if one considers
> each of these unit to be a frame in the sframe concept? I am not familiar
> enough with the insertable streams api in webrtc to know how that works in
> the case of slices and tiles.
>

Yes, to enable this each independently decodable unit would be a 'frame'
for SFrame. This would include H.264 slices/AV1 OBUs, as well as individual
layers in a scalable coding context.