Subject: [Sframe] Partial decodability and IDUs

From: Sergio Garcia Murillo <[email-masked]>
Date: Thu, 19 Nov 2020 22:39:57 +0100

Hi all,

As most of you already know, this morning I made a presentation in 
AVTCORE introducing the topic about the need to specify an agnostic 
video codec packetization format.

https://datatracker.ietf.org/meeting/109/materials/slides-109-avtcore-sframe-rtp-encapsulation-00

I got an AP for creating an initial draft so it could be reviewed and 
accepted.

However, there were two main concerns that we should address in this 
this group:

  * Historically, avtcore has explicitly designed not to be payload
    agnostic and  declined to standardized codec agnostic payload
    formats in  number of cases.  If that is to be changed, needs to be
    done deliberately.
  * Need to define the "minimum decoding unit" or "independently
    decodable unit", that SFrame will work with.

Regarding the second one

  * Full video frames (just use whatever is the encoder output)
  * Spatial layer frames
  * "independend decodable subframes" like h264 slices, vp8 partitions
    or av1 tiles which allows partial decodability which is mainly aimed
    for enhancing packet loss resilience.

Spatial layer frames is the minimum we should target as if not it will 
just prevent SFUs for using SVC codecs. So the question is if we should 
go deeper and implement lower partitions of the frames or not.

AFAIK, currently, libwertc does not support partial decodability and I 
personally haven't seen any practical usage of this in the RTC world 
(while it makes a lot of sense in streaming/broadcasting world), but 
would like to hear what is the view and experience of the other members 
of this group. Also note that if we are going to support them on SFrame 
this will require a greater effort because we will need to explicitly 
define how the frames must be split before being encrypted y SFrame for 
*each* possible video codec (h264,h265,vp8,vp9,av1,...).

There was also the question about how/if we should support other codec 
features like DON/interleaved mode for h264, which I also think we 
should not support mainly because we are not currently using it on 
webrtc implementations.

What do you think?

Best regards

Sergio

---

From: Justin Uberti <[email-masked]>
Date: Thu, 19 Nov 2020 13:53:57 -0800

The encoder needs to be aware of any mechanism to generate IDUs (e.g.,
slices), and typically each of these IDUs will be handed up to the consumer
individually. So the SFRAME layer doesn't need to do any splitting, it just
knows that it should treat each IDU as something it needs to individually
SFRAME and packetize.

On Thu, Nov 19, 2020 at 1:40 PM Sergio Garcia Murillo <
[email-masked]> wrote:

> Hi all,
>
>
> As most of you already know, this morning I made a presentation in AVTCORE
> introducing the topic about the need to specify an agnostic video codec
> packetization format.
>
>
> However, there were two main concerns that we should address in this this
> group:
>
>    - Historically, avtcore has explicitly designed not to be payload
>    agnostic and  declined to standardized codec agnostic payload formats in
>    number of cases.  If that is to be changed, needs to be done deliberately.
>    - Need to define the "minimum decoding unit" or "independently
>    - Full video frames (just use whatever is the encoder output)
>    - Spatial layer frames
>    - "independend decodable subframes" like h264 slices, vp8 partitions
>    or av1 tiles which allows partial decodability which is mainly aimed for
>    enhancing packet loss resilience.
>
>
> just prevent SFUs for using SVC codecs. So the question is if we should go
> deeper and implement lower partitions of the frames or not.
>
>
> personally haven't seen any practical usage of this in the RTC world (while
> it makes a lot of sense in streaming/broadcasting world), but would like to
> hear what is the view and experience of the other members of this group.
> Also note that if we are going to support them on SFrame this will require
> a greater effort because we will need to explicitly define how the frames
> must be split before being encrypted y SFrame for *each* possible video
> codec (h264,h265,vp8,vp9,av1,...).
>
>
> features like DON/interleaved mode for h264, which I also think we should
> not support mainly because we are not currently using it on webrtc
> implementations.
>
>
> What do you think?
>
>
> Best regards
>
> Sergio
>
>
> --
> Sframe mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sframe
>

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Thu, 19 Nov 2020 23:11:16 +0100

You are right regarding that the SFrame layer does not need to know what 
is feed in for encryption, but in order to be able to have a working end 
to end solution for webrtc, someone will need to define what and how 
this IDUs are generated and reassembled for each codec if we want to 
have interoperable implementations in different devices.

That process is codec-dependant and I would require quite a lot of 
effort (and also supporting it on the agnostic packetization), so I 
would prefer to have strong arguments in favor of doing it.

On 19/11/2020 22:53, Justin Uberti wrote:
> slices), and typically each of these IDUs will be handed up to the 
> consumer individually. So the SFRAME layer doesn't need to do any 
> splitting, it just knows that it should treat each IDU as something it 
> needs to individually SFRAME and packetize.
>
> On Thu, Nov 19, 2020 at 1:40 PM Sergio Garcia Murillo 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi all,
>
>
>     AVTCORE introducing the topic about the need to specify an
>     agnostic video codec packetization format.
>
>     <https://datatracker.ietf.org/meeting/109/materials/slides-109-avtcore-sframe-rtp-encapsulation-00>
>
>
>     I got an AP for creating an initial draft so it could be reviewed
>     and accepted.
>
>     However, there were two main concerns that we should address in
>     this this group:
>
>       * Historically, avtcore has explicitly designed not to be
>         payload agnostic and  declined to standardized codec agnostic
>         payload formats in  number of cases.  If that is to be
>         changed, needs to be done deliberately.
>       * "independend decodable subframes" like h264 slices, vp8
>         partitions or av1 tiles which allows partial decodability
>         which is mainly aimed for enhancing packet loss resilience.
>
>
>     Spatial layer frames is the minimum we should target as if not it
>     will just prevent SFUs for using SVC codecs. So the question is if
>     we should go deeper and implement lower partitions of the frames
>     or not.
>
>
>     AFAIK, currently, libwertc does not support partial decodability
>     and I personally haven't seen any practical usage of this in the
>     RTC world (while it makes a lot of sense in streaming/broadcasting
>     world), but would like to hear what is the view and experience of
>     the other members of this group. Also note that if we are going to
>     support them on SFrame this will require a greater effort because
>     we will need to explicitly define how the frames must be split
>     before being encrypted y SFrame for *each* possible video codec
>     (h264,h265,vp8,vp9,av1,...).
>
>
>     There was also the question about how/if we should support other
>     codec features like DON/interleaved mode for h264, which I also
>     think we should not support mainly because we are not currently
>     using it on webrtc implementations.
>
>
>     What do you think?
>
>
>     Best regards
>
>     Sergio
>
>
>     -- 
>     Sframe mailing list
>     [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/sframe>
>
>

---

From: Richard Barnes <[email-masked]>
Date: Thu, 19 Nov 2020 17:54:46 -0500

This may be too simplistic, but there's also a non-codec-specific approach
here that occurs to me: Just have the SFrame header have a length field.

I'm imagining a setup where we have the SFrame layer between the encode and
packetization layers, and:
- The encoding/decoding layer produces consumes each frame as a sequence of
IDUs
- The SFrame layer translates between IDUs and SFrame encrypted enits
(SEUs? in any case, each SEU is an encrypted IDU)
- The packetization / depacketization layer packs SEUs into packets

The only thing you need to make that work is (1) a mechanism for the
receiver to understand what chunks of the SEU sequence he has (e.g., fixing
reordering), and (2) a way to unpack SEUs if there can be multiple in a
packet.  It seems like (1) could mostly be a transport assumption.  For
(2), you would just need something like a length field.

As Sergio points out, there is a need for someone to know where the IDU
boundaries are, either at the SFrame layer (if the input is a whole frame),
or at the encode layer (if the encode-SFrame API can talk in terms of
IDUs).  But especially in the latter case, this framework keeps the
codec-specific stuff local to the encode layer, which is codec-specific in
any case.

There is some trade-off here, in that this framework doesn't expose any
encoded information to the SFU.  But ISTM that if you want that
functionality, then (a) you're going to have to have codec awareness
sprinkled all through the stack and (b) you're going to have to be really
careful designing the which-parts-to-encrypt scheme to avoid undermining
your security guarantees.  So I'm generally inclined toward the cleaner
abstraction here.

--Richard

On Thu, Nov 19, 2020 at 5:11 PM Sergio Garcia Murillo <
[email-masked]> wrote:

> is feed in for encryption, but in order to be able to have a working end to
> end solution for webrtc, someone will need to define what and how this IDUs
> are generated and reassembled for each codec if we want to have
> interoperable implementations in different devices.
>
> That process is codec-dependant and I would require quite a lot of effort
> (and also supporting it on the agnostic packetization), so I would prefer
> to have strong arguments in favor of doing it.
>
>
>
>> AVTCORE introducing the topic about the need to specify an agnostic video
>> codec packetization format.
>>
>>

---

From: Bernard Aboba <[email-masked]>
Date: Thu, 19 Nov 2020 15:13:22 -0800

Sergio said:

"You are right regarding that the SFrame layer does not need to know what
is feed in for encryption, but in order to be able to have a working end to
end solution for webrtc, someone will need to define what and how this IDUs
are generated and reassembled for each codec if we want to have
interoperable implementations in different devices."

[BA] The job of an SFrame sender is to encrypt and packetize the bitstream
provided by the encoder.  For SFU to act on the packetization in an optimal
way, some rules have to be followed (such as not packetizing frames from
different layers in the same packet). So the sender/packetizer will have
codec-specific logic, so it can parse the bitstream for meta-data, and
figure out how to do the packetization in a manner appropriate for that
codec.  This could include separately packetizing IDUs (slices/tiles), but
I'm not clear this needs to be required for all use cases.

The SFM does not peer into the payload, it only acts on the meta-data
included in RTP header extensions, so the recovery/forwarding/dropping
decision can be largely codec-independent.

The receiver decrypts and de-packetizes the bitstream, then provides it to
the decoder.  The recovery/decryption/de-packetizion process should also be
codec-independent.

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Fri, 20 Nov 2020 00:15:28 +0100

How to actually packetize the IDUs is not a big deal, you only need to 
have a start and end of IDU bits on the header and rely on increasing 
rtp cseq nums (i.e. no IDU interleaving), but at the end of the day 
someone will have to specify what an IDU in h264 means and what is its 
format (a group of nal units?).

I am afraid that we end up another doing a per-codec packetizer, which 
would be almost the same as just using current packetization formats and 
explain how to use encrypted data with them. And all for a feature that 
we are not current using in webrtc.

Best regards

Sergio

On 19/11/2020 23:54, Richard Barnes wrote:
> This may be too simplistic, but there's also a non-codec-specific 
> approach here that occurs to me: Just have the SFrame header have a 
> length field.
>
> I'm imagining a setup where we have the SFrame layer between the 
> encode and packetization layers, and:
> - The encoding/decoding layer produces consumes each frame as a 
> sequence of IDUs
> receiver to understand what chunks of the SEU sequence he has (e.g., 
> fixing reordering), and (2) a way to unpack SEUs if there can be 
> multiple in a packet.  It seems like (1) could mostly be a transport 
> assumption.  For (2), you would just need something like a length field.
>
> As Sergio points out, there is a need for someone to know where the 
> IDU boundaries are, either at the SFrame layer (if the input is a 
> whole frame), or at the encode layer (if the encode-SFrame API can 
> talk in terms of IDUs).  But especially in the latter case, this 
> framework keeps the codec-specific stuff local to the encode layer, 
> which is codec-specific in any case.
>
> There is some trade-off here, in that this framework doesn't expose 
> any encoded information to the SFU.  But ISTM that if you want that 
> sprinkled all through the stack and (b) you're going to have to be 
> really careful designing the which-parts-to-encrypt scheme to avoid 
> undermining your security guarantees.  So I'm generally inclined 
> toward the cleaner abstraction here.
>
> --Richard
>
> On Thu, Nov 19, 2020 at 5:11 PM Sergio Garcia Murillo 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     You are right regarding that the SFrame layer does not need to
>     know what is feed in for encryption, but in order to be able to
>     have a working end to end solution for webrtc, someone will need
>     to define what and how this IDUs are generated and reassembled for
>     each codec if we want to have interoperable implementations in
>     different devices.
>
>     effort (and also supporting it on the agnostic packetization), so
>     I would prefer to have strong arguments in favor of doing it.
>
>
>
>>     The encoder needs to be aware of any mechanism to generate IDUs
>>     (e.g., slices), and typically each of these IDUs will be handed
>>     up to the consumer individually. So the SFRAME layer doesn't need
>>     to do any splitting, it just knows that it should treat each IDU
>>     as something it needs to individually SFRAME and packetize.
>>
>>         As most of you already know, this morning I made a
>>         presentation in AVTCORE introducing the topic about the need
>>         to specify an agnostic video codec packetization format.
>>
>>         I got an AP for creating an initial draft so it could be
>>         reviewed and accepted.
>>
>>         However, there were two main concerns that we should address
>>         in this this group:
>>
>>             payload agnostic and declined to standardized codec
>>             agnostic payload formats in  number of cases.  If that is
>>             to be changed, needs to be done deliberately.
>>           * Need to define the "minimum decoding unit" or
>>             "independently decodable unit", that SFrame will work with.
>>
>>
>>         Regarding the second one
>>
>>         Spatial layer frames is the minimum we should target as if
>>         not it will just prevent SFUs for using SVC codecs. So the
>>         question is if we should go deeper and implement lower
>>         partitions of the frames or not.
>>
>>
>>         AFAIK, currently, libwertc does not support partial
>>         decodability and I personally haven't seen any practical
>>         usage of this in the RTC world (while it makes a lot of sense
>>         in streaming/broadcasting world), but would like to hear what
>>         is the view and experience of the other members of this
>>         group. Also note that if we are going to support them on
>>         SFrame this will require a greater effort because we will
>>         need to explicitly define how the frames must be split before
>>         being encrypted y SFrame for *each* possible video codec
>>         There was also the question about how/if we should support
>>         other codec features like DON/interleaved mode for h264,
>>         which I also think we should not support mainly because we
>>         are not currently using it on webrtc implementations.
>>
>>
>>         What do you think?
>>
>>
>>         Best regards
>>
>>         Sergio
>>
>>
>>         -- 
>>         Sframe mailing list

---

From: Bernard Aboba <[email-masked]>
Date: Thu, 19 Nov 2020 15:22:46 -0800

Sergio said:

"I am afraid that we end up another doing a per-codec packetizer, which
would be almost the same as just using current packetization formats and
explain how to use encrypted data with them. And all for a feature that we
are not current using in webrtc."

[BA] Some per-codec logic is required.  How else can the metadata be
obtained from the bitstream? But the question is whether it is *required*
to packetize IDUs separately, so they don't share fate.  I'd say this would
depend on the use case. If WebRTC didn't care enough about this to
implement it previously, why should SFrame make a difference?

On Thu, Nov 19, 2020 at 3:15 PM Sergio Garcia Murillo <
[email-masked]> wrote:

> have a start and end of IDU bits on the header and rely on increasing rtp
> cseq nums (i.e. no IDU interleaving), but at the end of the day someone
> will have to specify what an IDU in h264 means and what is its format (a
> group of nal units?).
>
> explain how to use encrypted data with them. And all for a feature that we
> are not current using in webrtc.
>
> Best regards
>
> Sergio
>
>
> I'm imagining a setup where we have the SFrame layer between the encode
> and packetization layers, and:
> - The encoding/decoding layer produces consumes each frame as a sequence
> of IDUs

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Fri, 20 Nov 2020 00:23:16 +0100

If we are able to say something like:

"The encoder is responsible of dividing the encoded byte stream for each 
spatial video frame in 1 or more byte chunks called IDUs before passing 
it to the SFrame layer independently, in such a way that if any of the 
IDUs is not fully received, the rest of the IDUs can be concatenated 
after decrypting and still be partially decodable by the decoder without 
doing any further byte stream manipulation."

Then I would be fine, but I doubt it could be as easy.

Best regards

Sergio

On 20/11/2020 0:13, Bernard Aboba wrote:
> Sergio said:
>
> "You are right regarding that the SFrame layer does not need to know 
> what is feed in for encryption, but in order to be able to have a 
> working end to end solution for webrtc, someone will need to define 
> what and how this IDUs are generated and reassembled for each codec if 
> we want to have interoperable implementations in different devices."
>
> [BA] The job of an SFrame sender is to encrypt and packetize the 
> bitstream provided by the encoder.  For SFU to act on the 
> packetization in an optimal way, some rules have to be followed (such 
> as not packetizing frames from different layers in the same packet). 
> So the sender/packetizer will have codec-specific logic, so it can 
> parse the bitstream for meta-data, and figure out how to do the 
> packetization in a manner appropriate for that codec. This could 
> include separately packetizing IDUs (slices/tiles), but I'm not clear 
> this needs to be required for all use cases.
>
> The receiver decrypts and de-packetizes the bitstream, then provides 
> it to the decoder.  The recovery/decryption/de-packetizion process 
> should also be codec-independent.
>
> On Thu, Nov 19, 2020 at 2:11 PM Sergio Garcia Murillo

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Fri, 20 Nov 2020 00:27:10 +0100

On 20/11/2020 0:22, Bernard Aboba wrote:
> Sergio said:
>
> "I am afraid that we end up another doing a per-codec packetizer, 
> which would be almost the same as just using current packetization 
> formats and explain how to use encrypted data with them. And all for a 
> feature that we are not current using in webrtc."
>
> obtained from the bitstream? But the question is whether it is 
> *required* to packetize IDUs separately, so they don't share fate.  
> I'd say this would depend on the use case. If WebRTC didn't care 
> enough about this to implement it previously, why should SFrame make a 
> difference?

Taking the AV1 dependency descriptor as an example, all of the elements 
are well defined in a codec agnostic way. Being simplistic in the 
example, there is no need for describing how to get the width and height 
of the video frame from an h264 stream, the encoder would be responsible 
for filling the the correct information. Or at least that was the idea 
behind the design.

Best regards

Sergio

---

From: Magnus Westerlund <[email-masked]>
Date: Fri, 20 Nov 2020 16:01:04 +0000

Hi,

If one look at this from the RTP payload level what becomes of potential use and
in some case necessary I think are the following.

The sender will put one or more IDU into independent SFRAMEs and provide the
meta data to it that can include the following:
 - Media Stream + Encoder instance
 - Scalability layer and depdendency information
 - Decoding order
 - Associated Capture/Presentation time
 - timestamp for any consumption related time line (if not redundant)
 - If it represent a switch point (IDR, Intra etc)
 - Will produce output (Video marker bit type, i.e. this IDU(s) will result in
decoded output) (likely for which layers)
 - Reliability level (How necessary the data is, parameter sets is necessary for
any decoding, versus a non-referenced slice). 

The one or more IDUs is a trade-off if between overhead and what benefits there
are to encoded different IDUs in indepdendent SFRAMES. For example encoding
H.26x parameterset NALUs independently depend on how one like to use them, or if
they should be teamed with their related "slice" NALUS, in a STAP style
aggregation of NALUs inside the SFRAME. 

I think the point is to look at this what would be needed to enable the
functionality in the RTP payload format and for the SFUs to be able to correctly
select what SFRAMEs to forward to a particular receiver. 

If some implementation makes is easy for themselves and aggregate more in one
SFRAME that should be possible. It is a trade-off to a large degree between
functionality, cost of doing repair of missing parts, and complexity to
implement. 

There are some basic level which should be very easy to accomplish and never
should be gone below becauase then one is truly throwing away the basic
functionalities of RTP. 

I still think one needs to write a bit of analysis of the different media
encoders functionality and what are shared concepts, what have special quirks
that needs to be thought about. We also I think need to find the terminolgy for
what the different aspects like I listed above really should be called and how
they are mapped to different encoders, in audio, video, tactile, etc. 

I will note that we didn't make much headway in the RTP streams debate before we
actually created the taxonomy of that. People where talking past each others. 

I will also note that a very interesting aspect of the reliability puzzle for
these codecs is what you can determine about a missing RTP packet based on that
you know which SSRC and sequence number that is missing. So if you have a
scalable codec and transmit all the packets on a single SSRC, then a missing
sequence number (detected through the gap) will not tell the receiver if it
needs the packet urgently or not (such a difference between a base layer or the
highest enchancement layer). In some applications that might not matter in
otheres it may. So in the later cases you might want to distribute the layers on
different SSRCs so that one can immediately determine how important the
information is. I think that possibility will exist if this RTP payload format
is done correctly.

Cheers

Magnus Westerlund

---

From: Sergio Garcia Murillo <[email-masked]>
Date: Sun, 22 Nov 2020 23:48:18 +0100

> select what SFRAMEs to forward to a particular receiver.

 From my point of view the that the partial decodability is not 
something that the SFU cares about.

The only RTCP recovery I am aware of that could apply for IDUs is the 
slice loss indication which requires macroblock information, which is a 
way lower information layer than the one SFUs works with (it may not 
even be available if payload is encrypted).

Moreover, the SFU will not rely the SLIs received from the receiving 
endpoint as it has no way of tracking SLIs requests from endpoints and 
map them to already sent ones in order to avoiding overflowing the 
encoder with SLIs requests. So I am afraid that the theoretical benefit 
of partial decoding would be reduced a lot in the typical usage of 
SFrame (i.e. no p2p use case).

Also, before adding support from it, I would like to get real 
implementation feedback as it is currently not implemented in 
(lib)webrtc and I have not heard anynone asking for it ever.

> functionality, cost of doing repair of missing parts, and complexity to

And also complexity to specify.

> actually created the taxonomy of that. People where talking past each others.

Are you proposing to do something like rfc7656 but for video codecs? I 
think that would be great, but I also understand that it would be an 
different draft than the packetization or description header, right?

> information is. I think that possibility will exist if this RTP payload format

While I think this information is very valuable (in fact, it is a must 
have from the SFU perspective), I think it belongs to the SFU 
information header extension part and not the packetization part 
(although they could be implemented together).

Best regards

Sergio

---

From: Bernard Aboba <[email-masked]>
Date: Sun, 22 Nov 2020 15:10:52 -0800

On Nov 22, 2020, at 2:48 PM, Sergio Garcia Murillo <[email-masked]> wrote:
> 
> The only RTCP recovery I am aware of that could apply for IDUs is the slice loss indication which requires macroblock information, which is a way lower information layer than the one SFUs works with (it may not even be available if payload is encrypted).
> 
> Moreover, the SFU will not rely the SLIs received from the receiving endpoint as it has no way of tracking SLIs requests from endpoints and map them to already sent ones in order to avoiding overflowing the encoder with SLIs requests.
> 
> Also, before adding support from it, I would like to get real implementation feedback as it is currently not implemented in (lib)webrtc and I have not heard anynone asking for it ever.

[BA] Although RPSI and SLI Sections are included in several codec documents, I have not seen them implemented. We do not support SLI or RPSI in our SFU. The reasoning is that with RTX and FEC and other messages supported in WebRTC (NAK, PLI, FIR), then SLI and RPSI do not provide incremental value. 

>> So in the later cases you might want to distribute the layers on different SSRCs so that one can immediately determine how important the
>> information is. I think that possibility will exist if this RTP payload format is done correctly.

[BA] We do support MRST with H.264/SVC, and found some advantages in doing so (not having to rewrite sequence numbers in an SFU). However MRST is not supported in VP8, VP9, AV1 and most H.264/SVC implementations, and the performance cost of translation outweighed the forwarding performance advantages.

---

From: Joel Alwen <[email-masked]>
Date: Mon, 23 Nov 2020 14:44:56 +0100

To start with, I think forward secrecy (FS) within epochs can be valuable for
SFrame. (E.g. for long or repeating convo's with the same set of participants.)

Further responses are inline...

> Raphael Robert <[email-masked]> Fri, 20 November 2020 16:23 UTC In other
> scenarios, where a fixed set of participants have reoccurring calls it might
> well be that there won’t be an epoch change for a considerable time (because
> no one joins or leaves the group).
> ...
> Option 3 could be much simpler, namely it would be enough to use a unique 
> session ID to derive a base key that is unique to the session:
> 
> sframe_epoch_secret = MLS-Exporter("SFrame 10 MLS", "", AEAD.Nk) 
> session_secret = HKDF-Expand(sframe_epoch_secret, session_id, AEAD.Nk) 
> sender_base_key[index] = HKDF-Expand(session_secret,
> encode_big_endian(index, 8), AEAD.Nk)
> 
> This is easy to implement and efficient but a little more error-prone since 
> forward secrecy will only be achieved if session_id is unique between 
> sessions. Re-using the same value will not yield forward secrecy.

I might not get what your saying, but if I did then I don't think this gives any
FS, even between sessions, as long as they use the same value for
sframe_epoch_secret regardless of the choices of session_id. (And if they dont
use the same sframe_epoch_secret value then I don't see how the value of
session_id matters for FS.)

Suppose sframe_epoch_secret is used to derive sender_base_key[i]. After session
i is done participants still need to be able to reproduce sframe_epoch_secret if
they want derive sender_base_key[i+1] for session i+1. In that case they could
just as well recompute sender_base_key[i] again too. So no there's no FS for
session i.

Alternatively they could pre-compute all sender_key_base values for potential
future epochs but that seems inelegant to me. Bellow I describe my preferred
solution to precomputing and storing all potentially useful sender_key_base values.

>> On 20 Nov 2020, at 16:17, Richard Barnes <[email-masked]> wrote:
>> 1. SFrame uses the same secret tree as MLS (export at the leaves) 
>> 2. SFrame exports a single secret, then makes its own secret tree 
>> 3. SFrame exports a single secret, then uses a simpler scheme (like the current one)

ATM I'm leaning towards option 2 or 3 (though not with a "simpler" schedule,
just different one more aligned with SFrame use cases).

When it comes to an SFrame epoch (i.e. a time period during which no one
joins/leaves the session) I think there are two constraints that informing which
FS symmetric key schedules are acceptable:

a. Parties shouldn't need to explicitly coordinate who will use which key/nonce
pairs during the epoch.
b. Ciphertexts in the media streams may be dropped and/or delivered out of order.

The tree-based symmetric key derivation hierarchy in MLS was explicitly designed
to accommodate these constraints and while being relatively efficient so, to me,
it looks like a good solution for SFrame as well. Thus option 2 doesnt sound bad
to me.

Having said that, Raphael highlights the case where epochs may be broken up into
sessions (e.g. when the same set of parties on an old call want to resume their
convo in a new call). If that's an important enough scenario to cater too then
the following Option 3 solution might be better. Namely, use one binary tree per
session and string the roots of the trees together in a symmetric ratchet (i.e.
a chain). This would minimize the state participants need to maintain between
sessions (effectively a single secret) while still giving us FS between (and
within) sessions despite them being part of the same epoch. For the sake of
concreteness:

chain_secret[0] = MLS-Exporter("SFrame 10 MLS", "", AEAD.Nk)
chain_secret[i] = HKDF-Expand(chain_secret[i-1], "chain", AEAD.Nk)
session_secret[i] = HKDF-Expand(chain_secret[i], "sesion"+session_id, AEAD.Nk)

and session_secret[i] is the secret at the root of the i-th sessions tree-based
symmetric key schedule. The j-th leaf of the tree initiates a symmetric ratchet
that defines the key/nonce pairs used to encrypt the j-th SFrame stream in the
session. So packet number k of SFrame stream j in session i is encrypted using
the key/nonce pair at position k in symmetric ratchet rooted at the leaf j of
the i-th tree; namely the one rooted at secret session_secret[i]. Yet between
session i and i+1 participants only need to store chain_secret[i+1].

- Joël