Subject: [payload] A few questions about DON in the H.265 payload

From: Jonathan Lennox <[email-masked]>
Date: Thu, 6 Mar 2014 00:18:07 +0000

When reading over the H.265 payload spec to understand DON and marker bits, I realized I had a few questions/comments about DON.

1. DON values are not required to be consecutive.  How does this interact with sprop-depack-buf-nalus?  If, as a receiver, I’m decoding a stream with sprop-depack-buf-nalus == 2, and I receive a packet with DON 1 and then one with DON 3, may I immediately release the packet with DON 1 to the decoder?

2. The definition of sprop-depack-buf-nalus says "When the RTP stream depends on one or more other RTP streams (in this case MST is in use), this parameter MUST be present and the value MUST be greater than 0”.  I don’t think this is sufficient — I think the depended-on stream must have sprop-depack-buf-nalus > 0 as well.  Otherwise, you can’t use the DON values to determine the total decode order across all the MST streams.

3. If you’re using MST but aren’t interested in re-ordering packets at all, how should sprop-depack-buf-nalus be set?  Even 1 would seem to imply that you're doing single-NALU reordering, forcing extra delay onto the decoding process.

4. I think labeling the DON fields “(optional)” in the packet diagrams is confusing, since depending on the value of sprop-depack-buf-nalus it’s either mandatory or forbidden.  I think “(conditional)” would be clearer.

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Thu, 6 Mar 2014 00:55:56 +0000

Thanks Jonathan for the comments. See my responses inline below.

BR, YK

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Jonathan Lennox
Sent: Wednesday, March 05, 2014 4:18 PM
To: [email-masked]; [email-masked]
Subject: [payload] A few questions about DON in the H.265 payload

1. DON values are not required to be consecutive.  How does this interact with sprop-depack-buf-nalus? If, as a receiver, I'm decoding a stream with sprop-depack-buf-nalus == 2, and I receive a packet with DON 1 and then one with DON 3, may I immediately release the packet with DON 1 to the decoder?

[YK]: You just count the number of NAL units buffered to check against the value of sprop-depack-buf-nalus, while the DON values are used to pick which of the buffered NAL units to be first output. This is the answer to your first question. And consequently, the answer is yes to your second question. 

2. The definition of sprop-depack-buf-nalus says "When the RTP stream depends on one or more other RTP streams (in this case MST is in use), this parameter MUST be present and the value MUST be greater than 0".  I don't think this is sufficient - I think the depended-on stream must have sprop-depack-buf-nalus > 0 as well.  Otherwise, you can't use the DON values to determine the total decode order across all the MST streams.

[YK]: Let us use an example. Assume we have two RTP streams carrying the bitstream, the base (i.e. the depended-on) stream s1  and the stream s2 that depends on s1. The value of sprop-depack-buf-nalus signalled for the stream s2 indicates the buffer size needed for de-packetizing both streams s1 and s2, not just s2 - you don't de-packetize a stream itself that depends on another stream. Furthermore, if the stream s1 is not interleaved (i.e. transmission order = decoding order), then the value of sprop-depack-buf-nalus signalled for the stream s1 can be set to 0 as this parameter only specifies the buffer size needed for de-packetization only when the stream s1 is received.

3. If you're using MST but aren't interested in re-ordering packets at all, how should sprop-depack-buf-nalus be set?  Even 1 would seem to imply that you're doing single-NALU reordering, forcing extra delay onto the decoding process.

[YK]: In MST, some reordering of NAL units across RTP streams is always needed. For example, if there is no interleaving within each of the RTP streams, then the value of sprop-depack-buf-nalus needs to be set to be equal to or greater than the minimum value that is sufficient for handling the inter-stream jitter, which can be caused by e.g. the first packet of the highest RTP stream actually comes before the first packet of a "lower" RTP stream. Or are you saying that it should be assumed that the NAL units always flow into the de-packetization buffer in the stream-dependency order (low to high)?

4. I think labeling the DON fields "(optional)" in the packet diagrams is confusing, since depending on the value of sprop-depack-buf-nalus it's either mandatory or forbidden.  I think "(conditional)" would be clearer.

[YK]: Agreed. Magnus made the same comment earlier.

---

From: Jonathan Lennox <[email-masked]>
Date: Thu, 6 Mar 2014 01:28:25 +0000

> Thanks Jonathan for the comments. See my responses inline below.
Mine are inline as well.

> [YK]: You just count the number of NAL units buffered to check against the value of sprop-depack-buf-nalus, while the DON values are used to pick which of the buffered NAL units to be first output.…

This doesn’t follow.  In this example, there are 2 buffered NAL units, which is <= sprop-depack-buf-nalus=2.  Or have I mis-read the algorithm?

To make this more explicit (so we don’t have to worry about < vs. <= cases), let’s say that my sprop-depack-buf-nalus==10, and I receive two packets: one with DON=1, and then one with DON=30.  These are the only two packets I have ever received.  Can I decode the packet with DON=1?

On the current language of the spec, the answer would be no — I have to wait for eight more NALUs to arrive.  (Unless I’ve mis-read the spec.)

I argue that a different definition of sprop-depack-buf-nalus, such that I could, would be more useful — that I can safely decode any packet whose DON value is <= (highest DON received) - (sprop-depack-buf-nalus).  This safely handles scenarios of packet loss and non-consecutive DON values.  Does it break any uses of DON?

> [YK]: Let us use an example. Assume we have two RTP streams carrying the bitstream, the base (i.e. the depended-on) stream s1  and the stream s2 that depends on s1. The value of sprop-depack-buf-na…

But how do I determine the relative placement of the NALUs of the base and dependent streams, without DON values in the base stream?  Consider for instance the example we had in our previous e-mail exchange, where an (SHVC) access unit consists of base VCL NAL units in stream 0, then enhancement VCL NAL units in stream 1, and then a trailing SEI in stream 0.  With DON it’s easy to communicate the total order of the NALUs, but if stream 0 doesn’t have DON values I don’t see how a receiver can reconstruct that.

I think my point is that there may need to be some set of parameters that indicates that a base packet stream carries DON values in its packets, even though its sprop-depack-buf-nalus value is 0.  Since this stream isn’t interleaved, these DONs would only be used for MST reconstruction, and would not be useful for the stream on its own.

> [YK]: In MST, some reordering of NAL units across RTP streams is always needed. For example, if there is no interleaving within each of the RTP streams, then the value of sprop-depack-buf-nalus nee…

You can’t assume that the sender knows how big the inter-stream jitter is, or the relative path delay of the MST streams.  These are network characteristics beyond its visibility, and need to be handled “upstream” of the H.265 decoding process just like standard intra-stream jitter is.  Thus, I was assuming dejittering and alignment had already been handled by the time packets were to be put into the de-packetization buffer.

I suppose I was thinking about putting packets into the buffer in stream-dependency order, or something like it, though I hadn’t thought through the details very clearly.  But I’m certainly thinking of a model where there’s no interleaving *within* any given stream.

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Thu, 6 Mar 2014 21:39:03 +0000

Further response inline below. I removed the overheads.

> [YK]: You just count the number of NAL units buffered to check against the value of sprop-depack-buf-nalus, while the DON values are used to pick which of the buffered NAL units to be first output.…

This doesn't follow.  In this example, there are 2 buffered NAL units, which is <= sprop-depack-buf-nalus=2.  Or have I mis-read the algorithm?

To make this more explicit (so we don't have to worry about < vs. <= cases), let's say that my sprop-depack-buf-nalus==10, and I receive two packets: one with DON=1, and then one with DON=30.  These are the only two packets I have ever received.  Can I decode the packet with DON=1?

On the current language of the spec, the answer would be no - I have to wait for eight more NALUs to arrive.  (Unless I've mis-read the spec.)

[YK]: You have the same understanding as me.

I argue that a different definition of sprop-depack-buf-nalus, such that I could, would be more useful - that I can safely decode any packet whose DON value is <= (highest DON received) - (sprop-depack-buf-nalus).  This safely handles scenarios of packet loss and non-consecutive DON values.  Does it break any uses of DON?

[YK]: I see your point. If there is no gap in DON values, no sharing of DON values (multiple NAL units having the same DON),  and no packet loss, the behavior is the same. I need to think a bit whether your suggestion (exactly as described above) work with cases when there are gaps in DON values or when there are sharing of DON values. However, the general idea of improving the algorithm to make it more friendly to packet losses is good. So let's study this a bit more.

> [YK]: Let us use an example. Assume we have two RTP streams carrying the bitstream, the base (i.e. the depended-on) stream s1  and the stream s2 that depends on s1. The value of sprop-depack-buf-na…

But how do I determine the relative placement of the NALUs of the base and dependent streams, without DON values in the base stream?  Consider for instance the example we had in our previous e-mail exchange, where an (SHVC) access unit consists of base VCL NAL units in stream 0, then enhancement VCL NAL units in stream 1, and then a trailing SEI in stream 0.  With DON it's easy to communicate the total order of the NALUs, but if stream 0 doesn't have DON values I don't see how a receiver can reconstruct that.

I think my point is that there may need to be some set of parameters that indicates that a base packet stream carries DON values in its packets, even though its sprop-depack-buf-nalus value is 0.  Since this stream isn't interleaved, these DONs would only be used for MST reconstruction, and would not be useful for the stream on its own.

[YK]: Good point - I got you. Originally we had a parameter tx-mode that is used together with sprop-depack-buf-nalus > 0 to determine the existence of some DON related fields. During a discussion with Bernard on the usefulness of tx-mode I incorrectly agreed that tx-mode was not useful. Actually tx-mode would be useful to enable non-interleaved base stream with sprop-depack-buf-nalus equal to 0. I'd suggest that we revoke that change for this issue, unless there is a better suggestion.

> [YK]: In MST, some reordering of NAL units across RTP streams is always needed. For example, if there is no interleaving within each of the RTP streams, then the value of sprop-depack-buf-nalus nee…

You can't assume that the sender knows how big the inter-stream jitter is, or the relative path delay of the MST streams.  These are network characteristics beyond its visibility, and need to be handled "upstream" of the H.265 decoding process just like standard intra-stream jitter is.  Thus, I was assuming dejittering and alignment had already been handled by the time packets were to be put into the de-packetization buffer.

I suppose I was thinking about putting packets into the buffer in stream-dependency order, or something like it, though I hadn't thought through the details very clearly.  But I'm certainly thinking of a model where there's no interleaving *within* any given stream.

[YK]: You know currently intra-stream jitter was assumed to be handled separately, not part of the de-packetization process. I think you are right that the handling of inter-stream jitter should also be handled similarly (separately, not part of the de-packetization process). Therefore, we should change the process to enable sprop-depack-buf-nalus to be equal to 0 for any stream as long as no interleaving is used within or across the stream and all the streams it depends. 

BR, YK