Subject: Re: [payload] #12 (rtp-h265): The max-fps parameter

From: Jonatan Samuelsson <[email-masked]>
Date: Tue, 4 Mar 2014 09:41:12 +0000

Thanks Ye-Kui for the suggested changes. I think they improve the situation to a large extent, but there is one more thing that I would like change; replacing the "MUST" with a "SHOULD" so that it becomes:

"The encoder SHOULD use a picture rate equal to or less than this value.  In cases where the max-fps parameter is absent the encoder is free to choose any picture rate according to the highest level and any signaled optional parameters."

I think this change would be needed in order to not redefine the level definitions of the HEVC specification. A sender that for some reason is not capable of producing lower picture rate (e.g. since the material is pre-encoded) should be allowed to use a picture rate higher than the value of max-fps, but it must then be aware that the receiver might not be capable of adequately present all decoded pictures.

Best Regards Jonatan 

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Wang, Ye-Kui
Sent: den 27 februari 2014 19:20
To: Magnus Westerlund; Geir Sandbakken (geirsand); [email-masked]
Subject: Re: [payload] #12 (rtp-h265): The max-fps parameter

I am with Magnus here. If a decoder cannot DECODE a picture rate allowed by a level, then the decoder is not even a confirming decoder. We usually don't standardize mechanisms for non-confirming decoders, unless there is a very strong reason. 

I think making the following changes (4 places in the first paragraph of the semantics) would resolve the issue, with conceptually good semantics, but the use of the parameter is still practically the same.

-----------------Start of the semantics, with suggested changes--------------------------
max-fps:
The value of max-fps is an integer indicating the maximum picture rate in units of hundreds of pictures per second that can be efficiently received (***CHANGE TO "effectively processed by the receiver"***).  The max-fps parameter MAY be used to signal that the receiver has a constraint in that it is not capable of decoding (***CHANGE TO  "processing"***) video efficiently (***CHANGE TO  "effectively"***) at the full picture rate that is implied by the highest level and, when present, one or more of the parameters max-lsr, max-lps, and max-br.

The value of max-fps is not necessarily the picture rate at which the maximum picture size can be sent, it constitutes a constraint on maximum picture rate for all resolutions.

Informative note: The max-fps parameter is semantically different from max-lsr, max-lps, max-cpb, max-dpb, max-br, max-tr, and max-tc in that max-fps is used to signal a constraint, lowering the maximum picture rate from what is implied by other parameters.

The encoder MUST use a picture rate equal to or less than this value.  In cases where the max-fps parameter is absent the encoder is free to choose any picture rate according to the highest level and any signaled optional parameters.
-----------------End of the semantics, with suggested changes--------------------------

Please express your opinion if this is not acceptable to you. 

BR, YK

-----Original Message-----
From: payload [mailto:[email-masked]] On Behalf Of Magnus Westerlund
Sent: Thursday, February 27, 2014 7:32 AM
To: Geir Sandbakken (geirsand); [email-masked]
Subject: Re: [payload] #12 (rtp-h265): The max-fps parameter

On 2014-02-27 11:18, Geir Sandbakken (geirsand) wrote:
> #12: The max-fps parameter
> 
> Lessons learned from video conferencing using H.264 showed that a lot 
> of equipment designed for 30 fps could not handle 60 fps even at 
> sufficiently low resolution. We do anticipate similar problems when 
> moving towards 120 fps in H.265.  It might not be caused by 
> limitations in the decoder itself,  but that the surrounding media 
> pipeline will not have been properly tested for 120 fps before being put into the wild.
> If we don't allow for a code point to be used by the "restricted"
> equipment, it will be required by the "flexible/proper" to include one. 
> This is what happened with max-fps in H.264 enabling higher frame 
> rates compared to lowering them.
> 

I do have concerns with defining a parameter that allow dynamically sub-setting the profile and levels that has been defined for HEVC. This can also create interoperability issues with any pre-encoded content to a particular profile and level.

The issues with the media pipeline has they been with the reception of the high framerate of packets and forwarding them to the decoder or the playback part, i.e. handling of the decoded picture?

If it is predominately the later it appears very reasonable to have this as an informative parameter. I will not make use of higher FPS than X, if you send me higher than X I will need to reduce that to X or lower before displaying it. That way we are not re-defining the profiles, we are ensuring that we can handle content encoded within the profile and still by taking care of the this in the end of your decoder chain you can protect the rest of the media framework from not yet supported frame-rates.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Services, Media and Network features, Ericsson Research EAB/TXM
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Färögatan 6                 | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

_______________________________________________
payload mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/payload

_______________________________________________
payload mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/payload

---

From: Stephan Wenger <[email-masked]>
Date: Tue, 4 Mar 2014 09:47:30 +0000

Let me push back once more.  Overwriting the limits of the level
definition is common practice in the video conferencing industry, and has
been done since staid industry changed to H.264 (which first introduced
the level concepts in video compression standards widely deployed in video
conferencing).  That SHOULD will be ignored, or overwritten by system
standards.
Is there any technical reason for not allowing overwriting level
limitations in scenarios where there is two way negotiation?
(I would be fine with a SHOULD, or even the removal of max_fps entirely,
in SDP declarative use scenarios.  I only care about O/A scenarios.)
Stephan

On 4.3.14, 9:41, "Jonatan Samuelsson" <[email-masked]>
wrote:

>Thanks Ye-Kui for the suggested changes. I think they improve the
>situation to a large extent, but there is one more thing that I would
>like change; replacing the "MUST" with a "SHOULD" so that it becomes:
>
>"The encoder SHOULD use a picture rate equal to or less than this value.
>In cases where the max-fps parameter is absent the encoder is free to
>choose any picture rate according to the highest level and any signaled
>optional parameters."
>
>I think this change would be needed in order to not redefine the level
>definitions of the HEVC specification. A sender that for some reason is
>not capable of producing lower picture rate (e.g. since the material is
>pre-encoded) should be allowed to use a picture rate higher than the
>value of max-fps, but it must then be aware that the receiver might not
>be capable of adequately present all decoded pictures.
>
>Best Regards Jonatan
>
>I am with Magnus here. If a decoder cannot DECODE a picture rate allowed
>by a level, then the decoder is not even a confirming decoder. We usually
>don't standardize mechanisms for non-confirming decoders, unless there is
>a very strong reason.
>
>I think making the following changes (4 places in the first paragraph of
>the semantics) would resolve the issue, with conceptually good semantics,
>but the use of the parameter is still practically the same.
>
>The value of max-fps is an integer indicating the maximum picture rate in
>units of hundreds of pictures per second that can be efficiently received
>(***CHANGE TO "effectively processed by the receiver"***).  The max-fps
>parameter MAY be used to signal that the receiver has a constraint in
>that it is not capable of decoding (***CHANGE TO  "processing"***) video
>efficiently (***CHANGE TO  "effectively"***) at the full picture rate
>that is implied by the highest level and, when present, one or more of
>the parameters max-lsr, max-lps, and max-br.
>
>The value of max-fps is not necessarily the picture rate at which the
>maximum picture size can be sent, it constitutes a constraint on maximum
>picture rate for all resolutions.
>
>Informative note: The max-fps parameter is semantically different from
>max-lsr, max-lps, max-cpb, max-dpb, max-br, max-tr, and max-tc in that
>max-fps is used to signal a constraint, lowering the maximum picture rate
>from what is implied by other parameters.
>
>The encoder MUST use a picture rate equal to or less than this value.  In
>cases where the max-fps parameter is absent the encoder is free to choose
>any picture rate according to the highest level and any signaled optional
>parameters.
>I do have concerns with defining a parameter that allow dynamically
>sub-setting the profile and levels that has been defined for HEVC. This
>can also create interoperability issues with any pre-encoded content to a
>particular profile and level.
>
>The issues with the media pipeline has they been with the reception of
>the high framerate of packets and forwarding them to the decoder or the
>playback part, i.e. handling of the decoded picture?
>
>If it is predominately the later it appears very reasonable to have this
>as an informative parameter. I will not make use of higher FPS than X, if
>you send me higher than X I will need to reduce that to X or lower before
>displaying it. That way we are not re-defining the profiles, we are
>ensuring that we can handle content encoded within the profile and still
>by taking care of the this in the end of your decoder chain you can
>protect the rest of the media framework from not yet supported
>frame-rates.
>
>Cheers
>
>Magnus Westerlund
>

---

From: Jonatan Samuelsson <[email-masked]>
Date: Tue, 4 Mar 2014 10:42:35 +0000

As far as I know, all other similar parameters in H.264 RTP PL and H.265 RTP PL extends on the levels defined in the video specification in the sense that you are capable of expressing for example "I can decode level 4.1 plus resolutions up to 4K" which can never put a level 4.1 compliant encoder in a difficult situation (i.e. everything in level 4.1 is still allowed). 

If the motivation for max-fps is only for SDP O/A, why isn't the SDP attribute a=framerate sufficient?

Best Regards Jonatan

-----Original Message-----
From: Stephan Wenger [mailto:[email-masked]] 
Sent: den 4 mars 2014 10:48
To: Jonatan Samuelsson; Wang, Ye-Kui; Magnus Westerlund; Geir Sandbakken (geirsand); [email-masked]
Subject: Re: [payload] #12 (rtp-h265): The max-fps parameter

Let me push back once more.  Overwriting the limits of the level definition is common practice in the video conferencing industry, and has been done since staid industry changed to H.264 (which first introduced the level concepts in video compression standards widely deployed in video conferencing).  That SHOULD will be ignored, or overwritten by system standards.
Is there any technical reason for not allowing overwriting level limitations in scenarios where there is two way negotiation?
(I would be fine with a SHOULD, or even the removal of max_fps entirely, in SDP declarative use scenarios.  I only care about O/A scenarios.) Stephan

On 4.3.14, 9:41, "Jonatan Samuelsson" <[email-masked]>
wrote:

>I am with Magnus here. If a decoder cannot DECODE a picture rate 
>allowed by a level, then the decoder is not even a confirming decoder. 
>We usually don't standardize mechanisms for non-confirming decoders, 
>unless there is a very strong reason.
>
>I think making the following changes (4 places in the first paragraph 
>of the semantics) would resolve the issue, with conceptually good 
>semantics, but the use of the parameter is still practically the same.
>
>The value of max-fps is an integer indicating the maximum picture rate 
>in units of hundreds of pictures per second that can be efficiently 
>received (***CHANGE TO "effectively processed by the receiver"***).  
>The max-fps parameter MAY be used to signal that the receiver has a 
>constraint in that it is not capable of decoding (***CHANGE TO  
>"processing"***) video efficiently (***CHANGE TO  "effectively"***) at 
>the full picture rate that is implied by the highest level and, when 
>present, one or more of the parameters max-lsr, max-lps, and max-br.
>
>maximum picture size can be sent, it constitutes a constraint on 
>maximum picture rate for all resolutions.
>
>max-fps is used to signal a constraint, lowering the maximum picture 
>rate from what is implied by other parameters.
>
>The encoder MUST use a picture rate equal to or less than this value.  
>optional parameters.
>can also create interoperability issues with any pre-encoded content to 
>a particular profile and level.
>
>If it is predominately the later it appears very reasonable to have 
>this as an informative parameter. I will not make use of higher FPS 
>than X, if you send me higher than X I will need to reduce that to X or 
>lower before displaying it. That way we are not re-defining the 
>profiles, we are ensuring that we can handle content encoded within the 
>profile and still by taking care of the this in the end of your decoder 
>chain you can protect the rest of the media framework from not yet 
>supported frame-rates.
>
>Cheers
>
>Magnus Westerlund
>

---

From: "Wang, Ye-Kui" <[email-masked]>
Date: Tue, 4 Mar 2014 17:16:23 +0000

Let me pull Tom into the discussion :-) Personally I am OK with the SHOULD language here too as I don't have a good counter argument against Jonatan's comment. 

Tom, would the SHOULD language be fine with you as well (as the proponent of this parameter)? Do you see any technical problem with this?

BR, YK

-----Original Message-----
From: Jonatan Samuelsson [mailto:[email-masked]] 
Sent: Tuesday, March 04, 2014 2:43 AM
To: Stephan Wenger; Wang, Ye-Kui; Magnus Westerlund; Geir Sandbakken (geirsand); [email-masked]
Subject: RE: [payload] #12 (rtp-h265): The max-fps parameter

Best Regards Jonatan

---

From: Tom Kristensen <[email-masked]>
Date: Fri, 7 Mar 2014 08:59:30 +0100

Well, I cannot see a reason for using SHOULD in offer/answer scenarios - as
Stephan argued for earlier here. If there is a limit and a reason for
signalling max-fps due, there is little use waiving the flag with a SHOULD
(which is often ignored, as we know).

I think we should rather remove max-fps for declarative usage (or relax the
MUST to SHOULD there).

-- Tom

On 4 March 2014 18:16, Wang, Ye-Kui <[email-masked]> wrote:

> Let me pull Tom into the discussion :-) Personally I am OK with the SHOULD
> language here too as I don't have a good counter argument against Jonatan's
> comment.
>
> Tom, would the SHOULD language be fine with you as well (as the proponent
> of this parameter)? Do you see any technical problem with this?
>
> BR, YK
>
> As far as I know, all other similar parameters in H.264 RTP PL and H.265
> RTP PL extends on the levels defined in the video specification in the
> sense that you are capable of expressing for example "I can decode level
> 4.1 plus resolutions up to 4K" which can never put a level 4.1 compliant
> encoder in a difficult situation (i.e. everything in level 4.1 is still
> allowed).
>
> been done since staid industry changed to H.264 (which first introduced the
> level concepts in video compression standards widely deployed in video
> in SDP declarative use scenarios.  I only care about O/A scenarios.) Stephan
>
>

-- 
# Cisco                         |  http://www.cisco.com/telepresence/
## [email-masked]  |  http://www.tandberg.com
###                               |  http://folk.uio.no/tomkri/

---

From: Jonatan Samuelsson <[email-masked]>
Date: Fri, 7 Mar 2014 08:20:02 +0000

Thanks Tom for your feedback,

For me to get a better understanding of if it is reasonable to make an exception from the design principle of not re-defining level definitions from the video specification, it would be very valuable with answers to my earlier question:

"If the motivation for max-fps is only for SDP O/A, why isn't the SDP attribute a=framerate sufficient?"

and to Magnus' earlier question:

"The issues with the media pipeline has they been with the reception of the high framerate of packets and forwarding them to the decoder or the playback part, i.e. handling of the decoded picture?"

Best Regards Jonatan

From: Tom Kristensen [mailto:[email-masked]]
Sent: den 7 mars 2014 09:00
To: Wang, Ye-Kui
Cc: Jonatan Samuelsson; Stephan Wenger; Magnus Westerlund; Geir Sandbakken (geirsand); [email-masked]; Tom Kristensen ([email-masked])
Subject: Re: [payload] #12 (rtp-h265): The max-fps parameter

Well, I cannot see a reason for using SHOULD in offer/answer scenarios - as Stephan argued for earlier here. If there is a limit and a reason for signalling max-fps due, there is little use waiving the flag with a SHOULD (which is often ignored, as we know).

I think we should rather remove max-fps for declarative usage (or relax the MUST to SHOULD there).

-- Tom

On 4 March 2014 18:16, Wang, Ye-Kui <[email-masked]<mailto:[email-masked]>> wrote:
Let me pull Tom into the discussion :-) Personally I am OK with the SHOULD language here too as I don't have a good counter argument against Jonatan's comment.

BR, YK

-----Original Message-----
From: Jonatan Samuelsson [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Tuesday, March 04, 2014 2:43 AM
To: Stephan Wenger; Wang, Ye-Kui; Magnus Westerlund; Geir Sandbakken (geirsand); [email-masked]<mailto:[email-masked]>
Subject: RE: [payload] #12 (rtp-h265): The max-fps parameter

Best Regards Jonatan

-----Original Message-----
From: Stephan Wenger [mailto:[email-masked]<mailto:[email-masked]>]
Sent: den 4 mars 2014 10:48
To: Jonatan Samuelsson; Wang, Ye-Kui; Magnus Westerlund; Geir Sandbakken (geirsand); [email-masked]<mailto:[email-masked]>
Subject: Re: [payload] #12 (rtp-h265): The max-fps parameter

On 4.3.14, 9:41, "Jonatan Samuelsson" <[email-masked]<mailto:[email-masked]>>
wrote:

>From: payload [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Wang,
>Ye-Kui
>To: Magnus Westerlund; Geir Sandbakken (geirsand); [email-masked]<mailto:[email-masked]>
>From: payload [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Magnus
>Westerlund
>To: Geir Sandbakken (geirsand); [email-masked]<mailto:[email-masked]>
>SE-164 80 Stockholm, Sweden | mailto: [email-masked]<mailto:[email-masked]>
>[email-masked]<mailto:[email-masked]>
>[email-masked]<mailto:[email-masked]>
>[email-masked]<mailto:[email-masked]>

_______________________________________________
payload mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/payload

--
# Cisco                         |  http://www.cisco.com/telepresence/
## [email-masked]<mailto:[email-masked]>  |  http://www.tandberg.com
###                               |  http://folk.uio.no/tomkri/