Subject: Re: [Cellar] FFV1 slices count

From: Jerome Martinez <[email-masked]>
Date: Thu, 12 May 2016 16:01:31 +0200

Le 27/04/2016 à 02:37, Michael Niedermayer a écrit :
> On Tue, Apr 26, 2016 at 10:01:26PM +0200, Jerome Martinez wrote:
>> Checking FFmpeg code for how FFV1 slices are constrained, I understand that:
>> - FFmpeg encoder tries to find a value compatible with some
>> constraints (minimum 2 rows and 2 columns, rows count must not be >
>> 2x columns count, columns count must not be > 2x rows count, and
>> slices count can not be more than 64). so 25 is acceptable (5x5,
>> OK), 26 is not (13x2 or 2x13 are the only combinations, NOK)
>> - FFmpeg decoder accepts any slices count below 256
>>
>> I don't see any of these constraints in the current FFV1 specifications.
>> I also don't see any reason to have these constraints: cases like 1
>> row and 64 columns could be used somewhere, not really worse than 8
>> rows and 8 columns (and maybe someday an encoder could compare
>> compression performance an select 1x64 for compression performance
>> with some frames). In 10 years, maybe we'll have 8K frames and CPUs
>> with 1024 low-powered cores everywhere, having 32x32 slices (slices
>> of 256x144 pixels) could make sense.
>>
>> On another side, the current specifications don't limit the count of
>> slices, so having more slices than pixels on a row (so we have
>> slices with 0 pixels) is theoretically possible.
>>
>>
>> I would like to have comments on:
>> - do we agree that we don't have any constraint on the slices count
>> in the specification. FFmpeg implementation is a developer decision
>> only and does not have any impact on the specification?
> There is:
>      5.6.1   Restrictions
>      In version 2 and later the maximum slice size in pixels is width*height
>                                                                 ------------
>                                                                      4
>      , unless the frame is smaller or equal
>      352x288 this is to ensure that fast multithreaded decoding is possible.

oops, I missed it.
I see that the 352x288 limit was added in mid-2015, after the "freeze" 
of the bitstream (but FFmpeg code is from 2012)

I am sceptical about the maximum slice size in pixels. the goal is 
multithreaded processing, but we could have an advanced encoder 
optimizing the decoder CPU load instead of pixels count. Example: let's 
take an hypothetical frame with the top half in black. with this slice 
size limitation, 2 threads would do nearly nothing
and 2 threads would be maybe overloaded.
instead of having this scenario for slices (num_h_slice=2 and 
num_v_slice=2):
12
34
we could imagine (num_h_slice=4 and num_v_slice=2)
1111 (slice 1 with slice_width of 4)
2344 (slice 4 with slice_width of 2)
in that case, 1 thread would do nearly nothing and 3 threads would be 
maybe with less overloading than with the limitation, but it is 
forbidden by the slice size limitation. We prevent encoder to imagine 
optimizations we don't currently think about.

Additionally, from my point of view, such restriction is an encoder 
policy, not a bitstream restriction. Actually, I see the restriction in 
the FFmpeg encoder but I don't see a reject of e.g. 353x288 FFV1.2 frame 
with 1 slice with the FFmpeg decoder, looks like the FFmpeg decoder can 
decode it (not tested).
Additionally, I think that "smaller or equal" to 2 numbers is not 
mathematically correct, and in the FFmpeg code:
s->num_v_slices = (avctx->width > 352 || avctx->height > 288 || 
!avctx->slices) ? 2 : 1;
if width is 353 and height is 1 (may be considered as stupid but a spec 
must take care of all cases), count of slices in FFmpeg  is 4 
(num_v_slices of 2, so 2 slices with 0 pixels? BTW the resulting FFV1 
stream from FFmpeg is buggy, I'll write a FFmpeg issue ticket about it) 
and specs are not clear about if it is "smaller or equal to" 352x288.

I propose to put such text in an annex, not part of the specification, 
as an informative recommendation instead of a bitstream restriction.

Proposal:

***
Annex A: Encoder settings
This annex is an infaormal recommendation
To ensure that fast multithreaded decoding is possible, starting version 
2, an encoder SHOULD set by default a slices count of 4 if frame size in 
pixels (width * height) is greater than 101376 else a slices count of 1.
To avoid too much slice header overhead, starting version 2, an encoder 
SHOULD NOT encode more than 4 + floor (width * height / 101376) slices 
per frame (example: maximum of 4 slices below a 352x288 frame, maximum 
of 24 slices for a 1920x1080 HD frame, maximum of 85 slices for a 
3840*2160 UHD-4K frame, maximum of 331 slices for a 7680*4320 UHD-8K frame)
Note: 101376 is the frame size in pixels of a 352x288 frame also known 
as CIF ("Common Intermediate Format") frame size format.
***

(BTW: I see that width and height, the ones relative to the frame, are 
not defined. I'll add some text about it too in order to say that such 
value comes from the container)

>
>> - should we add a constraint about num_h_slices must be less or
>> equal to the frame width, and num_v_slices must be less or equal to
>> frame height?
> I think a restriction stating that "each slice must contain at least
> one sample from each plane" should be added
> Otherwise that may lead to odd special cases where a slice has
> luma but no chroma samples

I propose:

***
num_h_slices MUST be less than ceil (width / h_chroma_subsample)
num_v_slices MUST be less than ceil (height / v_chroma_subsample)
***

so with e.g. a 3x1 4:2:0 frame:
- num_h_slices = ceil (3 / 2) = 2 so we have luma plane 2+1 horizontal 
pixels and chroma plane 1+1 horizontal pixels
- num_v_slices = ceil (1 / 2) = 1 so we have luma plane 1 vertical pixel 
and chroma plane 1 vertical pixel

>> - In the future, we could have "profiles" as constraints for
>> performance reasons, like "profile 1 = 720x576 pixels maximum, 64
>> slices maximum; profile 2 = 1920x1080 pixels maximum, 256 slices
>> maximum", in addition to the core specifications without such
>> arbitrary choices.
>>
>>
>> _______________________________________________
>> Cellar mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/cellar
>>
>
>
> _______________________________________________
> Cellar mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cellar

---

From: Michael Niedermayer <[email-masked]>
Date: Thu, 12 May 2016 17:31:36 +0200

Hi

On Thu, May 12, 2016 at 04:01:31PM +0200, Jerome Martinez wrote:
> I see that the 352x288 limit was added in mid-2015, after the
> "freeze" of the bitstream (but FFmpeg code is from 2012)
> 
> optimizing the decoder CPU load instead of pixels count. Example:

thats not really possible, a encoder does not know how a
decoder is implemented or how much CPU load a specific input would
cause.
Also it makes it harder to implement decoders that are guranteed
to be able to decode a stream.
Knowing that the max slice size is X is something one can write an
implementation against. Knowing that the encoder might have made an
effort to minimize CPU load is not so easy to design a decoder against
that is guranteed to be fast enough for for example real time decoding
of some specific resolution and frame rate.

> let's take an hypothetical frame with the top half in black. with
> this slice size limitation, 2 threads would do nearly nothing

or maybe they would all need roughly the same time,
I think a decoder implemeted as a FPGA or ASIC could fall in that
category

> in that case, 1 thread would do nearly nothing and 3 threads would
> be maybe with less overloading than with the limitation, but it is
> forbidden by the slice size limitation. We prevent encoder to
> imagine optimizations we don't currently think about.

and a decoder designed to support a max slice size of 352x288 would
then run out of memory, so every decoder would need to suport arbitrary
slice sizes
what is gained by the optimization you suggest ?
have you benchmarked it ?

> policy, not a bitstream restriction. Actually, I see the restriction

restricting what is allowed to exist in the bitstream or what an
encoder is allowed to generate is the same IMO

> in the FFmpeg encoder but I don't see a reject of e.g. 353x288
> FFV1.2 frame with 1 slice with the FFmpeg decoder, looks like the
> FFmpeg decoder can decode it (not tested).

FFmpeg has always been designed around the idea of being able to
decode everything that can be decoded especially when its easy, no
matter how broken and invalid the bitstreams are.
Of course FFmpeg should loudly warn about invalid things, and i guess
it fails to do that (didnt check)

> if width is 353 and height is 1 (may be considered as stupid but a
> spec must take care of all cases), count of slices in FFmpeg  is 4
> (num_v_slices of 2, so 2 slices with 0 pixels? BTW the resulting
> FFV1 stream from FFmpeg is buggy, I'll write a FFmpeg issue ticket
> about it) and specs are not clear about if it is "smaller or equal
> to" 352x288.
> 

> I propose to put such text in an annex, not part of the
> specification, as an informative recommendation instead of a
> bitstream restriction.

i think making this just a informative recommendation is a bad idea

if having it as a unconditional bitstream restriction is a disadvantage
for some use case then i suggest we add profiles/levels where each
would have a limit on for example the slice size and number of slices
and other parameters. That way one can implement a decoder that one
can say supports all content compliant to that profile and level.

But if a single unconditional limit could be used that would be simpler
and decoders could then still claim to support for example all content
up to 1080p 60fps

Can you provide an actual testcase where removing the limit improves
decoding speed or something else ?

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Why not whip the teacher when the pupil misbehaves? -- Diogenes of Sinope

---

From: Jerome Martinez <[email-masked]>
Date: Thu, 12 May 2016 20:39:44 +0200

Hi Michael,
Thanks for your feedback.
as you may guess, I disagree a bit, so let's go:

Le 12/05/2016 à 17:31, Michael Niedermayer a écrit :
> then run out of memory,

Currently, it is authorized that a frame of 706x576 has only 4 slices, 
limitation of a slice size of 352x288 in a decoder means that a 706x576 
"legal" frame could not be decoded.
The criticism of the current arbitrary restriction is that it does not 
make a lot of sense until we provide a full set of restrictions (e.g. 
with maximum frame width/height) in order to fit in limited hardware. 
for the moment, a UHD-8K (and more) frame with 4 slices is authorized 
when a 353x288 frame with 1 slice is forbidden. We need more time for 
thinking on such restrictions.

>   so every decoder would need to suport arbitrary
> slice sizes

For the moment, the idea is to avoid a future limitation, not to 
demonstrate it.

> encoder is allowed to generate is the same IMO

I disagree. An encoder can have some policies due to a specific goal 
(proselytism? ;-) ), this does not prevent a decoder to accept different 
cases. an encoder may refuse to encode 4 slices for 352x288 streams, it 
is a policy of the encoder, not what any encoder is allowed to generate. 
Having restrictions in what the developer of an encoder want to generate 
is one thing (there are good reason for doing so, no problem), having 
restriction in the bitstream itself is a decision for all encoders.
Another example if that FFmpeg encoder forces that s->num_h_slices < 
2*s->num_v_slices, but it is not in spec. This is the demonstration that 
there are policies in the encoder (the developer decided of such 
limitation) but this is not in the bitstream spec (the spec does not 
forbid another policy).
Here, the issue is that we arbitrary forbid 1 slice in case of 353x288, 
I think without enough reasons to forbid it (e.g. we accept huge slice 
size), it should only be an encoder policy and a decoder should be able 
to decode it until we have a full set of profiles/levels.

I reverse the questions:
if a decoder is able to decode 4 slices of a 7680x4320 frame (so max 
slice size of 3840x2160), why could it not decode a single slice of 
353x288 when the frame size is 353x288?
why should we forbidden 353x288 with 1 slice if we authorize 4 slices of 
3840x2160?
Do yo have a proof that a FPGA or ASIC could handle my example badly due 
to performance (which ones?) problems?

> it fails to do that (didnt check)

For the moment, the FFmpeg decoder quits when there are more than 256 
slices, and this limit is not in the specification.
This is problematic.
(and the arbitrary choice whatever is the frame size may be a 
limitation, as I wrote in my first email with e.g. CPUs with 1024 
low-powered cores. My proposal of limitations is a informal 
recommendation with a dependency of frame width/height because we don't 
limit the width/height for the moment, so I try to find a good balance 
without forbidding something else in case )

> can say supports all content compliant to that profile and level.

this is what I proposed in the initial email.
I propose to be informative only for FFV1.3 and to implement FFV1.4 with 
profiles/levels based on this informal recommendation, after having 
finished FFV1.3 specifications. forcing a single "profile" (i.e. slices 
pixel slice of width * height / 4) seems a bit hard (forbidding 1 slice 
for 353x288 forever, it is written "In version 2 and later", is not 
future-proof) and early (we did not study the impact of it).

> decoding speed or something else ?

The idea is to permit future development, not to prove it right now.

But let speak of frame byte size:
ffmpeg -y -f lavfi -i testsrc -t 1 -filter:v scale="353:288" -vcodec 
ffv1 -level 3 353x288-4slices.mkv
1st frame is 11048 bytes, file size is 242 kiB
ffmpeg -y -f lavfi -i testsrc -t 1 -filter:v scale="353:288" -vcodec 
ffv1 -level 3 -slices 1 353x288-4slices.mkv
1st frame 10138 bytes (-8%), file size is 222 kiB (-8% too)
(note: modified FFmpeg in order to accept this command)

this example may be considered as stupid (small file, small frame size) 
but it is just a demonstration that forcing some restriction for all 
FFV1 streams is sometimes not optimal.

***

If I don't succeed to convince you about the informal recommendation, 
would the use of "SHOULD" in the current spec (without changing the 
other parts) be acceptable?
 From RFC 2119: SHOULD means "that there may exist valid reasons in 
particular circumstances to ignore a particular item, but the full 
implications must be understood and carefully weighed before choosing a 
different course"
and what is your point of view about max slices count recommendation? do 
we recommend nothing? for the moment, as is or with my proposal about 
num_h_slices and num_v_slices, FFmpeg can not handle "legal" streams 
e.g. UHD-8K frame with 264 (22x12 slices of 350x360 pixels, so more that 
the current obligation of 4 slices for 353x288, it is a realistic 
scenario for an encoder not following FFmpeg encoder limitations and 
built for CPUs having lot of low powered cores).

Jérôme

---

From: Michael Niedermayer <[email-masked]>
Date: Thu, 12 May 2016 23:56:18 +0200

Hi

On Thu, May 12, 2016 at 08:39:44PM +0200, Jerome Martinez wrote:
> Currently, it is authorized that a frame of 706x576 has only 4
> slices, limitation of a slice size of 352x288 in a decoder means
> that a 706x576 "legal" frame could not be decoded.

The current restriction limits the slice size to a quarter of the
resolution for "non tiny" resolutions.
if we remove this restriction decoders must support slices that are
4 times as big. If the decoder runs on a quad core CPU it would
require a 4 times faster one to do that.

> The criticism of the current arbitrary restriction is that it does
> not make a lot of sense until we provide a full set of restrictions

see above, it quadruples the decodable resolution on the currently
commonly used CPUs compared to not having that restriction

i dont oppose doing something else at all, but if high resolution
single slice files get created and they use the context information
from each previous frame, they basically would be forced to be
decoded single threaded and probably not in realtime.

[...]
> (proselytism? ;-) ), this does not prevent a decoder to accept

we misunderstand each other here
what i meant was that the specification restricting what the bitstream
is allowed to look like and the specification restricting what an
encoder can generate is effectively the same

encoder implementations of course can and do have many more
restrictions for all kinds or reasons. Thats an implementation detail

[...]
> why should we forbidden 353x288 with 1 slice if we authorize 4
> slices of 3840x2160?

thats a good question as it shows the problem well i think
many decoders wont be able to decode 7680x4320 in realtime
thus your assumtation you start with already doesnt apply

if you have a really powerfull decoder it of course doesnt matter
 if you have 1 or 4 slices for cases where the decoder is vastly
overpowered. It does matters alot for cases that are at the edge of
being decodeable in realtime

> Do yo have a proof that a FPGA or ASIC could handle my example badly
> due to performance (which ones?) problems?

If you have a decoding pipeline in hw, id assume each pipeline step
would require a fixed and equal number of cycles, its probably
easier to implement that way. So it wouldnt be any faster with
black than with detailed data. Iam not a hw guy, i might be wrong
here of course

But lets look at the reverse, assume the hardware would need more
time to decode highly detailed data.
To be able to decode a specific resolution and slice resolution
the hardware must be able to decode the most detailed and hardest
picture or its users would be kind of unhappy occasionally
now would you make that hw able to decode higher resolutions if they
are all black or simpler?
i guess it depends on how much resolution one gains with real world
videos and what the aditional cost is

[...]

> The idea is to permit future development, not to prove it right now.

i dont see how the restriction would prevent future development
a future version could lift it for files under that future version

> this example may be considered as stupid (small file, small frame
> size) but it is just a demonstration that forcing some restriction
> for all FFV1 streams is sometimes not optimal.

indeed, thanks for looking for that case

> 
> ***
> 
> If I don't succeed to convince you about the informal
> recommendation, would the use of "SHOULD" in the current spec
> (without changing the other parts) be acceptable?
> implications must be understood and carefully weighed before
> choosing a different course"

i think some restriction should remain to ensure that files dont
needlessly end up requiring 4 times more time to decode
that can be a different restriction of course, i dont mind that at all

but if we end up having 1080p files that can be decoded in realtime
and others that need can only be decoded at a quarter of realtime
because they where encoded ignoring a recommandition that could be
a bad experience for the user trying to use these files
i think we should try to avoid that

> and what is your point of view about max slices count
> recommendation? do we recommend nothing? for the moment, as is or
> with my proposal about num_h_slices and num_v_slices, FFmpeg can not
> handle "legal" streams e.g. UHD-8K frame with 264 (22x12 slices of
> 350x360 pixels, so more that the current obligation of 4 slices for
> 353x288, it is a realistic scenario for an encoder not following
> FFmpeg encoder limitations and built for CPUs having lot of low
> powered cores).

honestly i dont know ATM
i think there should be a minimum slice size so that tiny slices like
10x10 arent causing problems but 350x360 does seem a reasonable size

[...]

Dictatorship: All citizens are under surveillance, all their steps and
actions recorded, for the politicians to enforce control.
Democracy: All politicians are under surveillance, all their steps and
actions recorded, for the citizens to enforce control.

---

From: Jerome Martinez <[email-masked]>
Date: Fri, 13 May 2016 12:26:04 +0200

hi,

> require a 4 times faster one to do that.

but a person having a 16-core CPU can complain that you force only 4 
slices instead of 16, and a person having a single-core CPU may complain 
that compression ratio is not as good as it could be without such 
restriction and he gets not value added of 4 slices.
this is the reason I talk of encoder policy: it depends so much of the 
goal of the person who does the encode.

I totally agree that we need profiles/levels, but fact is that we are 
not ready yet about it, having a full set of restrictions for a policy 
with e.g. real time in mind.
In other specifications.

> decoded single threaded and probably not in realtime.

Agreed. But we don't limit the frame width/height so this is not 
complete for guaranteeing something about real time.

> restrictions for all kinds or reasons. Thats an implementation detail

And here I think that the mandatory 4 slices is more an implementation 
detail that something relevant for a specification restriction.

> being decodeable in realtime

But another person could say that in few years we will all have 16-core 
CPU so that we must forbid less than 16 slices, for cases that are at 
the edge of
being decodeable in realtime.

> a future version could lift it for files under that future version

A future encoder could detect black frames and:
- encode 4 slices when a frame is complex
- encode 1 slice (slice_width=2 and slice_height=2) when a black frame 
is detected.

the difference is only few bytes saved (e.g. 439 bytes with 4 slices vs 
190 bytes with 1 slice for a 353x288 frame), but again it is a demo that 
it may be useful sometimes (e.g. if compression ratio is the goal rather 
than decoding speed, or an average between compression and decompression 
speed).

> that can be a different restriction of course, i dont mind that at all

this is the point: this is more a policy (different restrictions could 
be used) instead of a bitstream restriction due to technical things.

> i think we should try to avoid that

I agree. I never said that I want to remove the limitation, just that it 
may be not useful in all cases.
 From the discussion, I think we could agree on "SHOULD" here, because 
it clearly stipulate that "may exist valid reasons in particular 
circumstances to ignore a particular item, but the full implications 
must be understood and carefully weighed before choosing a different 
course" so it fits your goal.
I am against a "MUST".

> 10x10 arent causing problems but 350x360 does seem a reasonable size

but you want to forbid 350x360 slices in some cases (e.g. 350x360 
frames) ;-).
the point is that there is a start of definition of a profile but it is 
not complete. This is the reason I am reluctant to write "MUST" here, 
for the moment (until we get profiles/levels in the next version of FFV1)

I wrote a patch for this subject.

Jérôme

---

From: Michael Niedermayer <[email-masked]>
Date: Fri, 13 May 2016 20:32:26 +0200

On Fri, May 13, 2016 at 12:26:04PM +0200, Jerome Martinez wrote:
> slices instead of 16, and a person having a single-core CPU may
> complain that compression ratio is not as good as it could be
> without such restriction and he gets not value added of 4 slices.
> this is the reason I talk of encoder policy: it depends so much of
> the goal of the person who does the encode.

the restriction is a compromise indeed
the compression rate for real world videos at real world resolutions
should (i hope) be negligible and both 4 core and 16core users have
4 times a´faster decode times with it than with 1s slice videos
without the restriction

the compression difference is indeed as you showed not negligible in
corner cases, like a single frame at 353x288 and iam indeed not happy
about that ...

> 
> I totally agree that we need profiles/levels, but fact is that we
> are not ready yet about it, having a full set of restrictions for a
> policy with e.g. real time in mind.
> In other specifications.
> 
> >
> >

frame width/height is a quite different beast IMHO
like also frame rate it fundamentally is a parameter that says
how much temporal / spatial detail is in a video. double it and the
video becomes twice as hard to decode (approximately)

OTOH the slice segmentation is a ffv1 coding detail, it doesnt
change the actual raw samples

[...]
> But another person could say that in few years we will all have
> 16-core CPU so that we must forbid less than 16 slices, for cases
> that are at the edge of

yes

[...]

> I agree. I never said that I want to remove the limitation, just
> that it may be not useful in all cases.
> From the discussion, I think we could agree on "SHOULD" here,
> because it clearly stipulate that "may exist valid reasons in
> choosing a different course" so it fits your goal.
> I am against a "MUST".

what iam afraid of is that a implementor or even user reads
"SHOULD", tests it,finds 0.1% better compression and it working fine
on his high end box and  then encodes videos that way withiout
undertanding that they wont be decodeable in realtime on more
average systems
and with multiple/many people doing that suddenly we have material
all over the place that many people cannot watch even though the
one creating it did so without actually gaining anything worthwhile

[...]

When you are offended at any man's fault, turn to yourself and study your
own failings. Then you will forget your anger. -- Epictetus