Subject: In Defense of Interleaved Data Formats

From: Eric Fleischman <[email-masked]>
Date: Wed, 5 Mar 1997 15:05:00 -0800

The list of reasons given in Section 5.2 of RFC 1889 as to why RTP
sessions must be segregated by media type reminds me of a discussion I
overheard a few years back. The one guy was saying "Trucks should be
banned from the roadways. They are reckless, prone-to-overturn hazards
belching out huge amounts of pollution. Their compression breaks cause
deafness and their weight fosters the premature breakdown of the road
surfaces..." At this point he lost me because I started to think about
the implications of hauling manure, gravel, or plywood with cars. My
brother-in-law, of course, does exactly that by using a specially
constructed trailer made just for such purposes. But then, he gets stuck
in some pretty pitiful places. 

Mind you, I'm not adverse to cars - I own one myself and use it for my
daily commute. Similarly, I would gladly say "amen" in any praise
meeting about the advantages of payload-segregated RTP traffic. I
readily concur that this approach leads to some marvelously creative
solutions for a wide variety of practical problems.  I am actively
interested in Van Jacobson's and Steve McCanne's "Receiver-driven
Layered Multicast" research. Hey! I really do believe! Don't get me
wrong: I'm all for cars! ... I just want to use my truck when it comes
to hauling gravel. 

Separately encoded media RTP sessions are a marvelous solution to
conferencing applications. They can also be used for "On-Demand"
Streaming applications -- just like my brother-in-law can successfully
haul gravel with his trailer. However, there are many occasions when
"On-Demand Streaming" requires the use of interleaved data formats.
Here's a few:

1.	You can save substantial amount of overhead for sessions
accessed by low speed modems. The argument here is that by interleaving,
only one IP, UDP, and RTP header is used, instead of the X number of
duplicate headers for each of the X media types. (Of course everything
is packetized so this saving accumulates at a constant rate.) The
counter-argument against this is that Van Jacobson's header compression
can convert these extra headers down to the size of pinheads, saving
even more than the original savings. Then the counter-argument to that
counter-argument is that if you are spending that type of overhead
(i.e., on-the-fly compression) to process your "on demand" application
then the number of sessions you can simultaneously support on that
server is detrimentally impacted. (Is this a ploy by high-end machine
builders to sell more product or what?) 

2.	While some "on-demand" content is thrown together by
chimpanzees, most of it is carefully crafted by artisans. For this
reason, most on-demand streaming applications go to extraordinary
lengths to ensure that the multimedia experience presented to the client
is the identical multimedia experience which was designed by the content
creator. Interleaving encourages tight synchronization and offers a
higher degree of probability for preserving the designed experience in
the presence of data loss, network variability, and sunspots.

3.	On-demand media may be composed of many media types. Some of
these are "streaming" media types (e.g., voice, video, and MIDI) while
others are not (e.g., still images, URL flippings, script commands and
executables). "On demand uses" also encourage the innovative invention
of rare media types. Regardless, the net effect is that there will be N
media streams for on-demand usages where N may be quite a handful.
Interleaving permits these many media streams to be presented in a
harmonious, orchestrated and optionally synchronized fashion. While this
is an expansion on the previous point, it also seeks to highlight that
conferencing and "on-demand" uses are quite different when it comes to
range of media which must be coordinated and the variability and
precision demanded for controlling the interaction between these media
types.

4.	Disk-head movement efficiency on the server. The argument here
is that interleaving encourages more efficient disk storage approaches
with superior I/O processing capabilities. This directly effects the
construction of more efficient network streaming approaches. This
efficiency directly translates into server performance including the
ability to support a larger number of simultaneous "On-demand" sessions
by that server.

I own both a truck and a car. I similarly want to use media segregated
streams over RTP in environments where it makes sense to do so and to
use interleaved data streams over RTP in environments where it makes
sense to do so.
One exciting aspect of the current RTSP proposal is that it should
theoretically permit both approaches to be carried over RTP. Traditional
RTP traffic can be carried in the traditional way. Interleaved data
traffic can be supported via encapsulating the media definitions (e.g.,
the information contained within an ASF or RMFF header) which can be
obtained by an RTSP "get" command. A dynamic RTP payload type value can
then be associated with that header information for that session. The
ASF or RMFF data can then be "streamed" via RTP and interpreted
correctly.

---

From: Mark Handley <[email-masked]>
Date: Wed, 05 Mar 1997 19:36:18 -0500

>sessions must be segregated by media type reminds me...
...
>However, there are many occasions when

>accessed by low speed modems.
...
>server is detrimentally impacted.

Note that header compression is not end-to-end, but only between your
end-system and its local terminal server.  It doesn't impact 
multimedia data servers at all.

>2.  ... Interleaving encourages tight synchronization and offers a

Actually this isn't true.  Interleaving gives the network less
oportunity to preserve your audio preferentially from your video if
the network gets loaded (which is usually the right thing to do).

In addition, synchronisation is no easier.  You have all the same
information in separate RTP streams that you have in interleaved
streams.  In either case you need to cope with mismatches in clock
rate between the video stream and a receiving audio device that which
is not clocked at exactly the frequency the manufacturer claims.  Once
you're re-tuning the audio buffering periodically, it makes no
difference whether the streams were transported together or
separately.

>media streams for on-demand usages where N may be quite a handful.

This seems to be an argument *against* interleaving to me...

>construction of more efficient network streaming approaches.

Separate streams lets you store your audio and video on different
discs, and hence *improves* performance in some cases :-)

Besides, if your server isn't doing some form of tuning to available
bandwidth, it's going to be a dinosaur in the long run.  Right now you
might get away with it but the internet won't stand large numbers of
unadaptive streams, and so we'll get schemes like RED, CBQ, WFQ, etc
deployed to defend it.  Once the server is doing bandwidth adaption of
such streams, I don't really believe that the "interleaving impacts
the disc" argument holds well because you'll likely be using layered
codecs anyway.  At the very least, you won't be able to stream
straight from disc to net.

Mark

---

From: Amancio Hasty <[email-masked]>
Date: Thu, 06 Mar 1997 00:24:14 -0800

> discs, and hence *improves* performance in some cases :-)

Well, sort of ...

If you are going to have a server I would imagine that one can do software or
hardware disk stripping , if such is the case then perhaps an interleave
media stream is more efficient than two different media streams.
If one media stream requires more bandwidth than the other then it
may be nice to have separate streams for a better system utilization
by transmitting the prioritized stream. I would imagine that the best
way of handling interleaving vs. separate media streams is to leave 
the choice to the tool and not mandate it as a protocol procedure.

	Cheers,
	Amancio

---

From: Sampo Syreeni <[email-masked]>
Date: Fri, 7 Mar 1997 22:47:52 +0200 (EET)

> multimedia data servers at all.

But I assume some form can be implemented here too, if found advantageous?

> the network gets loaded (which is usually the right thing to do).

But gives a tighter synch with less buffering. And eases up the
requirements for throttle-back in RTP. It also gives the server a more
predictable data stream to work on.

> difference whether the streams were transported together or

How about the case when audio arrives slower than the corresponding 
video? In this case either sizable buffers, fast throttle-back or
frame-discarding abilities are required.

> This seems to be an argument *against* interleaving to me...

Hmm. Keep a separate state for each of the streams/connections and you
have quite a bit of data. So interleaving might ease up the requirements
imposed upon lower protocol layers. How about routing considerations, for 
example?

> discs, and hence *improves* performance in some cases :-)

But how about the basic case where you do  have both the streams
originating from the same server?

> bandwidth, it's going to be a dinosaur in the long run.

Very true. That is a good argument against interleaving. But I'd say at
the present BW adaptation isn't the norm. What we need now is efficiency
from simple servers. But in the future multiple independent data streams
will probably be the norm. At least if stream aggregation doesn't take
place in the lower levels...

Sampo Syreeni (Decoy/dAWN), student, <[email-masked]>