Subject: RE: In Defense of Interleaved Data Formats

From: Eric Fleischman <[email-masked]>
Date: Wed, 5 Mar 1997 18:46:43 -0800

Your comments appear to be written from a conferencing perspective. I
fear that I haven't helped you to understand the fundamental requirement
of a significant percentage (i.e., probably the majority) of "on demand"
usage. This requirement is that the "multimedia experience" is preserved
independently of what is happening on the network. Thus, if the network
becomes saturated, the worst thing to do would be to drop a given media
stream in order to preserve another stream based on network assumptions
as to which stream may be "more important" (i.e., your suggestion to
drop video but to preserve audio in a congested situation). This
incorrectly presumes that one media type is more important than another
in a "multimedia experience". They all are important if the "experience"
is to be maintained in tact. Now mind you, the content creator may
indeed choose for certain streams to be given preferential treatment but
it is simply impossible to know which stream will be given such
preference in the general case - unlike the conferencing experience.

Rather, the appropriate thing to do in the face of severe network
congestion would be to do one of the following:
*	Cancel the experience outright due to underlying network
problems
*	Delay the experience until such a time as the network can handle
the load
*	Continue the experience with "choppy" reception

Thus, as you can see, interleaving is the most appropriate approach for
preserving a multimedia experience since it is more difficult to manage
an appropriate response with segregated media streams and they are more
prone to distort the experience in the face of network congestion.

It is doubtful that you will be able to understand what I'm talking
about as long as you think as an engineer. I am talking "experience" not
"information". This is a left-brain, right-brain type of dichotomy. To
understand what I'm saying, you have to think as an artist. What we are
talking about is the preservation of an artistic creation across the
network. That this creation may educate or convey critical, essential
information does not diminish the fact that we are talking about
art-forms when we are talking about "on demand" streaming.

I therefore encourage you to re-read my original message from this
point-of-view. Should you do so, then I think that you will find points
2-4 to be very intuitive and obvious. I thank you for your input about
point #1 since that was information about which I was previously
unaware.

Also, you seem to be thinking only about audio and video when I say
"multimedia".  While interesting experiences are indeed created using
only those two media types, I would prefer you to think of audio + video
+ MIDI + URL flipping + streaming text + active background applications
like stock tickers + still images + ....   Why limit ourselves to only
two media types? Certainly the people making streaming "on-demand"
content don't.

I implied in my original message that there were many reasons supporting
interleaving of data, even though I only cited four. Here's another
(and, if you keep disagreeing with me, I'll have to keep giving other
reasons ;-) ):

When the artist creates the multimedia experience, the artist creates
that experience independent from any underlying network concerns (in the
general case) with the sole exception of data rates. The experience is
indeed targeted to be streamed at a given data rate and that rate
constrains how much data is available at any point of the timeline. The
artist doesn't care in many cases whether the experience is carried over
RTP or whether it is carried via TCP, UDP, or HTTP, IPX/SPX, SNA, or
whatever just as long as it arrives as designed.  Now you will perhaps
argue that streaming content can't be carried over connection-oriented
transports like the TCP or HTTP such as were mentioned in the previous
sentence. Experience has demonstrated that this observation may very
well be true for segregated media streams, however, it is emphatically
*not* true for interleaved media streams (given a certain preroll
value). 

Here's the deal: the content creator determines the experience but the
local client's environment determines the mechanism by which that
experience is delivered. When we made our product, therefore, we were
compelled to de-couple the experience from the transport. This means
that any given experience will need to be carried via RTP here but HTTP
there. Why HTTP and why native TCP? Well, it turns out that many users
are very concerned about their firewalls with the degree of concern
varying between companies. Some corporations will permit us to "bore
through" their firewalls via a UDP session, others will permit a single
port via TCP streaming, while others refuse to permit any holes in their
firewalls whatsoever. In the latter case we stream via HTTP if they
permit that protocol to enter their environment, otherwise we are
blocked out.

		-----Original Message-----
		From:	Mark Handley [SMTP:[email-masked]]
		Sent:	Wednesday, March 05, 1997 4:36 PM
		To:	Eric Fleischman
		Cc:	[email-masked]'; [email-masked]'
		Subject:	Re: In Defense of Interleaved Data
Formats 

		Your comments appear to be written from a conferencing
perspective. They don't seem to understand the fundamental requirements
of a significant percentage of "on demand" usage. This requirement is
that the "multimedia experience" is preserved independently of what is
happening on the network. Thus, if the network becomes saturated, the
absolutely wrong thing to do would be to drop a given media stream to
preserve another stream (i.e., your suggestion to drop video but to
preserve audio in a congested situation). Rather, the appropriate thing
to do would be to do one of the following:

			>2.  ... Interleaving encourages tight
synchronization and offers a
			>higher degree of probability for preserving the
designed experience in
			>the presence of data loss, network variability,
and sunspots.

		Actually this isn't true.  Interleaving gives the
network less oportunity to preserve your audio preferentially from your
video if the network gets loaded (which is usually the right thing to
do).
		In addition, synchronisation is no easier.  You have all
the same information in separate RTP streams that you have in
interleaved streams.  In either case you need to cope with mismatches in
clock rate between the video stream and a receiving audio device that
which is not clocked at exactly the frequency the manufacturer claims.
Once you're re-tuning the audio buffering periodically, it makes no
difference whether the streams were transported together or separately.
			>3.	On-demand media may be composed of many
media types. Some of
			>these are "streaming" media types (e.g., voice,
video, and MIDI) while
			>others are not (e.g., still images, URL
flippings, script commands and
			>executables). "On demand uses" also encourage
the innovative invention
			>of rare media types. Regardless, the net effect
is that there will be N
			>media streams for on-demand usages where N may
be quite a handful.

		This seems to be an argument *against* interleaving to
me...
			>4.	Disk-head movement efficiency on the
server. The argument here
			>is that interleaving encourages more efficient
disk storage approaches
			>with superior I/O processing capabilities. This
directly effects the
			>construction of more efficient network
streaming approaches.

		Separate streams lets you store your audio and video on
different discs, and hence *improves* performance in some cases :-)
		Besides, if your server isn't doing some form of tuning
to available bandwidth, it's going to be a dinosaur in the long run.
Right now you might get away with it but the internet won't stand large
numbers of unadaptive streams, and so we'll get schemes like RED, CBQ,
WFQ, etc deployed to defend it.  Once the server is doing bandwidth
adaption of such streams, I don't really believe that the "interleaving
impacts the disc" argument holds well because you'll likely be using
layered codecs anyway.  At the very least, you won't be able to stream
straight from disc to net.
		Mark

---

From: Karl Auerbach <[email-masked]>
Date: Thu, 6 Mar 1997 00:01:27 -0800 (PST)

You are arguing that interleaved representations are necessary to
preserve "the multimedia experience".  And further that the ability of
a non-interleaved, packet-per-media type approach to suppress one of
the media types of the presentation isn't useful because it would
destroy "the multimedia experience".

You seem to be saying that it is better to simply stop the
presentation, in total, rather than try to carry any less than perfect
rendition.

I don't agree.

The choice depends on the type of presentation.

When reality strikes, we will notice that a lot of presentations
happen to be audio and video presentations or conferences (often with
less than stellar camera work or microphone mixing -- hardly an
artistic creation.)

And our own experience has tought us that there are techniques, such
as layered encodings and the like, which can preserve the
intelligibility of the interchange at the cost of some audio quality
or even the video itself.

Interleaved streams will prevent us from making use of those
techniques on those sessions where quality degredation is not only
acceptable, but is desiragle to preserve the information content.

And on those presentations in which one wants to stop the data flow or
postpone it, whether the data is interleaved or not really makes no
difference.  It's just as easy to tell a server to stop sending two
separate RTP streams as it is to tell it to stop one.

And as someone mentioned, the argument for interleaving avoids dealing
with the fact that inside a receiving computer, the audio and video
paths are very different and have different time delays.  So the data
streams will be separated and independently timed for rendition no
matter whether they arrive in the same or different packets.

So even if I were to agree with you that total quality must not be
sacrificed, I still don't agree with you that that leads to the conclusion
that one must have interleaved streams.

> about as long as you think as an engineer.

Speaking as one who spends a lot of my time doing live theatre, I
wouldn't be so quick to deprecate the ability of engineers to
comprehend aesthetic issues.

OK, I'm sitting here pretending that I'm on-stage looking into the
lights and hearing the rustle of the audience while I'm waiting for
the music cue... .... .... Whew, that was one artistic experience!
... And I still don't agree with you.

> ... Now you will perhaps

Of course "streaming content" can be carried over connection-oriented
transports like TCP.  Just not in real-time.

TCP engines are designed with various algorithms to avoid adding to
network congestion.  As such they exhibit potentially long, possibly
difficult to bound, delays.  Unless you know the characteristics of
the TCP engines involved, you can't come up with a completely safe
"preroll" value.  There is always a chance that you will underrun.

Your comments about firewalls are well taken.  However, I have faith
that the firewall folks can figure out how to deal with UDP based
traffic.

			--karl--

---

From: Sampo Syreeni <[email-masked]>
Date: Fri, 7 Mar 1997 23:04:23 +0200 (EET)

On Wed, 5 Mar 1997, Eric Fleischman wrote:

> drop video but to preserve audio in a congested situation).

Indeed. This brings out the most basic difference between hand-crafted
multimedia 'experiences' and the needs of real-time conferencing. The
first carries artistic value as a whole, the second is primarily a means
of communication. As such, in the first case one needs to preserve the
overall experience whereas in the second case getting the message
delivered is of prime importance.

> two media types? Certainly the people making streaming "on-demand"

Because they're the hardest to handle due to their high resource
requirements and the relatively high priority they have among our
perceptual channels, probably. When you can do both audio and video, you
can certainly do the other things you refer to.

> *not* true for interleaved media streams (given a certain preroll

But it is true that as long as you have some minimum guarantee of
maximum latency between the different data streams, you can buffer
multiple streams as well as you can interleaved ones.

Sampo Syreeni (Decoy/dAWN), student, <[email-masked]>