Subject: SDP media alignment in SIP

From: Anders Kristensen <[email-masked]>
Date: Mon, 19 Apr 1999 15:45:32 +0100

On the subject of aligning SDP media streams rfc2543 says:

   The caller and callee align their media descriptions so that the nth
   media stream ("m=" line) in the caller's session description
   corresponds to the nth media stream in the callee's description.

As far as I can tell this is not possible in the case where the caller
has a single m= line citing multiple codecs (which is very common) and
the callee supports at least two of them but on different port numbers.
In this case the callee would appear to have to use multiple m= lines in
the response.

Cheers,
Anders

-- 
Anders Kristensen <[email-masked]>,
http://www-uk.hpl.hp.com/people/ak/
Hewlett-Packard Labs, Bristol, UK

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 19 Apr 1999 23:41:53 -0400

> In this case the callee would appear to have to use multiple m= lines in

Why would the callee do this? The port numbers are meant to demultiplex
multiple streams, and within each stream, the payload type in the RTP
demultiplexes the codec possibilities. Thus, the callee would respond
with a single m line, listing the one port number it expects to receive
the stream, along with the two codecs it supports, and their payload
type numbers.

-Jonathan R.
-- 
Jonathan D. Rosenberg                       Lucent Technologies
Member of Technical Staff                   101 Crawfords Corner Rd.
High Speed Networks Research                Holmdel, NJ 07733
FAX: (732) 834-5379                         Rm. 4C-526
EMAIL: [email-masked]
URL: http://www.cs.columbia.edu/~jdrosen

---

From: Anders Kristensen <[email-masked]>
Date: Tue, 20 Apr 1999 10:34:45 +0100

Jonathan Rosenberg wrote:
> demultiplexes the codec possibilities.

It doesn't seem very outlandish to imagine an application using multiple
"media handling" applications or software libraries for handling
different codecs, with each of these apps/libs using separate port
numbers. The last thing you'd want to do is to demultiplex RTP streams
at an application level when all actual media handling takes place
elsewhere.

---

From: eubpepe <[email-masked]>
Date: Tue, 20 Apr 1999 15:39:46 +0200

> at an application level when all actual media handling takes place
I have to agree on this point. Although it probably will be most common
to have all audio-handling in the same multimedia-handler, lots of
SIP-UA's will probably have plug-in capabilities for different session
types, and in that case one will end up in the situation described by
Anders.

Peter Peldan, PhD
Ellemtel Utvecklings AB
[email-masked]

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 20 Apr 1999 09:47:58 -0400

> at an application level when all actual media handling takes place

First off, different media (audio, video, games, etc) are each on
different m lines, which do have different ports. The list of codecs for
a given media stream indicates those codecs that can be used for a
single stream. Having separate libraries is just fine, but this doesn't
mandate different ports. Different applications for each codec is a
different story. The decode process is only one step in the receiver.
After decode, the audio must be placed into a playout buffer, loss must
be compensated for, etc. This is done on a stream by stream basis. So,
if you had each codec as a different app, these apps would need to feed
everyting back into a single, common per-media application anyway.
Remember also that the sender can change codecs dynamically as network
conditions vary. This will introduce interesting synchronization
problems if each codec is a separate app.

-Jonathan R.

---

From: Anders Kristensen <[email-masked]>
Date: Tue, 20 Apr 1999 15:25:39 +0100

Jonathan Rosenberg wrote:
> mandate different ports.

Of course having separate libraries doesn't *mandate* using different
port numbers but it doesn't seem unlikely that a library would want to
"own" its own ports.

> Different applications for each codec is a

Hang on. I'm just talking about the SDP body being returned in a 2xx
response to an INVITE. This describes the media the callee is prepared
to send and/or receive. Not what actually ends up being sent or
received. The callee should be able to list all audio codecs it
understands without telling the caller which one of these to use. I
agree that having multiple subsystems each trying to generate audio
output is not a receipe for anything good.

It's easy enough to fix the spec:
  - remove the alignment requirement completely, or 
  - retain the alignment requirement but say that a single m= line in
    an INVITE may translate into multiple m= lines in the response, or
  - solve the problem by disallowing it, i.e. say the callee MAY NOT
    list those additional codecs handled on different port numbers
    (not nice).

---

From: eubpepe <[email-masked]>
Date: Tue, 20 Apr 1999 18:33:38 +0200

> everyting back into a single, common per-media application anyway.

That won't be necessary if the operating system handles mixing of audio
streams itself. In Win32 it should work all right if the audio modules
uses DirectSound in cooperative mode.

> problems if each codec is a separate app.

I do not see any problems with this as long as the stream is sent to the
correct port: the other module, probably a dll or similar not a
stand-alone application, is already listening to the correct port and
immediatelt starts decoding the audio and feeds it to the sound card.
The other module expecting the old codec, simply stops receiving audio
packets and hence won't send audio to the soundcard. Should work
smoothly?
Peter Peldan, PhD
Ellemtel Utvecklings AB
[email-masked]

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 20 Apr 1999 14:29:06 -0400

> uses DirectSound in cooperative mode.

True. However, after thinking some more, its really important to get the
codecs on the same port. The reason is RTP processing. Lets say the
caller changes codecs. The sequence number space is the same. However,
if these codecs go to different ports (and different apps), the
old-codec app will stop seeing packets, and either think this is loss,
or silence, and in either case do the wrong thing. This is especially
problematic when the codecs change rapidly (as they do with the DTMF
payload format), since this will almost definitely be detected by loss
by the codec-specific application.

Similarly, what about RTCP? Which port would you send RTCP on? Would the
sender effectively need to generate two RTCP streams, one for each
codec? 

So, it seems the best way to handle this is to have a single entity
listening on the port for RTP, and then invoking libraries or other apps
from there to handle specific codecs.

-Jonathan R.

---

From: Peter Peldán <[email-masked]>
Date: Wed, 21 Apr 1999 08:55:47 +0200

> by the codec-specific application.

Why do we need to use the same sequence number space? If the new
codec-stream is to related to new sets of ports isn't it more natural to
start a new RTP/RTCP session with new sequencenumbers? 

With a permanent switch to the new codec, the old RTP-listener would
simply notice that packets stop coming, and for temporary switches like
DTMF there would be a period of silence in the old stream much like when
silence detection/voice activation is used.

> from there to handle specific codecs.

That may be the case, but such a requirement will severely restrict the
possibility of using a plug-in architecture in the UA. One would have to
specify an API between the RTP-module and the multimedia modules. And if
that API should allow true dynamical linking one would probably have to
use COM or Java to implement the RTP-part. (If using a dll -- or other
shared library -- for the RTP-module all multimedia modules using this
method would have to have access to exact the same RTP-dll at build
time, which again would severely restrict the possibility of using a
plug-in architecture)

Why is it so important to line up all media descriptors? Wouldn't it
simply be best to drop that requirement?

Peter Peldan, PhD
Ellemtel Utvecklings AB
[email-masked]

---

From: Jonathan Rosenberg <[email-masked]>
Date: Thu, 22 Apr 1999 00:53:48 -0400

> silence detection/voice activation is used.

The problem is this; when a normal receiver runs out of packets, this
can be either due to loss (in which case it might like to recover), or
due to silence. Now, there is a third case, which is codec change. The
right behavior in all three cases is different.  Should the receiver
initially decide its due to loss, it will try some kind of interoplation
or local repair to fill in the voice. If the lack of packets was due to
codec/port change, this will be mixed with the codec that just arrived
on the other port to the other application, causing strange results. In
the other case, of silence, many systems don't just stop sending audio
to the speaker. They will sometimes generate background noise, since
dead silence is disturbing to users. Again, if the app was wrong, and it
was due to a codec change, this background noise will also be mixed with
the new codec, making for a bizarre effect. The problem is worse here
since its very hard to tell the difference between silence and long-term
codec change.

The basic problem is that SN is used for loss detection, and now the old
codec won't know how whether there is loss, or codec change.

> time, which again would severely restrict the possibility of using a

As has been pointed out, the interfaces for accessing codecs I have seen
are not of the separate application per codec model, but through
libraries and API's, which is just fine.

> simply be best to drop that requirement?

Its best illustrated by an example. Consider a session which is a
classroom session between a student and a teacher. There are three
streams in the session - audio (the professor talking), a video of the
professor's face, and a video of the blackboard. When the professor
calls the student, the SDP contains three m lines, each listing two
codecs:

m=video 1 RTP/AVP 10 11   (camera)
m=video 2 RTP/AVP 10 11   (face)
m=audio 3 RTP/AVP 0 1

 Now, when the response comes, lets say the student wants to receive the
blackboard video on different ports, depending on the codec, as you
suggest, and the so it has two separate m lines. When this SDP is
returned to the professor:

m=video 5 RTP/AVP 10
m=video 6 RTP/AVP 11
m=video 7 RTP/AVP 10
m=audio 8 RTP/AVP 0 1

The professor has a dilemma. He has two video streams, but now three
places to send them. When the student "split" one of the m lines, which
one was it that was split? Are the first two lines the camera, and the
second the face, or the first one the face, and the second two, the
camera?

Basically, alignment of streams is needed since that is the only way to
associate streams in the request with streams in the response.

-Jonathan R.

---

From: Peter Peldán <[email-masked]>
Date: Thu, 22 Apr 1999 09:28:30 +0200

> codec won't know how whether there is loss, or codec change.

Ok in that case I see two solutions:
1. Just let it be and let the user live with these effects if he wants
to use two different audio-modules for handling audio sessions. The
effects would probably not be that big. If the first stream-handler
notices that no packets have arrived during the last 10 seconds it can't
interpolate and fill in, and if it adds background noice the level is
probably so low that it won't disturb the new stream containing audio. 
2. Define a new flag in the RTP-header for telling the receiver there
will be no more packets for a while but do not add background noice.
There is already a flag defined for Voice Activation/silence detection
and this flag would be similar.

> second the face, or the first one the face, and the second two, the

Does it matter? The professors' application could simply pick one of the
ports for the face and one of the ports for the camera. If the student
gets problems with synchronization etc it is really his problem. Either
he stops using two video-modules or he has to live with non-synchronized
streams.

> associate streams in the request with streams in the response.

It seems like all the arguments for disallowing mapping different codecs
to different ports is that the receiver might experience strange
synchroniczation effects. But isn't that up to the user using such a
setup to decide? Why should the protocol spec be restricting this
possibility? The recommended usage should still be to map all codecs to
the same port. In 99% of the audio-sessions only one audio-module would
probably be used and everything would work smoothly.

SUGGESTION: Instead of lining up the session descriptions one could
perhaps require the i=(media title) to be mandatory. If so, one could
easily relate session descriptors to each other by matching the i-field? 
>From the SDP-draft:

A single "i=" field can also be used for each media definition. In media
definitions, "i=" fields are primarily intended for labeling media
streams. As such, they are most likely to be useful when a single
session has more than one distinct media stream of the same media type.

It seems to be a good suggestion to require the i= field in the media
descriptor whenever there are more than one media descriptor of the same
media type, and alignment won't be needed?

Peter Peldan, PhD
Ellemtel Utvecklings AB
[email-masked]

---

From: Anders Kristensen <[email-masked]>
Date: Thu, 22 Apr 1999 14:01:29 +0100

> and this flag would be similar.

Not actually having integrated media into our UA yet, I would have
thought there was a third solution:  Given that the application KNOWS
about media changes (through the SIP signalling) isn't it more natural
for it to TELL the media handling module(s) that a stream is being
terminated rather than to rely on it to figure it out itself through a
lack of RTP packets? In this case the media module should obviously
refrain from generating background noice.

> > libraries and API's, which is just fine.

Well, as I understand it your own UA interfaces to NeVoT through a local
multicast "bus".  My guess is that there is no way you could do
app-level demultiplexing in this setup. Even when you are using a
library it doesn't seem likely that it would provide the sort of
interface required, since this is pretty grungy - I don't *want* to
handle RTP packets in my app.

> he stops using two video-modules or he has to live with non-synchronized

I agree that whatever synchronization effects may occur due to separate
RTP sequence number spaces are basically the receivers problem (it's
preferable to have less-than-perfect communication than no
communication), and he can solve them by removing a media module. 

> probably be used and everything would work smoothly.

And note that it is probably the case that in the vast majority of cases
where an m= line is split there will be no problem at all anyway, since
codec changes are likely the exception rather than the norm (OK,
admitted, this is a conjecture). The most common case of m= splitting
would be when the caller send something like

m=audio 1 RTP/AVP 0 1 2

and the calle responds with something like

m=audio 2 RTP/AVP 0 1
m=audio 3 RTP/AVP 0 2 3

The caller picks one of the codecs listed by the callee and uses the
corresponding port number throughout the call.

> media type, and alignment won't be needed?

Agreed!  Jonathan is right to want to correlate streams in the request
with streams in the response, and the i= field would appear to do the
trick.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Thu, 22 Apr 1999 10:01:32 -0400

Anders Kristensen wrote:

> > probably so low that it won't disturb the new stream containing audio.

Not neccesarily.. background noise levels can often be high (when
calling from a cell phone, for example).

> > and this flag would be similar.

This doesn't help, since the packet with the flag might be lost. So, you
still need to detect silence/codec-change based on lack of data.

> refrain from generating background noice.

Generally, no. The initial SDP exchange in the INVITE/200 OK establishes
capabilities. There is no additional signaling (normally) when a user
changes from one codec to the other among this set. This is necessary
for (1) adaptivity, (2) usage of things like the DTMF codec. With the
DTMF codec, there is no time to do a "logical channel open", signaling
the change in codecs by SIP. By the time you did this, the DTMF would be
over. It is possible to signal long term changes through SIP, by
re-INVITING with just the one codec. But you would still have the
problem that there are many cases where there is no signaling for a
codec change.

> with streams in the response, and the i= field would appear to do the

Not a bad suggestion. There are backwards compatibility issues, though.
The callee has to be sure that the caller supports "splitting" in this
fashion. One could solve this with a new rtpmap attribute. So, the
caller's SDP has i= with a session name, and a new a=rtpmap:allowsplit
attribute, which says whether the callee can split the media line. 

I still think that doing this will introduce serious performance issues,
as we've discussed, though...

-Jonathan R.

---

From: Anders Kristensen <[email-masked]>
Date: Thu, 22 Apr 1999 16:07:41 +0100

> problem that there are many cases where there is no signaling for a

OK, I see.  Incidentally, what does a DTMF codecs rtpmap attribute look
like?

Thinking out loud here, one compromise solution which allows for
multiple audio modules and solves the RTP SN problems might be as
follows: 

  For each stream in play (pun not intended) UAs are free to utilize
  any codec within a single m= line WITHOUT additional SIP signalling.
  When one codec is substituted for another which belongs to a
  different m= line in the peers SDP description it MUST do a SIP
  re-INVITE.

The last MUST should probably be a MAY or SHOULD. The point is to give
guidance on when a UAC switching codecs should make this explicit at the
SIP level through re-INVITEs as this solves (some of) the RTP problems. 
Sending DTMF should never need to result in a SIP re-INVITE. This might
mean DTMF RTP streams has to be treated specially, if not in the spec
then in applications.

> attribute, which says whether the callee can split the media line.

I guess this updated SDP usage would eventually be detailed in an
updated SIP spec with a new SIP version number, should Rough Consensus
ever be reached. In this case an incremented SIP version number might be
sufficient indication of such support.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Thu, 22 Apr 1999 23:41:43 -0400

> OK, I see.  Incidentally, what does a DTMF codecs rtpmap attribute look

I don't think it says anything about it in the current draft. Probably
something like:

m=audio 876 RTP/AVP 76
a=rtpmap:76 dtmf

> mean DTMF RTP streams has to be treated specially, if not in the spec

I think this is more an issue for the dtmf spec. THe current SIP spec,
as it is, allows you to do these re-INVITEs, so what you are suggesting
is more implementation guidance than protocol requirements.  

> sufficient indication of such support.

If we do move forward with this, I'm not sure it needs a new version
number. My proposed new SDP attribute would enable interoperability
without a version change.

-Jonathan R.

---

From: eubpepe <[email-masked]>
Date: Fri, 23 Apr 1999 08:36:07 +0200

> calling from a cell phone, for example).

Well the noice levels we are talking about here actually comes from the
receiver side. It is produced due to lack of received RTP-packets. And a
IP-phone producing high levels of background noice wouldn't be a hit :-)

> attribute, which says whether the callee can split the media line.

Hmm? What about when the caller wants to split the initial INVITE's SDP?
Is it enough to simply send two media descriptors with the same i=
field? A UAS which is unaware of the fact that media streams may be
split won't understand that and may perhaps simply see this as two
separate streams which he is invited to? If the callee then starts
sending audio to both ports we would experience really odd echo effects. 
One solution is of course that one requires that UAS's understand that
using the same i= field means that they belong to the same stream? I do
not see any other simple solution that will ensure backwards
compatibility. But again, this will only harm the user's -- the one that
uses two audio modules -- audio quality. 
> as we've discussed, though...

I don't. If the receiving side simply is clever enough to only add a
very low background noice to the audio when it stops receiving packets
you wouldn't even notice it. And if it adds loud noice, get  a better
audio-module.

Peter Peldan, PhD
Ellemtel Utvecklings AB
[email-masked]

---

From: Igor Slepchin <[email-masked]>
Date: Fri, 23 Apr 1999 12:21:06 -0400

Anders Kristensen wrote:
>  <...>
> The callee should be able to list all audio codecs it
> understands without telling the caller which one of these to use. 

It shouldn't list ALL codecs it understands, but those that form a
subset of codecs listed in INVITE SDP that it understands. In fact,
older versions of SIP draft allowed extending the list of codecs in the
response but this was removed as redundant - the caller already
specifies all the codecs it is willing to use in the INVITE so there is
no point in listing additional codecs that are not going to be used
anyway.

---
Igor Slepchin

---

From: Anders Kristensen <[email-masked]>
Date: Fri, 23 Apr 1999 17:28:23 +0100

> no point in listing additional codecs that are not going to be used

You're right. It doesn't really change anything in the discussion but
thanks for putting the record straight.