Subject: Re: SDP media alignment in SIP

From: "Stephen Sprunk" <[email-masked]>
Date: Wed, 21 Apr 1999 11:28:19 -0500

>Why do we need to use the same sequence number space? If the new
>codec-stream is to related to new sets of ports isn't it more natural to
>start a new RTP/RTCP session with new sequencenumbers?

It's the same RTP stream on the same ports, just with a different codec
suddenly.

>With a permanent switch to the new codec, the old RTP-listener would
>simply notice that packets stop coming, and for temporary switches like
>DTMF there would be a period of silence in the old stream much like when
>silence detection/voice activation is used.

Because the change wouldn't necessarily be permanent.  To avoid tandem
codings, a voicemail system would want to store the media sent to it in the
original codec on the chance that the listener could  understand that codec;
if not, it could transcode.  In that case, the prompts could be in G.711,
and the messages a mix of G.729, G.723, and others; in an average session,
you could have dozens of codec changes.

In that case, the media for codec 2 may show up before the media for codec 1
has finished playing, etc.  Synchronization is the receiver's job, and
without a consistent sequence number, it's impossible.

And let's not forget applications like RAT which already send RTP streams
with multiple payload types (of the same sample) in each packet.  That is
accepted behavior which RTP receivers are expected to handle.

>> So, it seems the best way to handle this is to have a single entity
>> listening on the port for RTP, and then invoking libraries or other apps
>> from there to handle specific codecs.
>
>That may be the case, but such a requirement will severely restrict the
>possibility of using a plug-in architecture in the UA. One would have to
>specify an API between the RTP-module and the multimedia modules. And if
>that API should allow true dynamical linking one would probably have to
>use COM or Java to implement the RTP-part.

Windows already has such an API for codecs already; it seems to work
extremely well.  Defining a Java interface for RTP playback seems to be a
very simple operation to me; since my Java experience is limited, maybe I'm
missing something.  A UNIX dll interface should be just as simple; Perl is
an excellent example of how easy it is for people to plug-in third-part dlls
to a program.

> (If using a dll -- or other
>shared library -- for the RTP-module all multimedia modules using this
>method would have to have access to exact the same RTP-dll at build
>time, which again would severely restrict the possibility of using a
>plug-in architecture)

I'd write it so that there was one RTP library which probably only had G.711
in it, and any other codec needed (as determined by incoming media) would be
loaded on the fly by that library, with the calling application knowing no
difference.  I don't want to debate application architecture this deeply,
but it doesn't seem like that big of a deal to implement.

Stephen

     |          |         Stephen Sprunk, K5SSS, CCIE #3723
    :|:        :|:        NSA, Network Consulting Engineer
   :|||:      :|||:       14875 Landmark Blvd #400; Dallas, TX
.:|||||||:..:|||||||:.    Pager: 800-365-4578 / 800-901-6078
C I S C O S Y S T E M S   Email: [email-masked]

---

From: Peter Peldán <[email-masked]>
Date: Thu, 22 Apr 1999 08:59:30 +0200

> It's the same RTP stream on the same ports, just with a different codec

Well the point here was exactly that we could allow different ports for
different codecs? (Let me first of all point out that I am NOT
recommending this as the default way of doing thing. I just say that
this may happen when a SIP-UA uses two different audio-modules in the
same session.) In the standard case where we have all codecs listed on
the same port, I agree that it is the same ports, it is the same stream
and the sequence number space is the same. 
However, if we allow different codecs to be mapped to different ports, I
do not see why we absolutely have to use the same sequence number space.
(Besides the reason coming from synchronization problems but that is
perhaps something the user using such a setup have to live with?)

> you could have dozens of codec changes.

Again, this may cause some strange overlap effects. But that is really
up to the user using such a construction to accept or not. If he cannot
live with it, he'd better use one multimedia module mapping all codecs
to the same port.

> accepted behavior which RTP receivers are expected to handle.

That's fine as long as all those codecs are mapped to the same port. If
the receiver did map some of the codecs to port A and some of the codecs
to port B, the RAT application just had to settle with using the
A-codecs or the B-codecs. 

> but it doesn't seem like that big of a deal to implement.

I believe this is the place where we do not understand each other: I do
not simply talk of writing my own audio-module handling all codecs
supported by the windows ACM (Audio Codec Manager). We have done that,
it works fine and I completely agree on everything you are saying. May
point is the following: Say that I do write my own audio module handling
all ACM-codecs, and then e.g Voxware has produced another audio-module
supporting their own super-duper non-ACM codec with superior packet-loss
handling and lots of other features. The user of the SIP-UA with the
multimedia plug-in architecture now wants to use both the standard
audio-module supporting all ACM-codecs as well as the Voxware
audio-module. In this case mapping all the standard ACM-codecs to one
port and the Voxware codecs to another port would be a feasible
solution. 

This issue is not about writing your own audio-modules, it is about
allowing third party vendors to write multimedia session plug-ins that
plug into your SIP-UA. 
> C I S C O S Y S T E M S   Email: [email-masked]

I agree that we probably shouldn't discuss application architecture
here, but since disallowing mapping different codecs to different ports
would restrict the flexibility for a SIP-UA by only allowing one module
for each session type, I believe the question is relevant.

Peter Peldan, PhD
Ellemtel Utvecklings AB
[email-masked]

---

From: Colin Perkins <[email-masked]>
Date: Thu, 22 Apr 1999 11:11:08 +0100

>allowing third party vendors to write multimedia session plug-ins that

Well, the way we do it in RAT is to have our own codec plugin scheme, which
can then call our own inbuilt codecs, ACM codecs and any other codec scheme
you feel like. I don't see a need to send the codecs on different ports to
be able to dispatch packets with a particular PT to a particular decoder
library.

This seems to be a software engineering problem, not a protocol issue...

-- 
Colin Perkins                   Email: [email-masked]
Department of Computer Science  Phone: +44 171 419 3666
University College London       WWW  : http://www.cs.ucl.ac.uk/staff/c.perkins/

---

From: Peter Peldán <[email-masked]>
Date: Thu, 22 Apr 1999 13:10:10 +0200

> be able to dispatch packets with a particular PT to a particular decoder

Well, my point is that this isn't simply a codec-issue. One multimedia
module vendor might have superior jitter-buffer handling etc. I DO NOT
simply want to allow different codecs, I want to allow entire multimedia
modules. That is the difference.

As an example: say that I have two audio-modules. One handles G.711,
GSM, G.723.1 but handles delayed/lost packets and jitter-buffers lousy.
I have another audio-module only handling the codec DX6000 but with
excellent handling of delayed/dropped packets etc. With your suggestion,
I would have to choose between using the standard codecs and lousy
quality otherwise, and the nonstandard codec DX6000 and excellent sound
quality. With my suggestion I may have both options in the same session
even. And, it doesn't cause much problems?
> This seems to be a software engineering problem, not a protocol issue...

Well it is a protocol issue if the protocol spec without good enough
reasons disallows this usage.

Peter Peldan, PhD
Ellemtel Utvecklings AB
[email-masked]

---

From: Anders Kristensen <[email-masked]>
Date: Thu, 22 Apr 1999 13:19:11 +0100

> This seems to be a software engineering problem, not a protocol issue...

It becomes a protocol issue because one size (e.g. RAT) does not fit
all. Our SIP user agent, like Ellemtels, delegates out the
responsibility of handling media to one or more media handling
subsystems, be they applications like RAT or libraries like the JMF. 
For us to use RAT someone has to write a UA module that communicates
with RAT, basically a RAT device driver.

It is perfectly reasonable for a UA to use more than one such device
driver for the same media type and codec at the same time, say, one for
RAT and one for the JMF.  The fact that RAT has its own plugin
architecture helps but it doesn't solve the basic problem that one media
handling subsystem is never going to handle all codecs in the world and
be the best for all of them, and hence it should be possible to several
such media sub-systems together.

-- 
Anders Kristensen <[email-masked]>,
http://www-uk.hpl.hp.com/people/ak/
Hewlett-Packard Labs, Bristol, UK

---

From: Colin Perkins <[email-masked]>
Date: Thu, 22 Apr 1999 14:27:54 +0100

>reasons disallows this usage.

I still don't see this... sure you would have to provide your plugin scheme
at a much lower level, and it may be that current libraries don't give you
access at that level, but that's no reason for breaking the protocol.

Alternatively, if you have out-of-band signalling you can use this to drive
the change of media handling library (as Anders also suggested).

Colin

---

From: Colin Perkins <[email-masked]>
Date: Thu, 22 Apr 1999 15:05:50 +0100

>such media sub-systems together.

So one has to architect the system such that there is a common RTP library,
with other media handling systems layered on top of that. I think (hope!)
we'd agree that RTP is sufficiently simple that a single library could fit
all, and that the complexity is in the playout buffer, codecs, etc. 

Still, that doesn't help if you wish to use and combine the media
frameworks available today, which don't give access at that level.
Although, to be honest I'd probably design the system such that it
rejects calls which cannot be entirely handled by a single framework. 
Having one call being handled by RAT and another by the JMF seems ok,
but trying to have a single call being handled by two frameworks is
possibly not worth the gain?

Basically, I'm arguing that a SIP user agent should have knowledge of what
each media handling framework supports, and negotiate the call such that 
it can be handled by a single framework. 

Colin

---

From: eubpepe <[email-masked]>
Date: Fri, 23 Apr 1999 08:47:40 +0200

> frameworks available today, which don't give access at that level.

That's exactly the point. If we believed that we could succeed in
agreeing on using the same RTP-library or API your suggestion would be
fine. But since there's no hope of that(?), it doesn't make much sense
constructing the protocol for that situation?

> rejects calls which cannot be entirely handled by a single framework.

Why reject them. Simply refrain from using both ports. If a caller sends
you an INVITE with a splitted media descriptor, you simply pick the best
fit and use that port. It is not that you are forced to send data to
both ports. That is only true if you really need to use codecs in both
descriptors.

> it can be handled by a single framework.

That is most likely what will happen in most cases. I just want to have
the possibility to send two media descriptors, and still hope that most
sessions will only use one of them. After all I do not expect the
generic call to use plenty of codec switches. In special cases like the
voicemail or dtmf -- a dying breed -- cases there will be well defined
switches from speech-voicemail and speech-dtmf, but besides that I do
not believe in frequent codec switching to adopt to network congestion
and such (just IMHO).
> Colin

Peter Peldan

---

From: Igor Slepchin <[email-masked]>
Date: Fri, 23 Apr 1999 12:49:19 -0400

See comments below.

> This seems to be a software engineering problem, not a protocol issue...

Agree wholeheartedly. What Peter Peldan described looks like hacking
some 3rd party software for the purposes it was never intended to handle
and I don't see why it should be supported by a protocol, especially if
this introduces all the synchronization problems described in previous
emails.

> constructing the protocol for that situation?

Who is "we" in the above paragraph? There is no need for all SIP UA to
agree on any API like this, at least it should not be part of SIP spec.
Rather, each UA software might define its own plug-in architecture, as
suggested by Colin and implemented in RAT, Netscape and tons of other
products that want to be extendable by 3rd parties. The architecture
would allow 3rd party media handlers to register with the SIP client to
handle a specific payload types, and then a part of SIP client would
have to handle RTP streams and hand over the RTP packets to the plug-ins
registered to handle the payload type specified in the packets. This
allows different media plug-ins to play parts of the same RTP stream. If
you want to standardize such an API to allow for plug-ins reusable
across SIP user agents, you are welcome to submit an internet draft.

> not believe in frequent codec switching to adopt to network congestion

OK, quoting from RTP spec:
For each participant, the session is defined by a particular pair of
destination transport addresses (one network address plus a port pair
for RTP and RTCP). This specifically prohibits sending packets from the
same RTP stream to different port numbers based on the codec used. And I
think that changing this part of RTP spec is a really bad idea, as in
addition to the synchronization problems described in previous emails,
having multiple media clients see only parts of RTP stream will totally
break RTCP statistics - the very heart of RTP.

In addition, even though you can probably configure you SIP client to
make multiple media clients to expect RTP packets on different ports,
there is no RTP stack that I am aware of that would allow sending parts
of the same RTP stream to different destination ports based on codec
used.

Frankly speaking, I don't really see why specifying ALL codecs you can
support is such a big deal that you want to suffer all the problems and
complications described in this and previous emails. After all,
specifying just a subset of caller supported codecs allows for nearly
just as good a conversation in most cases. You can always choose to use
the most efficient codec of those listed in INVITE and return just that
in response SDP.

---
Igor Slepchin

---

From: Peter Peldán <[email-masked]>
Date: Fri, 23 Apr 1999 21:51:01 +0200

> this introduces all the synchronization problems described in previous

This is completely wrong. Our intention is exactly what I described
several times: we want to allow third party vendors to produce
multimedia plug-ins for our SIP-UA without having to complicate the API
with the lower levels of RTP or similar. The API only needs to contain
basically SDP-queries and start/stop session. 

> products that want to be extendable by 3rd parties. 

Ok, we done that, and it works fine thank you.

>The architecture
> allows different media plug-ins to play parts of the same RTP stream.
 
This sounds very complicated and seems like bad design to actually have
the plug-in in the middle of the SIP-UA and the RTP-library. It must be
a much better solution to simply "hang" the plug-ins at the bottom
handling their own RTP-streaming.
And what if we suddely invent a new session type not using RTP at the
bottom. Should we have to specify a new API for that as well. Why
complicate things when it can be so easy?

> If

No I don't, but thanks for the suggestion. Since I do not believe in
this architecture, I leave the API-specification to someone who does.
Give it a try?

> same RTP stream to different port numbers based on the codec used. 

Well these will simply be two different RTP-streams then, won't they? 

>And I

For the RTP stream, the switching will simply look like a block of
silence in the old stream. Doesn't cause much harm to RTCP statistics?
If you would switch codec several times per second, I agree the
statistics would be strange, but not for the normal types of codec
switches one would experience in a normal session.

> of the same RTP stream to different destination ports based on codec

?? This is not up to the RTP-stack. It is decided at higher level. You
simply have two RTP-sessions started and select which one to send
through based on which codec you are using. Works fine with all
RTP-stacks I looked at.

> the most efficient codec of those listed in INVITE and return just that

Well, if I as a caller do not specify all codecs, I would have to -- as
a user -- make two call attempts before knowing if the session could be
setup. Not very desirable for a normal user. On the other hand, if I as
a callee gets an INVITE with a SDP containg codecs in both my plug-ins,
I might choose to only reply with codecs from one set just to improve
quality. But that would be up to the user to decide.

Peter Peldan

---

From: Igor Slepchin <[email-masked]>
Date: Fri, 23 Apr 1999 16:31:24 -0400

> Well these will simply be two different RTP-streams then, won't they?

I think I misunderstood your intentions. I somehow gathered from the
previous discussion that you actually wanted to have various receiving
media clients to handle the same RTP session depending on the payload
type of the RTP packets and have the sender change the destination port
correspondingly. Sorry for the confusion if I was wrong.

> quality. But that would be up to the user to decide.

I agree that all this will work just fine if you make different media
clients handle different RTP sessions. And I also think that it would be
nice to allow the callee to separate media descriptions and it looks
like this will be handled fine by adding i= line plus rtpmap to media
descriptions, just as Peter and Jonathan suggested. 

What I don't see is why there is a problem at the caller side: if the
UAC has two media clients that it wants to handle two different RTP
streams, it can specify them on two different media description lines,
like this:

m=audio 49170 RTP/AVP 0 3 5
m=audio 49180 RTP/AVP 67
a=rtpmap:67 FANCY_CODEC/16000/2

Why would it require two call attempts? Of course, this will look as two
separate media streams but as long as you are not concerned with
synchronization anyway, this seems to work fine.

---
Igor Slepchin

---

From: Peter Peldán <[email-masked]>
Date: Fri, 23 Apr 1999 23:08:30 +0200

Igor Slepchin said
>I think I misunderstood your intentions.

No Igor, you were more correct the first time but anyhow: I believe the
confusion seems to grow on both sides and it may be time to summarize
what we know:

1. Current suggestion in SIP is that we should/must line up all media
descriptors meaning that we cannot have one media-descriptor mapped to
two on the other side, or that we cannot allow one media-session being
described by two media-descriptors. If there are two media-descriptors
in the callers SDP, the caller wants two sessions and the callee should
accept both or send an error.

2. Ellemtel and HP-Labs plug-in architecture makes it desirable to allow
two media-descriptor for the same media-session. We wish to be able to
use two -- or more -- media descriptors for the same session. The callee
could accept the call by only supporting one of them.

What solutions do we see?

Solution 1. 
We allow multiple media descriptors for one session and label every
media-session with the i= field. There's no need to line up media
descriptors.
Advantage: Very flexible solution for using plug-ins. We do not tie up
the SDP to a too strict specification that may prevent future -- yet
unknown -- reasons for splitting session.
Disadvatage: We get backward compatibility problems. The callee's UAS
may not understand that the two media-descriptors belongs to the same
session, and may open two streams. We may get synchronization problems
when RTP-streams are switched between ports/modules.

Solution 2.
We keep the media-descriptor line-up requirement and make no changes.
Advantage: No changes to the current spec. We do not risk the kind of
synchronization problems described earlier.
Disadvatage: We tie up the SDP so strict that it may cause problems for
future session types that prefer to be splitted into multiple media
descriptors for whatever reason. It may cause an unflexible plug-in
architecture (which may be avoided to some extent by using the solution
below)

I believe solution 2 might be the best solution due to the problems of
backwards compatibility only. I am not happy about causing such an
inflexible SDP specification for SIP. Hopefully future session types can
all be fit into one media descriptor? 
The problem with using multiple e.g audio-plug-ins may perhaps be solved
by, in the UAC, automatically make another call with the other plug-in
if the first fails due to non-acceptabel media descriptor? The user will
in that case only notice a slight delay. For the callee there's no
problem since his UAS may choose what plug-in to use based on the
caller's SDP. We will only lose the possibility of switching between
plug-ins during the call, which perhaps noone really wants anyway?

Peter Peldan

---

From: Igor Slepchin <[email-masked]>
Date: Fri, 23 Apr 1999 18:40:46 -0400

Peter Peldán wrote:
> ...
> If there are two media-descriptors in the callers SDP, the caller 
> wants two sessions and the callee should accept both or send an error.

This is not the case. The callee is free to accept one stream but reject
the other by setting port number to 0 in the corresponding m= line. Here
is the quote from Appendix B.1 of the SIP spec:

If the callee wants to neither send nor receive a stream offered by
   the caller, the callee sets the port number of that stream to zero in
   its media description.

The example that follows shows how the callee can accept one of the two
offered video streams and reject the other one. The caller would then
send only those streams that the callee accepted.
 
> could accept the call by only supporting one of them.

I assume that media session here is not the same as an RTP session but
rather all the media coming from a single source, e.g., microphone or
video camera. The same media session (or part of it) might be sent over
multiple RTP sessions for whatever reason, e.g. because different
conference call participants prefer different codecs or some of them
support only audio but not video.

Then what you describe is already possible in the current draft: the
callee just returns 0 as a port number for the media descriptions it
does not want to support. I don't think that m= lines in INVITE are
intended to be treated as either all or none. However, this is true that
the callee may not be able to determine whether multiple m=audio lines
in INVITE's SDP describe same or different media streams.

> ...

The way I read the spec is that the UAS is supposed to be prepared to
receive two RTP streams if it accepts two media description lines.
Please correct me if I'm wrong.

Once again, I am not sure what you mean by switching an RTP stream
between ports. RTP stream is defined by its destination port (plus some
other parameters). What you could do is to have two RTP streams and send
part of the media session (e.g., voice) over one and part (e.g., DTMF)
over the other. This will likely cause synchronization problems for the
callee's media agents but I think this behavior is legal from RTP
standpoint.

---
Igor Slepchin