Subject: Re: [SIP] Simplify SIP

From: [email-masked]
Date: Sun, 20 Aug 2000 16:33:25 +0530

Agree totally.
I would think its too late now to have something like a "simply simpler
SIP"  right now.
The base draft does make some demands on the implementor trying to be
"fully" compliant to the ABNF.
I do not think that this is really an issue of it being so easy with Perl,
or whatever.
The simplicity or lack of it in  a protocol should not be decided just
because it takes 10 lines in Perl.

The fact is that there is a degree of irritant/baggage like Jonathan
mentioned - but its too late to do away  with them.

Those implementations that don't want to handle all these possibilites,
don't. Return an error or something. As it is, how
often do you see other products sending line-folded, evil-quoted, crazy
LWS, etc message ? The percentage would be
low. If you want to not support them. you would still be supporting most
cases (or so I believe). Most of the leniency
attributes of SIP are to do with "torture scenarios" and typically dont
occur. Those that may occurs are very simple to
do (even in C) (example extra LWSses around)

I do agree however, that if you are in the business of selling SIP
products, you would have to make it as "industry-stable"
as possible - which would therefore include these scenarios. The process of
re-invention could be endless - something
can always be bettered - but after a while, its
investment/time/compatibility vs leanness.

My 2c

Regds
Arjun

Jonathan Rosenberg <[email-masked]> on 08/18/2000 08:23:28 PM

To:   Gethin Liddell <[email-masked]>
cc:   Bertil Engelholm <[email-masked]>,
      [email-masked]

Subject:  Re: [SIP] Simplify SIP

If I had it over again, I would eliminate line folding, case
insensitivity, and a host of other things. But, when the spec was begun,
it was decided that rather than inventing new syntax, we would borrow
from a clearly successful, highly interoperable, well understood
protocol for which a whole lot of code was available. That meant we
inherited the good along with the bad.

Because we must be backwards compatible with rfc2543, there is no choice
but to insist implementations handle line folding and other things (in
reality its not so bad once you construct a tokenizer than knows about
line folding. In this way, it is handled once. My old C code for this
was on the order of 50 lines). We could have rfc2543bis insist on
sending in some canonical form, but it won't help parsers which must
anyway handle the various forms. I suppose eventually those could be
phased out, but there is a fair amount of SIP out there already, so I
don't know that non-canonical form would be gone that soon.

-Jonathan R.

Gethin Liddell wrote:
>
> i agree.
>
> the sip specification is not easy to implement, for the main reason
> that you have to be able to accept the same information in many
> different ways.
>
> conforming to the canonicial form should not be a hardship for anyone
> and will make everyone's implementations faster thanx to a, probably
> quite noticable, efficiency gain.
>
> On Fri, 18 Aug 2000, Bertil Engelholm wrote:
> > Hello,
> >
> > we would like to start a discussion about the possibility of making
> > SIP more simple than today. We realize that it's a little late to
> > make big changes in the protocol at this stage but the change we
> > propose seems quite small. This proposal is our personal proposal
> > and is not in any way reflecting the Ericsson point of view
> > (not yet anyway).
> >
> > We feel that SIP today is unnecessary complex which costs time
> > during development and testing of a SIP parser. It also cost
> > execution time which we can see no reason for. Maybe there are
> > historic reasons why SIP is specified as it is but since the
> > specification does not explain this (which it shouldn't) we
> > can't see the reason for certain constructions. We understand
> > that HTTP has been a great influence when specifying SIP but
> > we have not heard any explanation why.
> >
> > In general we think SIP allows too many optional ways of doing things.
> > Each of these options cost development time, testing time and
> > execution time in every SIP enabled product. And since some SIP
> > UA most likely is small devices with limited processor power, memory
> > size etc. it must be interesting to reduce the amount of code
> > needed to parse the SIP messages.
> > Every option also adds the possibility of introducing bugs which
> > we noticed during the last bakeoff. So by making SIP more
> > simple we think it will be easier to develop SIP enabled products
> > with better quality in shorter time. This would at the end help
> > spreading SIP even faster than today which we hope everyone is
> > interested in doing (at least in this mail group).
> >
> > So what we basically suggest is that only the canonical representation
> > should be used. All other formats should be depricated in the same
> > way as the different ways of terminating the SIP-headers was depricated
> > some time ago. In this way SIP will still be backwards-compatible
> > since everyone has to be able to receive the canonical representation.
> > It will just take some time for everyone to adapt their code so
> > they only send out the canonical representation. After a while it's
> > possible to make a new version of SIP that only allows the canonical
> > form. Then it's up to everyone to decide if they want to support
> > both SIP 2.0 as well as the new version or only the new version.
> >
> > According to the SIP specification (chapter 13.2 in rfc2543bis-01)
> > the canonical representation restricts the SIP syntax in this way :
> >
> > * No short form header fields;
> > * Header field names are capitalized as shown in the rfc;
> > * No white space between the header name and the colon;
> > * A single space after the colon;
> > * Line termination with a CRLF;
> > * No line folding;
> > * No comma separated lists of header values; each must appear as a
> > separate header;
> > * Only a single SP between tokens, between tokens and quoted strings,
> > and between quoted strings; no SP after last token or quoted string;
> > * No LWS between tokens and separators, except as described above for
> > after the colon in header fields;
> > * The To and From header fields always include the < and > delimiters
> > even if the display-name is empty.
> >
> > We can't see any problem in any of these restrictions. Some of them
> > only restricts how many white space characters you are allowed to use
> > so those restrictions can't be any problem to introduce. Why send extra
> > spaces that only adds unnecessary bytes to the messages ?
> >
> > To disallow the compact form of the headers might be a problem but
> > since using the compact form only saves about 70-80 bytes a UA still
> > have to be able to handle the case where the message don't fit in
> > one MTU. If this is a real problem why not define a compact form
> > for _all_ headers in SIP and only use that form ?
> >
> > To capitalize the header field names exactly as they are specified
> > in the RFC does not seem to be a problem either. Why should it be
> > possible to write e.g. Content-Length as cOnTeNt-lEnGtH ?
> >
> > Line folding is also something we can't see any reason for. Since
> > all UA/Servers have to be able to _receive_ any length of a header
> > line why can't they _send_ every header in one line ?
> >
> > To allow a comma separated list in the Via and Record-Route etc.
> > is one of the optional things that seems unnecessary. Especially
> > since it creates longer lines which means you might need line folding.
> > Since all UA that intend to handle Authentication have to use the
> > canonical representation anyway why not always use it ?
> >
> > We realize that it's a little bit late in the SIP development to
> > make changes in the protocol but we feel that these changes are
> > not very big since it's already specified in the RFC.
> >
> > If everyone is happy with todays specification we can live with
> > it too since we have already implemented it. But we feel that it
> > would help SIP to spread faster in the future by making it as
> > simple as possible to parse.
> >
> > So what do you think. Is this proposal something you think
> > is worth introducing or is it better to keep things as it is ?
> >
> > --
> > Bertil Engelholm      [email-masked]
> > Eric Turesson         [email-masked]
> >
> >
> >
> > _______________________________________________
> > SIP mailing list
> > [email-masked]
> > http://lists.bell-labs.com/mailman/listinfo/sip
> --
> Gethin Liddell
> Ubiquity Software Corporation
>
> http://www.ubiquity.net
> mailto:[email-masked]
>
> _______________________________________________
> SIP mailing list
> [email-masked]
> http://lists.bell-labs.com/mailman/listinfo/sip

--
Jonathan D. Rosenberg                       72 Eagle Rock Ave.
Chief Scientist                             First Floor
dynamicsoft                                 East Hanover, NJ 07936
[email-masked]                     FAX:   (973) 952-5050
http://www.cs.columbia.edu/~jdrosen         PHONE: (732) 741-7244
http://www.dynamicsoft.com

_______________________________________________
SIP mailing list
[email-masked]
http://lists.bell-labs.com/mailman/listinfo/sip

_______________________________________________
SIP mailing list
[email-masked]
http://lists.bell-labs.com/mailman/listinfo/sip

---

From: Henning Schulzrinne <[email-masked]>
Date: Sun, 20 Aug 2000 18:20:40 -0400

May I suggest that any proposals that mention simplification that is not
backward-compatible have the burden of proof showing that

- the collective effort spent discussing is less than the collective
effort coding them?

- substantially increases overall call processing speed (not just
parsing speed), where substantially is at least a few months of Moore's
law processor speed-ups?

It's clearly a good idea to send in canonical form (and already
encouraged for various encryption/authentication-using requests), but
the old "be liberal in what you accept" policy has advantages. After
all, these same rules on spacing and line folding have not exactly kept
email and HTTP from becoming popular or from scaling to very large
message volumes.

My guess would be that for a C/assembly-based optimized parser, extra
spaces have essentially no dynamic cost and modest static cost in table
space for the state transitions. Line folding probably has a
one-if-statement/line cost. (The major cost of additional spaces will be
copy costs, but those are only incurred by non-canonical senders. I
expect high-volume senders to be canonic senders, just as for email.)

---

From: Rohan Mahy <[email-masked]>
Date: Wed, 23 Aug 2000 18:10:38 -0700

Hi,

I think the proposal was:  "implementations SHOULD send in canonical 
format", or "sending in canonical format is RECOMMENDED".

These are both backward compatible.  Receiving torture cases would still be 
in the spec.

thanks,
-rohan

---

From: Bertil Engelholm <[email-masked]>
Date: Thu, 24 Aug 2000 12:47:32 +0200

> These are both backward compatible.  Receiving torture cases would still be

You got it. This was exactly what I ment in the proposal.

I have received a number of responses to this mail. There seems to be
three different views to the proposal.

1) Yes, the proposal would simplify my SIP parsing please accept the 
   proposal.

2) The proposal is OK, but since there are so many implementation
   of SIP and the change is not backwards compatible it's to late
   to do the change.

3) Don't change because we wan't the freedom of sending the SIP messages
   in any way we want. It's not many extra lines of code to parse the
   different options.

Group one is not much to comment since it accept the proposed changes.

The Group two comments I don't understand. The change I suggested IS
backwards compatible. What I suggested was to depricate all other
formats
except the cannonical form _when sending_. This means that everyone that 
today fullfills the rfc will be able to receive the canonical form. 
The proposal suggests that only the canonical form SHOULD be used when 
sending which opens up the possibility to remove the other formats in
the 
future. When that is done it will not be backwards compatible but
hopefully 
everyone have adopted to the canonical form at that time so it will not
be 
a big problem. 
This is the exact same thing that was done with the different ways of 
terminating the header lines. Why was that change accepted ? Was that 
change backwards compatible ? 

The third group want the freedom of sending the messages in basically
any
way they want. And they also say that it's no problem of parsing the 
different formats because a few lines of PERL will fix that. First of
all I'm not shure PERL is available on all small devices that will have
to parse the SIP messages. Secondly I think that when you want to send 
things in any way you want you also have to be prepared to receive
things 
in any way and that only complicates things without any gain at all.
In the telecom world, where I come from, we have limited amount of 
bandwidth and processor power but that don't seem to be the case here. 
If a product is more expensive than necessary (due to a more powerful 
processor or/and more memory) isn't that a problem ?  

Finally I just want to comment the PERL code that Brian F. G. Bidulock
sent in. The code looks nice but you actually managed to prove my point
with your code. You failed to take care of comma separated lists and 
the different ways of terminating a header line. So if the canonical
form would have been the only form your code would have worked fine.

So my point is : if you have several optional ways of doing things 
it's easy to forget one of them. So thanks Brian for proving my point.

-- 
Bertil Engelholm
AXE Research and Development        voice : +46 8 727 3499
SIP Security                        Fax   : +46 8 647 8276
S-126 25 Stockholm Sweden           E-mail:
[email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 24 Aug 2000 12:36:08 -0500

Bertil,

Do you even have a `C' complier on that phone of yours?  I suggest you
check out the POSIX regex functions which are as good as PERL.
No POSIX?  Why not get Pete to do it for you on a diode?

Pete Cordell wrote:                           (Fri, 18 Aug 2000 23:49:09)
> 
> An hour - I could do it on a diode in 2 seconds...  And where's your
> handling of comma separation!!!
> 
[snip]

Bertil Engelholm wrote:                     (Thu, 24 Aug 2000 12:47:32)
[snip]
> 
> The third group want the freedom of sending the messages in basically any
> different formats because a few lines of PERL will fix that. First of all
> I'm not shure PERL is available on all small devices that will have to
> parse the SIP messages. Secondly I think that when you want to send
> things in any way you want you also have to be prepared to receive things
> in any way and that only complicates things without any gain at all.  In
> the telecom world, where I come from, we have limited amount of bandwidth
> and processor power but that don't seem to be the case here.  If a
> product is more expensive than necessary (due to a more powerful
> with your code. You failed to take care of comma separated lists and the
> different ways of terminating a header line. So if the canonical form
> would have been the only form your code would have worked fine.

No, it handled line-folding, which is what I was discussing at the time.
It was an example.  See another example which does more below:

> 
> So my point is : if you have several optional ways of doing things it's
> easy to forget one of them. So thanks Brian for proving my point.
>

I didn't ``forget one,'' I just showed line folding!  I don't know what
point you think I helped you prove, 'cause I don't think you have a point.

[snip]

Working along on my PERL example, I have added a few features (in 15
minutes), trying ever so hard not too overlook several.

    #!/usr/bin/perl
    
    \$ = "\n";      # add new-line on print
    undef $/;       # read in whole file
    
    $msg = <>;
    
    ($header, $body) = split(/\r?\n\r?\n/,$msg,2);
    ($start, $header) = split(/\r?\n/,$header,2);
    
    $header =~ s/\r\ni:/\r\nCall-ID:/gi;
    $header =~ s/\r\nm:/\r\nContact:/gi;
    $header =~ s/\r\ne:/\r\nContent-Encoding:/gi;
    $header =~ s/\r\nl:/\r\nContent-Length:/gi;
    $header =~ s/\r\nc:/\r\nContent-Type:/gi;
    $header =~ s/\r\nf:/\r\nFrom:/gi;
    $header =~ s/\r\nt:/\r\nTo:/gi;
    $header =~ s/\r\ns:/\r\nSubject:/gi;
    $header =~ s/\r\nk:/\r\nSupported:/gi;
    $header =~ s/\r\nv:/\r\nVia:/gi;        # or the other way around
    
    $header =~ s/\r\n\s+/ /g; # handle line folding
    
    @headers = split(/(?:^|\r?\n)(\S+?)\s*:\s*/s,$header);
    shift @headers;
    
    for ($i=0;$i<=$#headers;$i=$i+2) {
        $value = $headers[$i+1];
        @parms =
          split(/(?:^|\s*,\s*)((?:(?:[^,"\\]|\\.)|"(?:[^\\"]|\\.)*")*)/sg,
                $value);
        shift @parms;
        for ($j=0;$j<=$#parms;$j=$j+2) {
            push @{$headers{lc $headers[$i]}}, $parms[$j];
            # lc handles capitalization
        }
    }
    
    print $start;
    foreach $k ( keys %headers ) {
        print $k.' : '.join(" , \n\t",@{$headers{$k}});
    }
    printf "\n".$body;

On your original beef list, here are the things
that the EXAMPLE handles:

Bertil Engelholm wrote:                     (Fri, 18 Aug 2000 10:56:16)
[snip]
> * No short form header fields;   
Short form fields works.
> * Header field names are capitalized as shown in the rfc;   
Case insensitive.
> * No white space between the header name and the colon;   
Handles whitespace.
> * A single space after the colon;   
Handles multiple spaces.
> * Line termination with a CRLF;   
Handles just LF.
> * No line folding;   
Unfolds lines.
> * No comma separated lists of header values; each must appear as a
Parses out comma separates lists.

I'll leave the last three as an exercise to you, Bertil.  (Notice that it
prints the message back out with spaces all over the place, folding lines,
all multiple headers concatenated into comma-separated lists. It can even
handle it's own ugly output ;)

>   even if the display-name is empty. 
[snip]

Here's my test message:

    INVITE sip:[email-masked] SIP/2.0
    Via: SIP/2.0/UDP 169.130.12.5
    From: <sip:[email-masked]>
    To: T. A. Watson <sip:[email-masked]>
    Call-ID: [email-masked]
    Cseq: 1 INVITE
    Content-Length: 829
    Encryption: PGP version=2.6.2,encoding=ascii,token=" \",yuck",
        thistoken="Better"

    hQEMAxkp5GPd+j5xAQf/ZDIfGD/PDOM1wayvwdQAKgGgjmZWe+MTy9NEX8O25Red
    h0/pyrd/+DV5C2BYs7yzSOSXaj1C/tTK/4do6rtjhP8QA3vbDdVdaFciwEVAcuXs
    ODxlNAVqyDi1RqFC28BJIvQ5KfEkPuACKTK7WlRSBc7vNPEA3nyqZGBTwhxRSbIR
    RuFEsHSVojdCam4htcqxGnFwD9sksqs6LIyCFaiTAhWtwcCaN437G7mUYzy2KLcA
    zPVGq1VQg83b99zPzIxRdlZ+K7+bAnu8Rtu+ohOCMLV3TPXbyp+err1YiThCZHIu
    X9dOVj3CMjCP66RSHa/ea0wYTRRNYA/G+kdP8DSUcqYAAAE/hZPX6nFIqk7AVnf6
    IpWHUPTelNUJpzUp5Ou+q/5P7ZAsn+cSAuF2YWtVjCf+SQmBR13p2EYYWHoxlA2/
    GgKADYe4M3JSwOtqwU8zUJF3FIfk7vsxmSqtUQrRQaiIhqNyG7KxJt4YjWnEjF5E
    WUIPhvyGFMJaeQXIyGRYZAYvKKklyAJcm29zLACxU5alX4M25lHQd9FR9Zmq6Jed
    wbWvia6cAIfsvlZ9JGocmQYF7pcuz5pnczqP+/yvRqFJtDGD/v3s++G2R+ViVYJO
    z/lxGUZaM4IWBCf+4DUjNanZM0oxAE28NjaIZ0rrldDQmO8V9FtPKdHxkqA5iJP+
    6vGOFti1Ak4kmEz0vM/Nsv7kkubTFhRl05OiJIGr9S1UhenlZv9l6RuXsOY/EwH2
    z8X9N4MhMyXEVuC9rt8/AUhmVQ==
    =bOW+

Here's the output:

    INVITE sip:[email-masked] SIP/2.0
    call-id : [email-masked]
    content-length : 829
    cseq : 1 INVITE
    encryption : PGP version=2.6.2 , 
        encoding=ascii , 
        token=" \",yuck" , 
        thistoken="Better"
    from : <sip:[email-masked]>
    to : T. A. Watson <sip:[email-masked]>
    via : SIP/2.0/UDP 169.130.12.5

    hQEMAxkp5GPd+j5xAQf/ZDIfGD/PDOM1wayvwdQAKgGgjmZWe+MTy9NEX8O25Red
    h0/pyrd/+DV5C2BYs7yzSOSXaj1C/tTK/4do6rtjhP8QA3vbDdVdaFciwEVAcuXs
    ODxlNAVqyDi1RqFC28BJIvQ5KfEkPuACKTK7WlRSBc7vNPEA3nyqZGBTwhxRSbIR
    RuFEsHSVojdCam4htcqxGnFwD9sksqs6LIyCFaiTAhWtwcCaN437G7mUYzy2KLcA
    zPVGq1VQg83b99zPzIxRdlZ+K7+bAnu8Rtu+ohOCMLV3TPXbyp+err1YiThCZHIu
    X9dOVj3CMjCP66RSHa/ea0wYTRRNYA/G+kdP8DSUcqYAAAE/hZPX6nFIqk7AVnf6
    IpWHUPTelNUJpzUp5Ou+q/5P7ZAsn+cSAuF2YWtVjCf+SQmBR13p2EYYWHoxlA2/
    GgKADYe4M3JSwOtqwU8zUJF3FIfk7vsxmSqtUQrRQaiIhqNyG7KxJt4YjWnEjF5E
    WUIPhvyGFMJaeQXIyGRYZAYvKKklyAJcm29zLACxU5alX4M25lHQd9FR9Zmq6Jed
    wbWvia6cAIfsvlZ9JGocmQYF7pcuz5pnczqP+/yvRqFJtDGD/v3s++G2R+ViVYJO
    z/lxGUZaM4IWBCf+4DUjNanZM0oxAE28NjaIZ0rrldDQmO8V9FtPKdHxkqA5iJP+
    6vGOFti1Ak4kmEz0vM/Nsv7kkubTFhRl05OiJIGr9S1UhenlZv9l6RuXsOY/EwH2
    z8X9N4MhMyXEVuC9rt8/AUhmVQ==
    =bOW+

-- 
Brian F. G. Bidulock
[email-masked]

---

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 24 Aug 2000 13:49:09 -0400

Even without regexp, in C or Perl, handling multiple spaces and line
folding is a minor inconvenience, on the order of at most a few tens of
lines of code and a few extra states in the parser (e.g., for handling
"header" and ":" as separate states instead of comparing just
"header:").

I'm not sure that robustness would be served by anybody relying on the
canonical representation. In some sense, you *want* a significant
fraction of the implementations to use non-canonical coding since this
makes it much more likely that you'll discover interoperability problems
earlier rather than after the beta test period. Finger-pointing at that
point is not particularly productive. "You mean you rejected the 911
call since the call had an extra space after the token?"

Thus, I don't see what the SHOULD buys you beyond surprises in the field
that take far longer to debug than spending a few hours extra on
building a robust parser to begin with.

All the noises about improved efficiency remain, so far, unproven
assertions.

-- 
Henning Schulzrinne   http://www.cs.columbia.edu/~hgs

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 24 Aug 2000 12:58:49 -0500

Henning,

Souce code for `formail' might give someone an idea: I haven't seen it
munge an e-mail header line or body, ever...  It even get's loosey goosey
with the Content-Length when given the '-y' flag.  It's written in `C',
portable to assembler, or diodes, I'm sure!

--Brian

Henning Schulzrinne wrote:                 (Thu, 24 Aug 2000 13:49:09)

-- 
Brian F. G. Bidulock
[email-masked]