Subject: [SIP] white space and SIP

From: Jonathan Rosenberg <[email-masked]>
Date: Fri, 18 Aug 2000 10:48:12 -0400

Gethin Liddell wrote:
> 
> For SIP to be efficient and quick, as i'm sure you'll agree it must be,
> it MUST be possible to parse in a message with out having to parse each
> individual header.

Absolutely. This is what I was trying (and not at all succeeding in
saying) when I said:
> so that you can detect headers without parsing the headers
> > themselves.

> 
> Ignoring the \CR and \LF, it is possible to get the structure of the
> messages and the headers split up by looking for the combination of
> CR, LF and white space indentation. great, this can be done quickly by
> simple byte-by-byte comparison.
> 
> This all goes out the window however when we allow, or expect, that in
> `special cases' a `\' at the end of the line actually marks another
> form of continuation.
> 
> for example, parsing the fast, efficient, lazy way, the header:
> 
> Waring: 307 foo "some info \
> some more info"
> 
> Will be parsed as 2 headers, with the second header being very wrong
> and breaking everything.
> 
> If we want this to be parsed properly, then from the very begining we
> have to parse the header and understand it.
> 
> So what if we don't understand it?

I agree. In my coffee-starved state of mind I apparently believed the
escaping would help, but it doesn't. So, I think we need to specify that
if a CR, LF, or CRLF appears in a quoted string, these MUST be properly
line folded.

-Jonathan R.
-- 
Jonathan D. Rosenberg                       72 Eagle Rock Ave.
Chief Scientist                             First Floor
dynamicsoft                                 East Hanover, NJ 07936
[email-masked]                     FAX:   (973) 952-5050
http://www.cs.columbia.edu/~jdrosen         PHONE: (732) 741-7244
http://www.dynamicsoft.com

_______________________________________________
SIP mailing list
[email-masked]
http://lists.bell-labs.com/mailman/listinfo/sip

---

From: Anders Kristensen <[email-masked]>
Date: Fri, 18 Aug 2000 18:52:29 +0200

> if a CR, LF, or CRLF appears in a quoted string, these MUST be properly

Having had a look at the granddaddy of SIP syntax, RFC 822, it seems to
me that the SIP spec is no more and no less vague on when CR and LF are
allowed in quoted strings and what they mean than 822 is.  The
definitions are pretty much the same as far as I can tell.

From rfc822:

     quoted-string = <"> *(qtext/quoted-pair) <">; Regular qtext or
                                      ;   quoted chars.
     quoted-pair =  "\" CHAR                     ; may quote any char
     CHAR        =  <any ASCII character>        ; (  0-177,  0.-127.)

So this grammar does allows CR and LF inside quoted strings. The
question then is whether they're only alowed as part of folded lines
(what we want) or actually as part of the string itself, and the answer
is that it's the former...

        Each header field can be viewed as a single, logical  line  of
        ASCII  characters,  comprising  a field-name and a field-body.
        For convenience, the field-body  portion  of  this  conceptual
        entity  can be split into a multiple-line representation; this
        is called "folding".  The general rule is that wherever  there
        may  be  linear-white-space  (NOT  simply  LWSP-chars), a CRLF
        immediately followed by AT LEAST one LWSP-char may instead  be
        inserted.  Thus, the single line

            To:  "Joe & J. Harvey" <ddd @Org>, JJV @ BBN

        can be represented as: [...]
	
            To:  "Joe &
             J. Harvey" <ddd @ Org>, JJV @ BBN

I don't think this would have been clear from the grammer alone, and so
I guess that's why we're having this dicussion. Maybe it's been
clarified for email in more recent RFC's. Anyway, given these semantics
for email I think it's most reasonable to interpret 2543 the same way.
No modifications needed, really.

--
Anders Kristensen

---

From: jimmy Zhang <[email-masked]>
Date: Fri, 18 Aug 2000 16:44:01 -0700

I deeply feel that the syntax description is archaic, we should probably
write them in perl's regex format, which should help
parsing tremendously.

--

Jimmy zhang ([email-masked])

Software Engineer

---

From: Bobby Sardana <[email-masked]>
Date: Sat, 19 Aug 2000 19:12:54 -0700

Greetings:

jimmy Zhang wrote:

> I deeply feel that the syntax description is archaic, we should
> probably write them in perl's regex format, which should help
> parsing tremendously.

Can you provide an example and the benefits gained by perl's regular
expression format? Also, how will the folks who have parsers in 'C' and
Java be able to benefit from this format?

Regards,

Bobby Sardana.
[email-masked]

---

From: jimmy Zhang <[email-masked]>
Date: Mon, 21 Aug 2000 16:02:26 -0700

Sure.

 By using regex in Perl format, people will, first of all, be forced to
understand Perl's regex. Then they will discover their
old ways of parsing SIP is a total waste of time, and finally, become a
smart programmer.

Any questions, let me know.

Bobby Sardana wrote:

> expression format? Also, how will the folks who have parsers in 'C'
> and Java be able to benefit from this format?
>
> Regards,
>
> Bobby Sardana.
> [email-masked]
>
>>
>>
>>
--

Software Engineer

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 21 Aug 2000 16:54:52 -0400

http://www.ietf.org/internet-drafts/draft-ietf-drums-smtpupd-12.txt
http://www.ietf.org/internet-drafts/draft-ietf-drums-msg-fmt-08.txt

are probably the most recent relevant documents here. If somebody can
apply their text interpretation skills here and divine what's meant for
this case. In those documents, CR and LF only show up as being allowed
in obs-text or obs-qp, the obsolete syntax. This would suggest removing
them from SIP as well. I've sent email to the editors of those documents
to see if they can enlighten us.

As a simple example, consider

Header: "quoted string \
Subject: "

-- 
Henning Schulzrinne   http://www.cs.columbia.edu/~hgs

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 27 Aug 2000 18:08:10 -0500

Henning,

Try this one:

>   Header: "quoted string \
>   Subject: \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
>      folded"
>
>   v=0 

count those backslashes (their an odd number).  Anything but a turing
machine will have difficulty with this one.  Most simple regular
expressions will hack the body off between `Subject' and `folded'.

--Brian

Henning Schulzrinne wrote:               (Mon, 21 Aug 2000 16:54:52)

-- 
Brian F. G. Bidulock   ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
                       ¦ trying  to adapt the  world  to himself. ¦
                       ¦ Therefore  all  progress  depends on the ¦
M.O.A.M.O.N.N.T.O.M.E. ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Gethin Liddell <[email-masked]>
Date: Tue, 29 Aug 2000 10:14:51 +0100

In an attempt at closure on this (or has it already been closed ?) as
it is a little point and we could spend our time better on other stuff
i suggest some simple text as follows:

P. 119 (End of C.1):

-----

The backslash character ("\") MAY be used as a single-character quoting
mechanism only within quoted-string and comment constructs. NOTE the
characters "\r" and "\n" cannot be escaped by this mechanism so as not
to conflict with line folding and header separation.

quoted-pair =  "\" (0x00 - 0x09 | 0x0b | 0x0c | 0x0e - 0x7f)

-----

This will (hopefully) spell it out for any new comers to this problem
without them having to trall through other rfc's or for this thread in
the mailing list archives.

HTH

gethin

-- 
Gethin Liddell
Ubiquity Software Corporation

http://www.ubiquity.net
mailto:[email-masked]

---

From: Henning Schulzrinne <[email-masked]>
Date: Tue, 29 Aug 2000 06:36:43 -0400

Added (modulo wording changes). Thanks.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 29 Aug 2000 09:50:09 -0500

Gethin,

Just as long as it is understood that the backslash character
cannot be used anywhere else in the start line or headers.
Otherwise there is the parsing concern of having to recognize
headers before understanding that CR or LF preceded by a
BACKSLASH truely escapes the CR or LF.

Consider the following to split headers:

        precondition:     ^(([^\\]([\\].)*?)*?)
        match condition:  (\n|\r\n?)
        postcondition:    [^ \t]

--Brian

Gethin Liddell wrote:                 (10:14:51 +0100 Tue, 29 Aug 2000)

[email-masked] ¦ world; the unreasonable one persists in  ¦

---

From: Gethin Liddell <[email-masked]>
Date: Tue, 29 Aug 2000 16:12:49 +0100

Brian

I'm sorry, i think you've misunderstood me. what i'm saying is that
BACKSLASH never escapes CR or LF.

where ever CR or LF appear, they MUST be taken as header separators or
line folding.  nothing more, nothing less. they CANNOT be escaped.

so BACKSLASH can appear pretty much in any header (where allowed by the
bnf of course ;) )

thus CR & LF have been removed from the quoted-pair syntax below
because they cannot be escaped.

do you agree?

http://www.ubiquity.net

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 29 Aug 2000 10:51:39 -0500

Gethin,

Gethin Liddell wrote:                 (16:12:49 +0100 Tue, 29 Aug 2000)
> > > quoted-pair =  "\" (0x00 - 0x09 | 0x0b | 0x0c | 0x0e - 0x7f)

Pardon me, I'm not reading well today...

> >         match condition:  (\n|\r\n?)

Well that has some (extremely) minor advantages:  It removes 5 lines of
assembler from my splitter ;)

But tell me: what is the use of backslashing other control characters
other than 0x08?  Particularly when backslashing those are optional?
(Only ,;"() really need be backslashed and only " must be within quoted
text and ) within a comment.  What's the point of having control
characters at all?

-- 
Brian F. G. Bidulock
[email-masked]

---

From: Gethin Liddell <[email-masked]>
Date: Tue, 29 Aug 2000 16:58:05 +0100

> assembler from my splitter ;)

minor? it allows `\' to appear quite happily in any header without
confusion.  ahhh, you mean minor change to code, big change in
flexibility of the spec.

> > > > quoted-pair =  "\" (0x00 - 0x09 | 0x0b | 0x0c | 0x0e - 0x7f)

what is the point in backslashing 0x08? the only characters that MUST
be escaped are `"' in quoted string and `(' and `)' in comment (you
can have sub-comments).  there is absolutely no need what so ever to
escape any other characters.  so why did i leave them in the BNF?
because if i had of removed them someone else would have complained
about that.  besides i suppose this way is more backwards compatible if
you really want an explanation (and that's the nearest it gets ;) ). 
and of course, we can now parse quoted string and comment by simply
ignoring any character after a `\' without having to validate it to one
of `"', `(' and `)' (assuming of course that we have stripped line
folding prior to this :) ).

http://www.ubiquity.net

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 29 Aug 2000 11:58:13 -0500

Gethin,

Gethin Liddell wrote:                 (16:58:05 +0100 Tue, 29 Aug 2000)
[snip]
> what is the point in backslashing 0x08? the only characters that MUST

Sorry, I meant 0x09.

> you really want an explanation (and that's the nearest it gets ;) ). 

`nuff said.

> folding prior to this :) ).

Speaking of line folding, now when I am folding lines I must watch out
for the construction [\\][ \t], otherwise I might erroneously place a LF
where the space was and violate the new BNF.  Otherwise, I have to scan
through and remove those constructions.  (I know, you'll say don't build
them that way in the first place, but most headers I don't create as a
proxy, I receive them.)  How about excluding %x09 and %x20 from the list
as well?  That way I know that I can always fold a line at TAB or SP.

quoted-pair = "\" (0x00 - 0x08 | 0x0b | 0x0c | 0x0e - 0x1f | 0x21 - 0x7f)

-- 
Brian F. G. Bidulock
[email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 29 Aug 2000 15:31:59 -0500

Gethin,

You might want to take that double backslash
out of the list too:  less problems when there's
three of them in a row.  Such as ////" vs /////".

 quoted-pair =
    "\" (0x00 - 0x08 | 0x0b | 0x0c | 0x0e-0x5b | 0x5d-0x1f | 0x21-0x7f)

Brian Bidulock wrote:                 (11:58:13 -0500 Tue, 29 Aug 2000)

-- 
Brian F. G. Bidulock
[email-masked]

---

From: Henning Schulzrinne <[email-masked]>
Date: Wed, 30 Aug 2000 03:47:41 -0400

I would like to stay compatible with the DRUMS (revision of RFC 822)
work, even though escaping will practically only be used in limited
circumstances. I don't think it's worth diverging here.

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 30 Aug 2000 03:38:42 -0500

\
\
"

Henning,

What makes the DRUMS case any different?  Backward compatibility?  Does
this mean to be backwards compatible I should still continue to squeeze
received quoted text with [^\\]([\\].)* but only generate per the BNF?  I
thought that this was handled by the cannonical form: as in, the cannonical
form expresses what I should send, whereas, the BNF expresses what I should
receive (if I am a server).  Are you saying that I should replace my server
expressions from [^\\]([\\].)* to [^\\]([\\][\x00-\x08\x0b\x0c\x0e-\x7f])*
I think it makes sense to still use [^\\]([\\].)* regardless of the
definition of quoted-pair and use the cannonical rules when sending.

Quite frankly, I think [\\][\n] in a comment is a pretty natural
language kinda thing when trying to mark up a message by hand or include
human readible debuggin information.

Did DRUMS have some reasons why they did this?

--Brian

Henning Schulzrinne wrote:           (03:47:41 -0400 Wed, 30 Aug 2000)

-- 
Brian F. G. Bidulock
[email-masked]

---

From: Gethin Liddell <[email-masked]>
Date: Wed, 30 Aug 2000 09:33:31 +0100

> circumstances. I don't think it's worth diverging here.

agreed.  

also there are far too many implementations out there that will not want
any more changes to this as their code will work fine at the moment but
to start removing more stuff will mess them up.  And it is such
an insignificant part of sip anyway.

gethin

http://www.ubiquity.net

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Wed, 30 Aug 2000 03:49:21 -0500

Henning,

Forget it.  Judging from what happened to my last header line nobody
handles it for 822 anyway!  Looks like there are those replacing any
[\\][^\"] in a quoted string with [^\"] before doing anything else.  My
escaped LFs turned into LF LF and the rest jumped into the body.

Brian Bidulock wrote:                (03:38:42 -0500 Wed, 30 Aug 2000)
> \     <--------
> \     <--------
> "     <--------
[snip]

-- 
Brian F. G. Bidulock
[email-masked]