Subject: [SIP] 2543 bis: Section H.1 URI Syntax/header syntax

From: "Rick Workman" <[email-masked]>
Date: Fri, 05 May 2000 09:14:09 -0400

Regarding header syntax in the Open Issues section, it's suggested that
hname and hvalue can't be *uric because of the obvious ambiguities. One
suggestion is to just disallow  "&" and "=". Another possibility is to
just remove reserved as an alternative in the uric rule, since the text
states that "All URL reserved characters in the header names and values
must be escaped", i.e., there's no point in including reserved in the
BNF for uric, at least for this purpose.

It seems the general intent is to allow the specification of SIP headers
in SIP URL's. Since the message header field-name [6.6] is defined to be
a token, this suggests that it's sufficent that hname is a token as
well. Unfortunately, the generic definition of field-value isn't quite
so helpful:

field-value = *(field-content | LWS)
field-content = <the OCTETs making up the field-value and consisting
                 of either *TEXT-UTF8 or combinations of token,
                 separators and quoted-string>

So perhaps the first thing to do is to replace the ABNF prose rules for
field-content and TEXT-UTF8 with something more formal, e.g.,

field-content = quoted-string | separator | token | TEXT-UTF8char

where

TEXT-UTF8 = TEXT-UTF8char | LWS
TEXT-UTF8char = %x21-7e | %x80-ff

Ideally, we'd like to equate header values to field values, i.e.,

hvalue = field-value

but field-value is even more general than the current 1*uric used for
hvalue, so the problem gets worse. In the absence, of a more useful
definition of field-content, there seem to be two approaches:

1. Follow the current spec and escape every reserved character. My own
feeling is that this renders most things somewhat unreadable (if that
matters), e.g.,

sip:[email-masked];maddr=239.255.255.1;ttl=15

becomes:

sip%3aj.doe%big.com%3bmaddr%3d239.255.255.1%3bttl%3d15

2. The other approach is to use an "envelope" syntax, like
quoted-string, i.e.,

hvalue = token | quoted-string

Now only quotes have to be escaped (as a quoted-pair), but these seem to
be relatively infrequent in most of the SIP examples I've seen. The rule
for converting an hvalue to a field-content :
a) if it's a token, it is the field-content
b) if it's a quoted-string, remove the outer quotes and convert each
contained quoted-pair to corresponding character.

Now if quoted-string is a problem for the reason cited earlier in a
similar discussion on parameter names and values (double quote marks are
excluded from URI's), use a single quoted string instead. Single quote
is allowed in URI's and I don't think it's used for any SIP syntax.
However, this only excludes them from the outer level syntax unless we
want to exclude quoted-strings from all of SIP overall. (Or maybe just
from any SIP contained in URL headers.)

In any case, approach 2. seems more attractive to me, but I'm not sure
what current practice is in this area.

Rick Workman
Nortel Networks
Ottawa

P.S. Minor point: In section G "Changes to Be Made", a bug is recorded
for the Call-ID header which proposes to change the field-content to
token. I agree with the intent of the proposal, but I don't think token
allows @. This may break a lot of existing implementations since the
current RFC insists on @ being present. Also note that the current spec
also allows @ in the local-id which may cause a bit of confusion, but I
suspect most implementations just treat Call-ID as a single character
sequence anyway.

---

From: Jonathan Lennox <[email-masked]>
Date: Fri, 5 May 2000 10:17:59 -0400 (EDT)

> BNF for uric, at least for this purpose.

My inclination is to just disallow "&" and "=", as well as any other
syntactially obnoxious characters such as ",".  I think it's most in the
spirit of the URI syntax.

> TEXT-UTF8char = %x21-7e | %x80-ff

This isn't a complete definition of TEXT-UTF8.  Maybe this is pedantic, but
a proper BNF for the RFC 2279 definition of UTF8 would be:

TEXT-UTF8 = LWS
          | %x21-7e
          | UTF8-NONASCII

UTF8-NONASCII = %xc0-df 1UTF8-CONT
              | %xe0-ef 2UTF8-CONT
              | %xf0-f7 3UTF8-CONT
              | %xf8-fb 4UTF8-CONT
              | %xfc-fd 5UTF8-CONT

UTF8-CONT : %x80-bf

Splitting out UTF8-NONASCII from TEXT-UTF8 allows easy syntactic definitions
of such things as quoted string bodies (qdtext), which are currently
textually defined as < any TEXT-UTF8 except " or \ >:

qdtext : LWS
       | %x21 | %x23-5b | %x5d-7e
       | UTF8-NONASCII

The question still arises of how do you encode UTF8 field content in a
header parameter.  I think the only right solution is to encode first in
UTF8, then to do hex %-escaping; the alternative, %-escaping 16-bit or
32-bit unicode, is just awful.

> sip%3aj.doe%big.com%3bmaddr%3d239.255.255.1%3bttl%3d15

This isn't correct, because the point of escaped characters is to
distinguish *URL* syntax characters from field characters.  Thus, the URLs:

and

sip:[email-masked];maddr=239.255.255.1%3Bttl%3D15

are *different* URLs; the second has only a single parameter, whose name is
"maddr" and whose value is "239.255.255.1;ttl=15".  I think anything we do
needs to be sure to preserve this distinction.

-- 
Jonathan Lennox
[email-masked]

_______________________________________________
SIP mailing list
[email-masked]
http://lists.bell-labs.com/mailman/listinfo/sip

---

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 05 May 2000 11:27:30 -0400

> UTF8-NONASCII = %xc0-df 1UTF8-CONT

Anything formal for TEXT-UTF8-TRIM?

Maybe = TEXT-UTF8-NOLWS *TEXT-UTF8

but that doesn't express the lack of trailing LWS, although I'm not sure
this actually matters from a parsing perspective.

> 32-bit unicode, is just awful.

http://www.ietf.org/internet-drafts/draft-masinter-url-i18n-05.txt
describes this problem and does indeed suggest that the bytes are
hex-encoded if needed. Unfortunately, it's still a pretty rough draft
and unlikely to be citable in RFC2543bis.

---

From: Jonathan Lennox <[email-masked]>
Date: Fri, 5 May 2000 12:47:45 -0400 (EDT)

> Maybe = TEXT-UTF8-NOLWS *TEXT-UTF8

TEXT-UTF8-TRIM = TEXT-UTF8-NOLWS *TEX-UTF8 TEXT-UTF8-NOLWS
               | TEXT-UTF8-NOLWS
               | ; empty string

This should work.  I'm not sure if you want to allow the last one.

-- 
Jonathan Lennox
[email-masked]

---

From: "Rick Workman" <[email-masked]>
Date: Fri, 05 May 2000 13:59:08 -0400

Another possibility which includes the empty string (I think this works):

TEXT-UTF8 = *(TEXT-UTF8char | LWS)

TEXT-UTF8-TRIM = *TEXT-UTF8char *(*LWS TEXT-UTF8char)

TEXT-UTF8char = %x21-7e | UTF8-NONASCII

; same as before

UTF8-CONT = %x80-bf

These are subtly different from the current TEXT-UTF8 and TEXT-UTF8-TRIM definitions in that the repeat
has been moved inside the rule, e.g., the subject header ABNF should be changed to:

Subject = ("Subject" | "s") ":" TEXT-UTF8-TRIM

Rick

---

From: "Rick Workman" <[email-masked]>
Date: Fri, 05 May 2000 13:36:40 -0400

I think the big picture argument you're making is that it's better to define a
reserved character set for hvalue that just removes the obvious offenders ("&",
"=", and ",") from the current definition. A couple of observations:

1. The text that says "All URL reserved characters in the header names and
values must be escaped" has me confused. Either it's just wrong, or there really
isn't much point in defining the reserved set in the definition, i.e., any
characters not in the uric set of characters must be escaped.

2. If this is true, characters not in the uric set which must escaped include
"<", ">", "\", double quote and space. These are present in SIP and could be
included in the reserved set for hvalue to improve readability, but are either
excluded or "unwise" according to the URI spec.

3. A SIP URL, including all the headers, can't contain any line breaks. This
probably shouldn't come as a surprise to anyone since most people have had to
deal with long http URLs in emails etc.

Thanks for the clarification on the TEXTUTF-8. In my opinion, we're not pedantic
enough when it comes to using ABNF. Prose rules ought to be strongly
discouraged. They are often used because or lack of understanding of ABNF or
sheer laziness. The few remaining cases are usually misguided attempts to
capture semantics in a syntax specification.

Rick

---

From: Henning Schulzrinne <[email-masked]>
Date: Fri, 05 May 2000 19:03:12 -0400

Rick Workman wrote:
> These are subtly different from the current TEXT-UTF8 and
> TEXT-UTF8-TRIM definitions in that the repeat has been moved inside
> the rule, e.g., the subject header ABNF should be changed to:
> 

I updated the spec accordingly.

-- 
Henning Schulzrinne   http://www.cs.columbia.edu/~hgs

---

From: "Rick Workman" <[email-masked]>
Date: Mon, 08 May 2000 11:41:34 -0400

The current SIP BNF allows the general form of URI (or URI-reference?) to
be used in the request line and various headers. Observations and
questions:

1. A valid form of URI (RFC 2396)is the empty sequence since:

URI-reference = [ absoluteURI | relativeURI ] [ "#" fragment ]

This would permit the request line and various headers (To, From,
Contact. ..) without an address specification which doesn't seem to make
much sense. RFC 2396 specifies that the empty sequence URI refers to the
start of the current document; is there any corresponding abstraction for
SIP transcations?

2. The relative URI references a resource relative to a base URI of a
hierarchial scheme. So a similar question: is this a useful concept for
SIP sessions?

3. Finally RFC 2396 states (my emphasis):
  "When a URI reference is used to perform a retrieval action on the
   identified resource, the optional fragment identifier, separated from
   the URI by a crosshatch ("#") character, consists of additional
   reference information to be interpreted by the user agent after the
   retrieval action has been successfully completed.  As such, it is not
   part of a URI, but is often used in conjunction with a URI."

So this very general URI seems to have a number of properties which don't
seem to be relevant when used in place of a SIP URL. Since generality
often has a price either in implementation or specification, i.e., lots
of semantic constraints, does it make sense to restrict the syntactic
form of URI used in place of a SIP URL, e.g., to the absolute-URI form?

Rick Workman
Nortel Networks
Ottawa

---

From: "Rick Workman" <[email-masked]>
Date: Wed, 10 May 2000 14:38:56 -0400

In an attempt to get some closure on the SIP-URL header BNF, how about:

headers     = "?" header *("&" header)
header      = hname "=" hvalue
hname       = 1*(hname_char)
hvalue      = *(hvalue_char)

hname_char  = alphanum | "-" | "." | "!" | "*" | "_" | "+" | "'" | "~"

hvalue_char = hvreserved | unreserved | escaped
hvreserved  = ";" | "/" | "?" | ":" | "@" | "=" | "+" |"$"

Notes:
1. This isn't strictly backward compatible but I can't see any practical
issues. (Same as proposed change to other-param?)
2. hname_char set is a subset of token (to be consistent with
"field-name") which doesn't include "%" (URI "delims") or "`" (URI
"unwise").
3. hvreserved is a subset of the current reserved rule eliminating "&",
"," as per earlier discussion, and "[" and "]" (both URI "unwise").
4. Scanning the current header definitions, at least the following must
be escaped:
   - any required LWS, e.g., for token separation or in rfc1123 dates
   - double quotes in quoted strings
   - "\" in quoted-pair (comments and quoted-strings)
   - "%" and "`" in tokens
   - "<" and ">" in name-addrs
   - "&" in any embedded SIP-URL headers
   - "," in # lists
   - any illegal characters in TEXT-UTF8
   - any others?
5. Multi-byte UTF-8 characters will be represent by a 2*escaped , i.e.,
the UTF-8 character CA98 will escape to %CA%98. (I think this was the
intent of Jonathan Lennox's previous proposal.)

Also, I'd like to see the text concerning reserved characters and
escaping clarified. If the BNF can be tightened up to the point where the
allowable characters for any SIP-URL component have been clearly defined,
then, by definition, any other characters must be escaped. Calling these
other characters reserved confuses them with those in the "reserved" rule
in the URI spec (and the current SIP RFC). Am I missing something?

Rick Workman
Nortel Networks
Ottawa

---

From: "Rick Workman" <[email-masked]>
Date: Thu, 11 May 2000 15:27:25 -0400

Didn't get this quite right. RFC 2732 (IPv6 Literal Addresses in URL's)
updates RFC 2396 to include "[" and "]" in the URL reserved set.
Corrected ABNF is:

hvalue_char = hvreserved | unreserved | escaped
hvreserved  = ";" | "/" | "?" | ":" | "@" | "=" | "+" | "$" | "[" | "]"

Rick Workman
Nortel Networks
Ottawa

> 1. This isn't strictly backward compatible but I can't see any
> practical issues. (Same as proposed change to other-param?)
> 3. hvreserved is a subset of the current reserved rule eliminating
> "&", "," as per earlier discussion, and "[" and "]" (both URI
> "unwise").
> 4. Scanning the current header definitions, at least the following
> must be escaped:
> 5. Multi-byte UTF-8 characters will be represent by a 2*escaped ,
> i.e., the UTF-8 character CA98 will escape to %CA%98. (I think this
> was the intent of Jonathan Lennox's previous proposal.)
>
>
> escaping clarified. If the BNF can be tightened up to the point where
> the allowable characters for any SIP-URL component have been clearly
> defined, then, by definition, any other characters must be escaped.
> Calling these other characters reserved confuses them with those in
> the "reserved" rule in the URI spec (and the current SIP RFC). Am I
> missing something?
>
>
> Rick Workman
> Nortel Networks
> Ottawa