Subject: Line break conventions: SIP, RTSP, SDP, HTTP

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 24 Apr 1997 09:36:28 -0400

Like all other Internet text-based protocols, SIP and RTSP (and HTTP,
naturally) use the CRLF convention to represent line breaks. Besides
historical precedent, this makes it easy to use the 'universal telnet
protocol emulator' with these protocols. This, however, doesn't work
when the SIP and RTSP payloads are SDP, as their current line break
character is LF (\n), the Unix convention. Similarly, for email delivery
of SDP as text or via http, it is likely that some email systems and web
browsers or text will use the canonical line break convention (CRLF) and
thus break anal-retentive SDP parsers that insist on only LF. Needless
to say, a certain commercially relevant operating system uses the CRLF
convention for text, so that handediting on these systems is going to be
difficult.

HTTP also (RFC 2068, section 3.7.1, "Canonicalization and Text
Defaults") says:

--- begin quote ---

   Internet media types are registered with a canonical form. In
   general, an entity-body transferred via HTTP messages MUST be
   represented in the appropriate canonical form prior to its  
   transmission; the exception is "text" types, as defined in the next
   paragraph.
 
   When in canonical form, media subtypes of the "text" type use CRLF as
   the text line break. HTTP relaxes this requirement and allows the
   transport of text media with plain CR or LF alone representing a line
   break when it is done consistently for an entire entity-body. HTTP
   applications MUST accept CRLF, bare CR, and bare LF as being  
   representative of a line break in text media received via HTTP. In
   addition, if the text is represented in a character set that does not
   use octets 13 and 10 for CR and LF respectively, as is the case for
   some multi-byte character sets, HTTP allows the use of whatever octet
   sequences are defined by that character set to represent the
   equivalent of CR and LF for line breaks. This flexibility regarding
   line breaks applies only to text media in the entity-body; a bare CR
   or LF MUST NOT be substituted for CRLF within any of the HTTP control
   structures (such as header fields and multipart boundaries).
 
--- end quote ---

Given Internet precedent, the rule of 'be liberal in what you accept'
and the integration with SIP and RTSP, I would strongly suggest that SDP
follow the attitude of HTTP: Allow any of the combinations outlined
above. The cost to implementations is negligible.

The current specification is not all that clear 

"Text records such as the session name and information  may  contain 
any
printable  8  bit ISO 8859-1 character with the exceptions of 0x0a (new-
line) and 0x0d (carriage return).  Carriage Return  is  prohibited,  and
Newline is used to end a record."

This is the only in-text reference to line break conventions [and, from
the wording, only applies to the text records] beyond the BNF (which is
slightly wrong, since it's not ASCII code 10 but rather the binary value
10; Internet protocols also call it LF, not newline).

Also, in line with recent IETF RFCs and IESG recommendations (see RFC
2130), the spec should use UTF-8 or UTF-7, not 8859-1, as the default.
Since UTF-8 is the recommended form, there should be a motivation why
UTF-7 is used. (The only candidate for a 7-bit channel is non-MIME email
using cut-and-paste, as far as I can tell, except that this would fail
on DOS/Windows boxes unless the linebreak convention is changed; true
7-bit email would break badly with 8859-1, obviously.) 

Henning

---

From: Mark Handley <[email-masked]>
Date: Thu, 24 Apr 1997 09:50:26 -0400

Henning,

Your comments about CRLF in SDP and UTF-8 rather than UTF-7 make
sense, and echo my own recent feelings on this.  The latter certainly
won't break any implementations I know of.  Changing the SDP spec to
allow either LF or CRLF (with a preference for LF simply because it's
shorter) seems fine with me, although it may cause some problems with
current implementations.

Is anyone going to object if I make these changes?

Changing the default from 8859-1 to 10646/UTF-8 is clearly the
cleanest thing to do, but at the time that I specified 10646 as an
option, there was still a large amount of debate about its inability
to cope with Chinese and Japanese properly.  Has that debate now died
down?

Mark

---

From: Henning Schulzrinne <[email-masked]>
Date: Thu, 24 Apr 1997 10:13:09 -0400

> Henning,

A related suggestion would be to bring the BNF in line with other
similar specs (i.e., without listing CRLF explicitly for each field).

> to cope with Chinese and Japanese properly.  Has that debate now died

Clearly, 8859-1 doesn't deal with Chinese at all, so that's not exactly
a reason to keep it... However, the issue you are referring to ("Han
unification") is roughly that certain Han (Chinese) characters have
different glyphs (look different) in different languages based on the
Chinese characters (I'm not a character set expert, but I think this is
roughly right). Thus, for proper rendering, both an indication of
character set and language is needed. RFC 2130 has some discussion of
this topic. Thus, my suggestion:

- use 10646/UTF-8 since it covers far more languages more cleanly than
the alternatives and is not obviously biased towards one country or set
of countries (as opposed to 8859-1, which, if we were on the liberal
arts side of the university, I'd be flogged for as a perpetuator of
cultural bias, racism, Eurocentrism, dead white males, etc. :-))

- add a language parameter like charset to the definition to avoid the
Han unification problem as well as allow better filtering (if the
announcement is in Ungarian, I may not want to display it since the
likelihood that I benefit from it is somewhat limited; also makes it
possible to have several announcement in different languages). This
should be both a session-level and media-level attribute.

Henning