Subject: [AAA-WG]: [issue] Bring back UTF8 datatype.

From: Mark Eklund <[email-masked]>
Date: Fri, 18 May 2001 19:01:32 -0400 (EDT)

Submitter name: Mark Eklund
Submitter email address: [email-masked]
Date first submitted: 18-May-2001
Reference:
Document: Base
Comment type: T
Priority: 2
Section: 4.3
Rationale/Explanation of issue:

  First of all, sorry to drag up old issues.  Here's the thoughts.

  * At one time we had Data, Complex, and String formats

  * When the datatypes were discussed in a meeting it was decided
    that Data, Complex, and String were the same datatype.  They
    were combined to be Octetstring.

  * Now, if something is actually UTF8 format, it will be Octetstring
    with the text that it is in UTF8 format.

  * Octetstring is defined as either some collection of data bytes,
    or a message in UTF8 format.

  * There is nothing to prevent one extension from defining an AVP
    as UTF8 and another extension defining it as just bunch of bytes.

  * The argument that an UTF8 formatted message is simply an Octetstring
    is akin to saying that an Integer32 and an Unsigned32 are just a
    two four byte numbers.  Following that logic, those numbers should
    be combined and the AVP definition should state that the number is
    in signed format.

Requested change:
    Add a new type of UTF8 and change all Octetstrings that say
    they are UTF8 to this type.

====================================================================

---

From: Mark Eklund <[email-masked]>
Date: Tue, 29 May 2001 10:52:12 -0400 (EDT)

erik.guttman and schoenw,

I have made the request to make UTF8 an official defined type in the 
Diameter Base draft.  I was told that it was removed because it was
not a primitive type as defined in SMI.  It would be convenient 
to have an officially defined type of UTF8 so that other Diameter
extensions could refer to it explicitly instead of doing what is 
now done by defining it as OctetString and then describing it as
UTF8 format in the text.

I was given the impression that one of you could give me guidance
on how to add this.

I read RFC1155, "Structure and Identification of Management
Information for TCP/IP-based Internets".  In section 3.2.1, it
defines the Primitive Types of INTEGER, OCTET STRING, OBJECT
IDENTIFIER, and NULL.  In section 3.2.3 it defines Defined Types
of Counter, Gauge, TimeTicks, IpAddress, and Opaque which are
derived from the Primitive Types.  I would like to do something
similar with UTF8 derived from OctetString.  The wording is

      OctetString
	 The data contains arbitrary data of variable length. Unless
	 otherwise noted, the AVP Length field MUST be set to at least
	 9 (13 if the 'V' bit is enabled).  AVP Values of this type
	 that do not align on a 32-bit boundary MUST have the necessary
	 padding.

         UTF8
	     This application-wide type is represented as an
	     OctetString.  It is used to transmit (human readable)
	     character string data uses the UTF-8 [24] character set
	     and is NOT NULL-terminated. The minimum Length field MUST
	     be 9, but can be set to any value up to 65504 bytes.

Currently I would indent it after the definition of OctetString,
in section 4.3, "AVP Data Formats" but it could be placed in a
separate section like 4.3.1, "Defined Data Types".

Do you have any suggestions/comments?

Thanks,

-mark

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 31 May 2001 15:17:22 +0200

>>>>> Mark Eklund writes:

Mark> I have made the request to make UTF8 an official defined type in
Mark> the Diameter Base draft.  I was told that it was removed because
Mark> it was not a primitive type as defined in SMI.  It would be
Mark> convenient to have an officially defined type of UTF8 so that
Mark> other Diameter extensions could refer to it explicitly instead
Mark> of doing what is now done by defining it as OctetString and then
Mark> describing it as UTF8 format in the text.

I argued in the AAA DT that Diameter should have a small set of base
types and that there should be a mechanism to define new types which
add restrictions to the core base types. Experience with SNMP tells us
that the number of data types you have needs to be changed over time
and that having to touch the base protocol to introduce something new
is way too costly.

With this in mind, I recommended to stay with an orthogonal set of
base types (distinguished by type tags) and I suggested to have a kind
of a typedef mechanism (in SMIng terms) or a textual convention
mechanism (in SMIv2 terms) that allows to define derived types which
have additional semantics. The WG did not want to go with a formal
data type definition language and so we ended up where we are now.

I still believe that it would be valuable to formally say something
like the following (copied from draft-ietf-sming-modules-01.txt):

   typedef Utf8String {
       type        OctetString;
       format      "65535t";      // is there a better way ?
       description
          "A human readable string represented using the ISO/IEC IS
           10646-1 character set, encoded as an octet string using
           the UTF-8 transformation format described in RFC 2279.

           Since additional code points are added by amendments to
           the 10646 standard from time to time, implementations must
           be prepared to encounter any code point from 0x00000000 to
           0x7fffffff.  Byte sequences that do not correspond to the
           valid UTF-8 encoding of a code point or are outside this
           range are prohibited.

           The use of control codes should be avoided. When it is
           necessary to represent a newline, the control code
           sequence CR LF should be used.

           The use of leading or trailing white space should be
           avoided.

           For code points not directly supported by user interface
           hardware or software, an alternative means of entry and
           display, such as hexadecimal, may be provided.

           For information encoded in 7-bit US-ASCII, the UTF-8
           encoding is identical to the US-ASCII encoding.

           UTF-8 may require multiple bytes to represent a single
           character / code point; thus the length of a Utf8String in
           octets may be different from the number of characters
           encoded.  Similarly, size constraints refer to the number
           of encoded octets, not the number of characters
           represented by an encoding.

           Note that the size of an Utf8String is measured in octets,
           not characters.";
   };

You can of course do the same informally in a new section "Derived
Data Types" - but you will end up with problems when you get name
clashes because vendors introduce their own types and so on.

Note that I also suggested to not define the Address AVP with its 
own type tag. Instead, I suggested to define addresses as special
interpretations of OctetStrings. This again gives you flexibility
to add new address formats as the need shows up.

Note also that you probably need to define a bunch of additional
derived types that are frequently used such as a TimeStamp or
NetworkAccessIdentifier and so on.

/js

-- 
Juergen Schoenwaelder      Technical University Braunschweig
<[email-masked]>  Dept. Operating Systems & Computer Networks
Phone: +49 531 391 3289    Bueltenweg 74/75, 38106 Braunschweig, Germany
Fax:   +49 531 391 5936    <URL:http://www.ibr.cs.tu-bs.de/~schoenw/>

---

From: David Frascone <[email-masked]>
Date: Thu, 31 May 2001 08:50:21 -0500

I brought up a similar point a while ago.  I wanted a "printable as" type
attribute, so you could tell if an octet string was text, or was binary 
(unprintable) data.  Also, I wanted to be able to tell if an Integer32 was
a time.

The general consensus of the WG was that the protocol should be left alone,
but the dictionaries in the implementation could contain extra informaiton
about how to best display the AVP.  So, in a dictionary, there would be
a flag stating that a FQDN was really a UTF8, and that a Timestamp was
really a time_t.  But, the protocol would not (and should not) care.

I was swayed in their direction, and I think the same applies here.  The
existing types can contain any possible data we need to model.

-Dave

---

From: Mark Eklund <[email-masked]>
Date: Thu, 31 May 2001 18:34:28 -0400 (EDT)

Juergen,

Is the following acceptable.

 1. I renamed "AVP Data Formats" to "AVP Base Data Formats".
 2. I created a new section, "AVP Derived Data Formats".
 3. I stated that an AVP Derived Data Format MUST only be
    defined in one RFC.
 4. I stated that a new AVP Derived Data Format MUST be
    registered with IANA.  (This may be a little extreme.)
 5. I moved Address to the AVP Derived Data Formats.
 6. I added a time format to the AVP Derived Data Formats.
 7. I added UTF8String and used most of the explanation from
    your example to describe it.
 8. Additionally, I specified that UTF8String MUST not contain
    an octet with a value of zero.
 9. I added a DiameterIdentity format (formerly FQDN) to the AVP
    Derived Data Formats.
10. I added an Enumerated format to the AVP Derived Data
    Formats.

Below is the new text.  Is the basic idea getting closer to
respectable?  If so, any comments on the details?

4.3  AVP Base Data Format

   The Data field is zero or more octets and contains information
   specific to the Attribute. The format and length of the Data field is
   determined by the AVP Code and AVP Length fields. The format of the
   Data field MUST be one of the following base data types or a data
   type derived from the base data types.  In the event that a new
   AVP Base Data Format is needed, a new version of this RFC must be
   created.

      OctetString
         The data contains arbitrary data of variable length. Unless
         otherwise noted, the AVP Length field MUST be set to at least 9
         (13 if the 'V' bit is enabled).  AVP Values of this type that
         do not align on a 32-bit boundary MUST have the necessary padding.

      Integer32
         32 bit signed value, in network byte order. The AVP Length
         field MUST be set to 12 (16 if the 'V' bit is enabled).

      Integer64
         64 bit signed value, in network byte order. The AVP Length
         field MUST be set to 16 (20 if the 'V' bit is enabled).

      Unsigned32
         32 bit unsigned value, in network byte order. The AVP Length
         field MUST be set to 12 (16 if the 'V' bit is enabled).
         Unsigned32 values used to transmit time data contains the four
         most significant octets returned from NTP [18], in network byte
         order.

      Unsigned64
         64 bit unsigned value, in network byte order. The AVP Length
         field MUST be set to 16 (20 if the 'V' bit is enabled).

      Float32
         This represents floating point values of single precision as
         described by [30].  The 32 bit value is transmitted in network
         byte order. The AVP Length field MUST be set to 12 (16 if the
         'V' bit is enabled).

      Float64
         This represents floating point values of double precision as
         described by [30].  The 64 bit value is transmitted in network
         byte order. The AVP Length field MUST be set to 16 (20 if the
         'V' bit is enabled).

      Float128
         This represents floating point values of quadruple precision as
         described by [30].  The 128 bit value is transmitted in network
         byte order. The AVP Length field MUST be set to 24 (28 if the
         'V' bit is enabled).

      Grouped
         The Data field is specified as a sequence of AVPs.  Each of
         these AVPs follows - in the order in which they are specified -
         including their headers and padding.  The AVP Length field is
         set to 8 (12 if the 'V' bit is enabled) plus the total length
         of all included AVPs, including their headers and padding.

4.4 AVP Derived Data Formats
      In addition to the AVP Base Data Formats, applications may define
      data formats derived from the AVP Base Data Formats.  New AVP Derived 
      Data Formats MUST be registered with IANA.  A derived Data Format
      can only be defined in one active (non-obsolete) RFC.

      An application that uses AVP Derived Data Formats other than those
      defined in the base protocol MUST have a section "AVP Derived Data
      Formats" that includes each of these formats.  In that section, 
      each format is either defined or listed with a reference to the
      RFC that defines this format.  If the AVP Derived Data Format is
      defined, it SHOULD use a format similar to the format
      definitions below.

      The below AVP Derived DATA Formats are commonly used by applications.
      
      Address
	 The Address format is derived from the OctetString AVP Base 
         Format. It represents 32 bit (IPv4) [17] or 128 bit (IPv6)
         [16] address, most significant octet first. The format of the
         address (IPv4 or IPv6) is determined by the length. If the
         attribute value is an IPv4 address, the AVP Length field MUST
         be 12 (16 if 'V' bit is enabled), otherwise the AVP Length
         field MUST be set to 24 (28 if the 'V' bit is enabled) for
         IPv6 addresses.
      
      Time
         The Time format is derived from the Unsigned32 AVP Base
	 Format.  This is 32 bit unsigned value containing the four
         most significant octets returned from NTP [18], in network byte
         order.
	 
	 This represent the number of seconds since 0h on 1 January
	 1900 with respect to the Coordinated Universal Time (UTC).

	 On 6h 28m 16s UTC, 7 February 2036 the time value will overflow.
	 NTP [18] describes a procedure to extend the time to 2104.
      
      UTF8String
	 The UTF8String format is derived from the OctetString AVP
         Base Format.  This is a human readable string represented
         using the ISO/IEC IS 10646-1 character set, encoded as an
         OctetString using the UTF-8 [29] transformation format
         described in RFC 2279.

         Since additional code points are added by amendments to the
         10646 standard from time to time, implementations MUST be
         prepared to encounter any code point from 0x00000001 to
         0x7fffffff.  Byte sequences that do not correspond to the
         valid UTF-8 encoding of a code point or are outside this
         range are prohibited.

	 The use of control codes SHOULD be avoided. When it is
         necessary to represent a newline, the control code sequence
         CR LF SHOULD be used.

         The use of leading or trailing white space SHOULD be avoided.

         For code points not directly supported by user interface
         hardware or software, an alternative means of entry and
         display, such as hexadecimal, MAY be provided.

         UTF-8 may require multiple bytes to represent a single
         character / code point; thus the length of a UTF8String in
         octets may be different from the number of characters
         encoded.

         Note that the size of an UTF8String is measured in octets,
         not characters."

	 The UTF8String MUST not contain any octets with a value of 
         zero.

      DiameterIdentity
	 The DiameterIdentity format is derived from the OctetString AVP
         Base Format.  It uses the UTF-8 encoding and has the same
	 requirements as the UTF8String.  In addition, it must follow
         the following encoding rules.

            Diameter-Identity  = "diameter://" fqdn [ port ]
                                  [ transport ]

            fqdn               = Fully Qualified Host Name
  
            port               = ":" port
            ; If absent, the default Diameter port (TBD) is assumed.

            transport          = ";transport=" ( "tcp" | "sctp" )
            ; If absent, the default SCPT [26] protocol is assumed.

         The following are examples of valid DiameterIdentities:

            diameter://host.abc.com
            diameter://host.abc.com:6666
            diameter://host.abc.com;transport=tcp
            diameter://host.abc.com:6666;transport=tcp

      Enumerated
         Enumerated is Derived from the Unsigned32 AVP Base Format.
         This contains a list of valid values and their interpretation 
         and is described in the Diameter application introducing the AVP.

-mark