Subject: Structured data over TCP?

From: Patrik Fältström <[email-masked]>
Date: Wed, 27 May 2009 07:46:13 +0200

It was many years ago popular to use line breaks as delimiters in  
data, and on each line have essentially attribute/value pairs.

Then we got XML that today seems to be what people want to use  
(because there are libraries, and tons of programmers that can (not?)  
implement XML parsers). We also have "content-length prefixed" XML  
data, that make things sort of simpler.

Sort of in parallell, we have xmlrpc, that to some degree solve  
slightly different problems.

Personally, I have lately looked at JSON, and after serializing data  
and then use Netstrings (http://cr.yp.to/proto/netstrings.txt), I must  
say I really like it.

For those that do not know Netstrings, here is an example from the spec:

> For example, the string "hello world!" is encoded as <31 32 3a 68 65  
> 6c 6c 6f 20 77 6f 72 6c 64 21 2c>, i.e., "12:hello world!,". The  
> empty string is encoded as "0:,".

Implementing is very simple, and you know how many bytes to read. An  
example JSON-netstring thing can look like (for a 3-object array with  
a hash as the third object, where the hash as one entry):

29:[1, "info", {"msg": "Hello"}],

In the DNS provisioning world we have come up with epp (RFC 3730) that  
has a binding to TCP (RFC 3734). The binding is a prefixed-based XML  
blob that is passed over the wire:

> EPP Data Unit Format (one tick mark represents one bit position):
>
>     0                   1                   2                   3
>     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                           Total Length                        |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>    |                         EPP XML Instance                      |
>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+//-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Now, we need some more protocols here (between DNS operator and the  
registrar), and the question is whether the same mechanism is  
recommended, or whether one should use something else?

Do Apps Area have any general feeling about this?

    Patrik

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 27 May 2009 18:56:32 +0900

Hello Patrik,

Some points, not necessarily in any particular order:

- I haven't been able to figure out what the trailing ',' in netstrings 
is for. Any idea?

- Netstrings don't say what encoding the data is in; that needs to be 
settled separately. XML can handle many character encodings if 
necessary, with sensible defaults, and there is no chance character 
encoding issues get forgotten. Json seems to be UTF-8 only in practice, 
which in most cases is about the best you can do these days.

- Json's data types are closer to the data types available in many 
(especially dynamic) programming languages when compared with XML. XML 
has the advantage that it can represent both data as well as (textual) 
documents. The later would have to be done with a hopeless hack in Json.

- Json can be directly interpreted in JavaScript. Some lazy developer 
may do it. But it would be an extremely bad idea, because it creates 
great security holes.

- Both Json and XML are widely used on the Web, in what's called Ajax. 
But there are usually no length prefixes in either case.

- There is RFC 3470 for the use of XML in the IETF. Please have a look 
if you haven't already done so. I'm not sure there is something similar 
for Json.

- You say "Now, we need some more protocols here", but to me, it looks 
more like you want to exchange some more data. Some people might 
disagree, but I think there is absolutely no need to invent a new 
protocol every time some data has to be transferred.

- Whether you call it "more protocols" or just "more data formats", if I 
were at either side of the data exchange, I wouldn't so much care about 
whether XML is easier to implement or Json or what, but I'd really hate 
to have a ton of different protocols/formats, each with a different 
technology. So unless the community involved thinks that this EPP Data 
Unit Format is hopelessly broken, I'd use the same scheme for all the 
other protocols/formats.

Regards,     Martin.

On 2009/05/27 14:46, Patrik Fältström wrote:
> It was many years ago popular to use line breaks as delimiters in data,
> and on each line have essentially attribute/value pairs.
>
> Then we got XML that today seems to be what people want to use (because
> there are libraries, and tons of programmers that can (not?) implement
> XML parsers). We also have "content-length prefixed" XML data, that make
> things sort of simpler.
>
> Sort of in parallell, we have xmlrpc, that to some degree solve slightly
> different problems.
>
> Personally, I have lately looked at JSON, and after serializing data and
> then use Netstrings (http://cr.yp.to/proto/netstrings.txt), I must say I
> really like it.
>
>> 6c 6c 6f 20 77 6f 72 6c 64 21 2c>, i.e., "12:hello world!,". The empty
>> string is encoded as "0:,".
>
> example JSON-netstring thing can look like (for a 3-object array with a
> hash as the third object, where the hash as one entry):
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Apps-Discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: John C Klensin <[email-masked]>
Date: Wed, 27 May 2009 07:55:20 -0400

Patrik,

More on this when I've had time to study the Netstrings spec,
but a quick history review for those who may not remember...

--On Wednesday, May 27, 2009 07:46 +0200 Patrik Fältström
<[email-masked]> wrote:

>...
>> For example, the string "hello world!" is encoded as <31 32
>> 3a 68 65   6c 6c 6f 20 77 6f 72 6c 64 21 2c>, i.e., "12:hello
>> world!,". The   empty string is encoded as "0:,".

This is one of the oldest string formats around.  Modulo the
fact that first-generation FORTRAN didn't have lower-case
characters, it would have been written as "12Hhello world!".
The machine-level coding of character strings on the
world-oriented machines of that generation would have been 

    First word: 00...1100 (binary, left padded, e.g., 12)
    Coded characters packed into as many subsequent words as
needed, with trailing padding.
    
On the one character-oriented machine of that generation I
remember, strings were delimited by special "word marks"; I
can't remember whether the length was stored as well.

In that generation, many of us learned that people really are
not good at counting characters, at least not if there are
significantly more than a half-dozen of them.  That led in only
slightly later generations of languages to, e.g., 'hello world!'
with the compiler doing the counting as needed, but
length-string pairs remained the most common form until C became
prevalent.  As you know, the DNS uses that form internally, so
it certainly isn't new to the Internet either.

Coming full circle?

    john

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 27 May 2009 14:39:42 +0200

On 27 maj 2009, at 11.56, Martin J. Dürst wrote:

> - I haven't been able to figure out what the trailing ',' in  
> netstrings is for. Any idea?

I like it being there, because if you end up being out of sync when  
parsing, you can find "the next string" by looking for ',' followed by  
[LEN] followed by a ':'.

> - Netstrings don't say what encoding the data is in; that needs to  
> be settled separately. XML can handle many character encodings if  
> encoding issues get forgotten. Json seems to be UTF-8 only in  
> practice, which in most cases is about the best you can do these days.

Correct.

> (especially dynamic) programming languages when compared with XML.  
> XML has the advantage that it can represent both data as well as  
> (textual) documents. The later would have to be done with a hopeless  
> hack in Json.

Agree. And I only talk about "structured data" in my case. I do not  
know how many discussions I have been in about XML (note: I am not an  
XML pro, the contrary) on whether things should be <x y="z">foo</x> or  
<x><y>z</y><q>foo</q></x> etc. And the mapping back and forth between  
a data structure and XML seems to be "kind of interesting". Parsing  
XML seems to be ok, but generating...

The JSON libraries I have been working with lately very very easily  
both produces and parses the serialized data.

> - Json can be directly interpreted in JavaScript. Some lazy  
> developer may do it. But it would be an extremely bad idea, because  
> it creates great security holes.

Agree.

> - Both Json and XML are widely used on the Web, in what's called  
> Ajax. But there are usually no length prefixes in either case.

Ok, I do not know enough about this.

That is why I propose the addition of netstring format for the  
serialized JSON data. And that is why the XML blob in epp as a length  
prefix.

> - There is RFC 3470 for the use of XML in the IETF. Please have a  
> look if you haven't already done so. I'm not sure there is something  
> similar for Json.

I will look again at it.

> - You say "Now, we need some more protocols here", but to me, it  
> looks more like you want to exchange some more data. Some people  
> might disagree, but I think there is absolutely no need to invent a  
> new protocol every time some data has to be transferred.

I agree with you here, but the is your point that epp should be used  
also between DNS provider and registrar?

FYI: epp is _extremely_ complex as the data model is "interesting". A  
normal DNS operator definitely do not need that power. But they might  
need other things. That is not included in the epp protocol (but could  
theoretically be managed by extensions).

epp is very very optimized for the registry/registrar model.

> - Whether you call it "more protocols" or just "more data formats",  
> if I were at either side of the data exchange, I wouldn't so much  
> care about whether XML is easier to implement or Json or what, but  
> I'd really hate to have a ton of different protocols/formats, each  
> with a different technology. So unless the community involved thinks  
> that this EPP Data Unit Format is hopelessly broken, I'd use the  
> same scheme for all the other protocols/formats.

I would say "no" to use of epp between dns operator and registrar.

    Patrik

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 27 May 2009 14:43:26 +0200

> Coming full circle?

I think so, as I have lost count on how many times I have seen quoting  
issues in XML data passed around...counting bytes is sometimes pretty  
simple...and together with the JSON format that, as Martin stated so  
well, is better for data structures (but not as good at text)...that  
was something I liked.

But, this is exactly why I also enjoy this discussion. I think the  
discussion is need.

    paf

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 27 May 2009 15:07:05 +0200

On 27 maj 2009, at 14.47, Hollenbeck, Scott wrote:

> Some of the current mappings are definitely optimized for that  
> model, but I'm not so sure that the same can be said of the protocol  
> core itself.  If one were to throw out the domain and host mappings,  
> for example (instead of trying to extend them), and create a new DNS  
> operator mapping, couldn't the more "interesting" aspects of the  
> registry/registrar model be overcome?

Agree.

The complex data structure of epp is because you have domain, contact  
and hosts as separate objects, and then sponsoring clients for each  
one of them. I.e. the data model is kept coherent by pointers in the  
objects that the client have to manage themselves.

Specifically when the registries have set up their own models on what  
happens when transfers of objects are done etc.

A simpler model is possible when someone that have authenticated have  
access to certain domains (that they are holder or techc for, for  
example). Then it is possible to in a very simple way "just" upload DS  
for the domain (and that command is replacing _all_ ds that is  
stored). The XML for doing that operation is quite complex in epp...  
Just because epp is a generalized protocol for the operations a  
registrar is to make in a registry (transfer, register, update records  
etc).

My JSON blob for replacing DS data for a domin do for example look  
like this (with one key only):

{"dsdata": [{"keyTag": "59344", "digestType": "1", "digest":  
"529C963AFEA3F938B87E277405072316639B18B0", "alg": "5"}], "domain":  
"example1.se"}

This results in sort of the following two epp commands:

<command>
   <update>
     <domain:update xmlns:domain="urn:ietf:params:xml:ns:domain-1.0"
             xsi:schemaLocation="urn:ietf:params:xml:ns:domain-1.0
             domain-1.0.xsd">
       <domain:name>example1.se</domain:name>
     </domain:update>
   </update>
   <extension>
     <secDNS:update xmlns:secDNS="urn:ietf:params:xml:ns:secDNS-1.0"
             xsi:schemaLocation="urn:ietf:params:xml:ns:secDNS-1.0
             secDNS-1.0.xsd">
       <secDNS:add>
         <secDNS:dsData>
           <secDNS:keyTag>0</secDNS:keyTag>
         </secDNS:dsData>
       </secDNS:add>
     </secDNS:update>
   </extension>
   <clTRID>46645-013555262222</clTRID>
</command>

<command>
   <update>
     <domain:update xmlns:domain="urn:ietf:params:xml:ns:domain-1.0"
             xsi:schemaLocation="urn:ietf:params:xml:ns:domain-1.0
             domain-1.0.xsd">
       <domain:name>example1.se</domain:name>
     </domain:update>
   </update>
   <extension>
     <secDNS:update xmlns:secDNS="urn:ietf:params:xml:ns:secDNS-1.0"
             xsi:schemaLocation="urn:ietf:params:xml:ns:secDNS-1.0
             secDNS-1.0.xsd">
       <secDNS:add>
         <secDNS:dsData>
           <secDNS:keyTag>59344</secDNS:keyTag>
           <secDNS:alg>5</secDNS:alg>
           <secDNS:digestType>1</secDNS:digestType>
           <secDNS:digest>529C963AFEA3F938B87E277405072316639B18B0</ 
secDNS:digest>
         </secDNS:dsData>
       </secDNS:add>
     </secDNS:update>
   </extension>
   <clTRID>46645-013555262222</clTRID>
</command>

    Patrik

---

From: "Hollenbeck, Scott" <[email-masked]>
Date: Wed, 27 May 2009 08:47:42 -0400

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Patrik Fältström
> Sent: Wednesday, May 27, 2009 8:40 AM
> To: "Martin J. Dürst"
> Cc: [email-masked]
> Subject: Re: Structured data over TCP?

[snip]

I'm more than a disinterested third party when it comes to EPP, so please take my comments with an appropriately-sized grain of salt.

> I agree with you here, but the is your point that epp should 
> be used also between DNS provider and registrar?
> 
> FYI: epp is _extremely_ complex as the data model is 
> "interesting". A normal DNS operator definitely do not need 
> that power. But they might need other things. That is not 
> included in the epp protocol (but could theoretically be 
> managed by extensions).
> 

Some of the current mappings are definitely optimized for that model, but I'm not so sure that the same can be said of the protocol core itself.  If one were to throw out the domain and host mappings, for example (instead of trying to extend them), and create a new DNS operator mapping, couldn't the more "interesting" aspects of the registry/registrar model be overcome?

-Scott-

---

From: "Vijay K. Gurbani" <[email-masked]>
Date: Wed, 27 May 2009 08:37:44 -0500

John C Klensin wrote:
> This is one of the oldest string formats around.  
[...]

This is very interesting.  The format specified by Patrik
is known to me as bencoding (binary encoding) as used by
BitTorrent.  While BitTorrent does not create a hex
representation, "Hello World!" will be bencoded as
12:Hello World!, and an integer 3 base 10 will be encoded
as i3e ('i' for integer and 'e' delimiting end of integer).
There are similar constructs to represent lists and
dictionaries as well.

I had always assumed that this was a BitTorrent-specific
format, but it appears that the format has a rich history.

Thanks,

- vijay
-- 
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60566 (USA)
Email: vkg@{alcatel-lucent.com,bell-labs.com,acm.org}
Web:   http://ect.bell-labs.com/who/vkg/

---

From: Andrew Newton <[email-masked]>
Date: Wed, 27 May 2009 09:43:46 -0400

On May 27, 2009, at 8:39 AM, Patrik Fältström wrote:

> serialized JSON data. And that is why the XML blob in epp as a  
> length prefix.

Just a guess here, but this is probably due to AJAX using HTTP and  
HTTPs content length header.

-andy

---

From: Tony Finch <[email-masked]>
Date: Wed, 27 May 2009 16:57:35 +0100

On Wed, 27 May 2009, Patrik Fältström wrote:
>
> counting bytes is sometimes pretty simple...

Implementations of byte-count protocols have a history of security
vulnerabilities associated with framing errors. It's evidently more
difficult than it seems.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 27 May 2009 10:49:52 -0500

> Do Apps Area have any general feeling about this?

You left out a great many other encodings (like all the ASN.1 encodings,
XDR, ...).  It's all the same to me.  Typically the main issue is: can
the selected encoding be easily implemented by the existing
implementors and new interested implementors?

Typically the answer to that question in regards to ASN.1 encodings is
"no".  The lack of a good, simple syntax+encoding with lots of popular
opensource tools tends to result in every new protocol having a new
encoding :(

A few examples:

 - LDAP       -> ASN.1/DER
 - IMAP       -> text-based protocol
 - SSHv2      -> informal syntax + custom encoding
 - DNS        -> ASCII art representations of bits-on-the-wire as syntax
                 and hand-crafted encoding
 - DIGEST-MD5 -> ABNF syntax with straightforward encoding derived from
                 the ABNF

I'm afraid it's really the wild west out there: pick an off-the-shelf
syntax/encoding, or just make one up.

I tend to prefer the use of a formal syntax -- ASN.1, ABNF, XDR, ...

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Wed, 27 May 2009 09:17:10 -0700

> Do Apps Area have any general feeling about this?

I was going to start typing in a lengthy answer but then I remembered
I gave a tutorial at IETF70 on the choice of data formats for use in
Internet Protocols.  It's centered around XML but I think covers many
of the other issues that have come up in this.  Available for your
reading pleasure at http://www.tbray.org/tmp/IETF70.pdf

 -Tim

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 27 May 2009 11:05:41 -0500

> vulnerabilities associated with framing errors. It's evidently more

<generalization take_with="grain of sand">

The more redundancy in the encoding, the more bugs in the resulting
encoder/decoder code.  The TLV encodings of ASN.1 are a good example;
compare to less chatty encodings like PER or XDR, or to hand-crafted
formats like IP and TCP headers.  XML is very chatty, but being text
based helps, I think.  Nesting structures (bags of stuff) probably hurts
too, particularly when variable-length arrays of items with
variable-length encodings are involved (e.g., security bugs in XDR have
mostly involved XDR arrays).

</generalization>

Nico
--

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 27 May 2009 19:44:27 +0200

> HTTPs content length header.

I think so, but with XML you could also sort of "parse until you hit  
the end tag", although I guess that is dangerous.

Anyone written the actual code that slurp XML from the wire?

What did you do?

    paf

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 27 May 2009 12:47:35 -0500

> format, but it appears that the format has a rich history.

There is nothing new in the world of data serialization :)

In LISP S-expressions are the data serialization format, and
S-expressions are also the form of all LISP programs.  It doesn't get
much simpler than that.  And that goes back many decades.  Throw in the
destructuring-bind macro and you have something not unlike a JSON
equivalent of XPath.

Perl Data::Dumper, Icon's whatever it was called, etcetera -- all of
these are just the equivalents of LISP's serialization scheme, but for
languages where the program is written in a very different syntax.  The
Icon serializer produces output very similar to what you describe is
used in BitTorrent.

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Wed, 27 May 2009 11:05:42 -0700

2009/5/27 Patrik Fältström <[email-masked]>:

> I think so, but with XML you could also sort of "parse until you hit the end
> tag", although I guess that is dangerous.

That was a semi-explicit design goal: that you could shut the network
connection down when you saw the end-tag and not worry if the other
end was being sluggish in closing the connection.  I used to emphasize
this a lot when I was Global Chief XML Salesguy, and people always
nodded their heads.

> Anyone written the actual code that slurp XML from the wire?

Yes.  And despite the point above, I've never seen an implementation
that actually does close the input stream when it sees the end-tag :)

In practice it doesn't seem to have been a problem.  -Tim

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 27 May 2009 11:28:41 -0700

> parsing, you can find "the next string" by looking for ',' followed by 

Count-versus-parse is an ancient choice.  As computer scientists, we like the 
determinism and simplicity and presumed efficiency of counting to the end (or 
using the count to jump directly to the end)  rather than scanning for a 
delimiter.

So it's unfortunate to discover that it frequently is a poor choice.  For 
example, in a world of serial communications, parsing isn't necessarily more 
expensive.  And it tends to be more robust, as noted in the above comment.  And 
also as noted earlier, humans don't count very well. So for a scheme that 
considers the full "lifecycle" of use including human actors, a parsing-based 
scheme often is actually simpler.

On the average, in application space, the human factor tends to dominate the 
choice, or should, IMO.

Of course, it's possible to have a parsing syntax that is ridiculously verbose, 
and one can argue that XML crosses that line.  The only problem is that XML is 
so popular, the claim that it is too verbose needs some diligent defense.  More 
generally, any existing scheme that has wide deployment gets the benefit of 
familiarity.

So I think any new scheme needs to make a compelling case for superiority that 
overcomes the community's lack of familiarity with it and, therefore, the 
considerable community cost of adopting it.

Indeed, netstrings appears to be a variant of a very old scheme that has had 
some use but has not remained popular.  For all the apparent benefits of the 
scheme, one should worry about why its model has not been very popular for 
applications.

One should be especially curious about a counting scheme that includes scanning 
constructs.  While sounding nicely robust, it raises a basic question about the 
actual utility of the counting.  After all, if you still have to build and test 
the mechanism that uses the scanning construct, why bother with counting?

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Tony Finch <[email-masked]>
Date: Wed, 27 May 2009 21:02:14 +0100

On Wed, 27 May 2009, Dave CROCKER wrote:
>
> One should be especially curious about a counting scheme that includes
> scanning constructs.  While sounding nicely robust, it raises a basic question
> about the actual utility of the counting.  After all, if you still have to
> build and test the mechanism that uses the scanning construct, why bother with
> counting?

Netstrings are designed to be human-readable (hence the textual encoding
of the number and the parser-friendliness) but they still support
unencoded binary payloads (no need for byte stuffing).

Tony.

---

From: John C Klensin <[email-masked]>
Date: Wed, 27 May 2009 17:05:52 -0400

--On Wednesday, May 27, 2009 11:28 -0700 Dave CROCKER
<[email-masked]> wrote:

>...
> Count-versus-parse is an ancient choice.  As computer
> scientists, we like the determinism and simplicity and
> presumed efficiency of counting to the end (or using the count
> to jump directly to the end)  rather than scanning for a
> delimiter.
> 
> So it's unfortunate to discover that it frequently is a poor
> choice.  For example, in a world of serial communications,
> parsing isn't necessarily more expensive.  And it tends to be
> more robust, as noted in the above comment.  And also as noted
> earlier, humans don't count very well. So for a scheme that
> considers the full "lifecycle" of use including human actors,
> a parsing-based scheme often is actually simpler.

On the other hand, if the structure is long or might involve
nesting, parsing-based systems risk loss of state or miscounting
delimiters, either of which can have catastrophic results.
Counting systems can also fail if systems miscount or some items
are lost, but that is a different sort of failure.   Often the
right solution is a presentation form that uses delimiters,
works for users on input and has an intuitive interpretation on
display and a transmission (or protocol-level) form that uses
counts.  It is perhaps not an accident that describes what is
done with the DNS.

>...
> One should be especially curious about a counting scheme that
> includes scanning constructs.  While sounding nicely robust,
> it raises a basic question about the actual utility of the
> counting.  After all, if you still have to build and test the
> mechanism that uses the scanning construct, why bother with
> counting?

Perhaps more important, if one can determine the end of the same
string by _either_ counting or scanning, one immediately has to
confront the question of what to do if they yield inconsistent
results, e.g., whether that is an error condition that
invalidates the string (or data structure) or, if one takes
priority, what the other one means.  I think the technical term
for that sort of situation is "big mess".

I agree with Dave that this is an ancient choice.  I also
suggest (as Dave does indirectly) that we have made most of the
possible mistakes several times already and that we should try
to avoid reinventing them again if possible.

    john

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 27 May 2009 15:58:09 -0700

John C Klensin wrote:
> On the other hand, if the structure is long or might involve nesting,
> parsing-based systems risk loss of state or miscounting delimiters, either of
> which can have catastrophic results.

That's a fragility argument.  As Patrik commented for, the comma in the
count-based scheme was for recovery from possible loss of parsing state.  So
count-based systems aren't immune from the problem.  Overall, I'm hard-pressed
to believe that ASN.1 is more robust than, say, XML or ABNF.

> Counting systems can also fail if systems miscount or some items are lost,
> but that is a different sort of failure.   Often the right solution is a
> presentation form that uses delimiters, works for users on input and has an
> intuitive interpretation on display and a transmission (or protocol-level)
> form that uses counts.  It is perhaps not an accident that describes what is 
> done with the DNS.

DNS uses small, rather simple data objects.  That makes the counts model trivial.

SNMP uses ASN.1, but that choice was forced onto the SNMP folks in a misguided
effort to make its data object compatible with CMIP and was complained about for
years by the SNMP folk.

As for using different schemes between presentation versus transfer, that
invites translation errors.  Besides that, the human-computer interface of 
"display" and "input" are their own environments, not needing an "exchange" 
encoding standard.  If the suggestion is to map from user input to some sort of 
parsed presentation-layer format and then map that down to some sort of 
count-based transfer-layer format, I don't understand the benefit.  Again, it 
invites translation error, as well as adding the overhead of the extra 
translation step.

Ultimately, my own view about new meta-formats is that they need to supply a 
compelling, practical value proposition.  A number of existing meta-formats have 
an enormous installed based of software and experience and clearly function 
robustly and efficiently.

It's not that one can't improve on that.  It's that any claim of improvement 
needs to be compellingly practical.  Not just as an exercise in cleanliness or 
efficiency, but in fixing some current problem among the available choices.

d/

-- 

   Dave Crocker

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 28 May 2009 09:46:27 +0900

It should be mentioned that although C (with string constants) and C's 
standard libraries do use a null byte delimiter to indicate string 
length, I think the general practice for 'fat' libraries and other 
languages implemented on top of C has been to use initial length 
indicators rather than null byte delimiters, for quite a long time. 
Initial length indicators are more efficient in many algorithms, for 
long strings, C's strlen() is quite inefficient. So it's less of a 'full 
circle' and more of a "we always have had both".

But the advantages and disadvantages of these two techniques for 
internal processing and the advantages and disadvantages for data 
transfer are not exactly the same. I think a main reason why some main 
IETF protocols (e.g. HTTP) don't use initial length indicators is that 
these aren't suited for streaming, where you don't know how much data 
you're going to send when you start. If streaming isn't a concern for 
the data exchanges you are considering, then initial length indicators 
may be fine.

Regards,    Martin.

On 2009/05/27 20:55, John C Klensin wrote:
>>> For example, the string "hello world!" is encoded as<31 32

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 28 May 2009 10:17:20 +0900

On 2009/05/28 7:58, Dave CROCKER wrote:

> John C Klensin wrote:
>> It is perhaps not an accident that describes
>> what is done with the DNS.
>

Also, DNS is extremely widely deployed, and extremely central to the 
operation of the Internet, that having double layers (a syntax for 
users/zone files/... and another on the wile), even if maybe it wasn't 
the best idea, has survived. I'm not sure that less important protocols 
with such a double layer would have that much success. In other words, I 
suspect that DNS is successful despite this double layer, not because of 
it. [The only positive reason I can see for the double layer is that 
DNS, much, much more than the higher-level data exchanges that we are 
discussing here, gains a lot from small data sizes that can be sent in a 
single packet over UDP.)

> invites translation errors. Besides that, the human-computer interface
> of "display" and "input" are their own environments, not needing an
> "exchange" encoding standard. If the suggestion is to map from user
> input to some sort of parsed presentation-layer format and then map that
> down to some sort of count-based transfer-layer format, I don't
> understand the benefit. Again, it invites translation error, as well as
> adding the overhead of the extra translation step.

Agreed.

> Ultimately, my own view about new meta-formats is that they need to
> supply a compelling, practical value proposition. A number of existing
> meta-formats have an enormous installed based of software and experience
> and clearly function robustly and efficiently.

I agree. Given the wide usage of Json in Ajax, I wouldn't call it a new 
format, although of course it's good to look at whether the advantages 
for Ajax translate into advantages in other usages.

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Thu, 28 May 2009 10:25:39 +0900

On 2009/05/28 2:44, Patrik Fältström wrote:
> I think so, but with XML you could also sort of "parse until you hit the
> end tag", although I guess that is dangerous.

Any XML parser essentially does that. I don't know of any XML parser 
which you can tell "Read that data stream, but only up to byte X". The 
question is how to integrate the XML parser with the layer that provides 
the length information.

The easiest way is to read the length info, then read the rest of the 
data (up to that length) into a buffer of that length, then hand that 
buffer over to the XML parser. That works with smaller data, but not 
with really large data (what is small and what is large may vary 
depending on context).

Parse until you hit the end tag isn't more dangerous with a correctly 
implemented XML parser than parsing a really, really long XML document; 
with both, you can run out of memory.

With regards to the comma as a delimiter for netstring, I think selling 
it as a means to synchronize is a bad idea. If you care about security 
you don't want to synchronize, you want to stop processing.

[One good thing about XML is that it is very clear that any kind of 
error is an error, and processing stops. Json may not be that clear.]

Regards,    Martin.

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 27 May 2009 22:16:51 -0700

Hi -

This is an aside to the real point of this thread....

> From: "Dave CROCKER" <[email-masked]>
> To: <[email-masked]>
> Sent: Wednesday, May 27, 2009 3:58 PM
...
...

No, no, no.   (1) The "compatibility" between SNMP and CMIP was never
more than window dressing or propaganda.  (I've implemented both.)
The grammars of the two protocols are surprisingly different, as are the
information models.  One can kludge gateways, but the alleged
compatibilities are of no help whatsoever in doing it.

(2)  The use of BER certainly was not an issue - there were no serious
efforts to use some other encoding scheme when it came time to do
SMP, or SNMPv2, or SNMPv2p, or SNMPv2*, or SNMPv2c, or SNMPv3.

(3) SNMP's SMIv1 was a good demonstration of what ASN.1 didn't do
well as a data modeling language, which was why SMIv2 (just like
CMIP's GDMO) was *not* built on the ASN.1 language.  But this
was really independent of the use of (an adapted subset of) BER
for the encoding.

Randy

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 28 May 2009 11:02:30 +0200

On 28 maj 2009, at 02.46, Martin J. Dürst wrote:

> I think a main reason why some main IETF protocols (e.g. HTTP) don't  
> use initial length indicators is that these aren't suited for  
> streaming, where you don't know how much data you're going to send  
> when you start. If streaming isn't a concern for the data exchanges  
> you are considering, then initial length indicators may be fine.

Agree, in many protocols you absolutely want to be able to start  
sending data without buffering "the whole thing". Look at message/ 
partial content-type.

When writing code that reads from a socket, I really enjoy knowing how  
many bytes I have to read, so I can "just" ask the socket  
implementation for that many bytes. And not write that parser that  
read one byte at a time...phew...so netstring or epp-like length  
prefixed things is something I like.

Then, the reason I like JSON was simply the difference stated in the  
excellent presentation by Tim, that XML has advantages (document-like  
things, internationzation etc), but when "just" wanting structured  
data "structs", and always use UTF-8, well...JSON is really really  
nice. Maybe because I as a programmer much faster understood how the  
JSON libraries work than XML libraries ;-) Yes, I talk one second on  
implementation from scratch, and another about reusing libraries.

    Patrik

---

From: Tony Finch <[email-masked]>
Date: Thu, 28 May 2009 14:56:33 +0100

> > I think a main reason why some main IETF protocols (e.g. HTTP) don't use
> > initial length indicators is that these aren't suited for streaming, where
> > you don't know how much data you're going to send when you start.

That's not a particularly good example. In most situations HTTP uses
Content-Length for delimiting entity bodies using an initial count. It
also supports a chunked Content-Transfer-Encoding for streaming purposes,
where each chunk has an initial count.

HTTP's original framing relied on closing the underlying connection, which
isn't reliable.
http://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable

HTTP does not have any kind of delimiter-based framing. Dot stuffing as
used in SMTP and NNTP is a better example.

On Thu, 28 May 2009, Patrik Fältström wrote:

> Agree, in many protocols you absolutely want to be able to start sending data
> without buffering "the whole thing". Look at message/partial content-type.

The message/partial content type is designed for getting large media
through relays with low message size limits. It has nothing to do with
streaming.

Tony.

---

From: Patrik Fältström <[email-masked]>
Date: Thu, 28 May 2009 16:08:42 +0200

On 28 maj 2009, at 15.56, Tony Finch wrote:

>> sending data

I was thinking of the tagging of the message parts. Only the last part  
have to have an indication of how many parts there are (total). You do  
not have to know that when you receive data, start to split in pieces:

> Three parameters must be specified in the Content-Type field of type
> "message/partial":  The first, "id", is a unique identifier, as close
> to a world-unique identifier as possible, to be used to match the
> fragments together. (In general, the identifier is essentially a
> message-id; if placed in double quotes, it can be ANY message-id, in
> accordance with the BNF for "parameter" given in RFC 2045.)  The
> second, "number", an integer, is the fragment number, which indicates
> where this fragment fits into the sequence of fragments.  The third,
> "total", another integer, is the total number of fragments.  This
> third subfield is required on the final fragment, and is optional
> (though encouraged) on the earlier fragments.  Note also that these
> parameters may be given in any order.

    Patrik

---

From: Alessandro Vesely <[email-masked]>
Date: Fri, 29 May 2009 15:45:45 +0200

> reading pleasure at http://www.tbray.org/tmp/IETF70.pdf

That gives 404 Not Found. You probably meant 
http://www.ietf.org/proceedings/07dec/slides/xmltut-0.pdf

Yes, it is centered on XML. May I ask why is JSON only suggested if 
"[t]he expected lifetime of the data is short"?

Besides 
http://nih.blogspot.com/2008/09/ive-been-asked-lot-about-use-of-soap-in.html
also Google is apparently going to dismiss SOAP, as reported in 
http://blog.feedly.com/2009/03/03/jsonrest-vs-xmlsoap/. Although there 
is a javascript library for SOAP in 
http://www.codeplex.com/JavaScriptSoapClient most javascript toolkits 
(see http://en.wikipedia.org/wiki/Comparison_of_JavaScript_frameworks) 
use JSON only. Does that tendency further decrease XML's attractiveness?

Thanks for the tutorial
Ale

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 29 May 2009 12:50:31 -0500

> "[t]he expected lifetime of the data is short"?

JSON assumes that the client and server agree on a schema.  With XML the
schema that the document complies with is referenced by the document
itself.

Therefore, if you have a volatile API then you need to have a way to do
versioning if you use JSON.  Then again, if you have data that has a
long at-rest lifetime, then you really want to specify its schema
somewhere, and once you're at that point you're better off with XML.

Or at least I think that would be the argument.

> use JSON only. Does that tendency further decrease XML's attractiveness?

Just because SOAP uses XML doesn't mean that using XML implies using
SOAP.

There's now a JSON DOM, or work on it, which I expect will increase
JSON's attractiveness.  But in general I expect that _today_ JavaScript
that edits the page's DOM will typically use XML documents retrieved
with XMLHttpRequest for the simple reason that that's the simplest thing
to do.  A JSON DOM may change that, for all I know.

Nico
--

---

From: Tim Bray <[email-masked]>
Date: Fri, 29 May 2009 11:30:25 -0700

> schema that the document complies with is referenced by the document

Neither of these assertions is correct.  Good client and server
designers will, where possible, try to decouple from the message
details as much as possible.  For example, in the case of JSON, send
things as dicts and check to see whether the keys you need are there.
And in the case of XML, the schema is not necessarily present, and
including a pointer to one is considered bad practice by many
including me, and furthermore, schema-based message validation at
runtime is highly unusual.

> somewhere, and once you're at that point you're better off with XML.

You need to have a way to version (or make an explicit choice not to)
in any protocol-design exercise.  It's not clear that XML-vs-JSON
either helps or hurts.

The argument for XML in the case of archival formats is that the label
density is greater, so it's a bit more self-describing, so may
preserve its usability longer.

 -Tim

---

From: Alessandro Vesely <[email-masked]>
Date: Sat, 30 May 2009 10:50:37 +0200

>> somewhere, and once you're at that point you're better off with XML.

In both cases one can learn the field names from a data snippet. 
Semantics, rules, constraints, and similar stuff, require a spec.

> in any protocol-design exercise.  It's not clear that XML-vs-JSON

Versioning the data model may be loosely related with procedural 
requirements. Not that SOAP-vs-REST is a better conundrum... a 
framework might provide either web service transparently (e.g. 
that's what wso2 is apparently trying to achieve.)

> preserve its usability longer.

However, the actual data on disk does not have to be laid down in 
the same format as the representation that is exchanged over TCP. An 
app may use the internal storage means it prefers, according to the 
amount of data and the read/write ratio.