Subject: FYI: Apache Avro

From: Mark Nottingham <[email-masked]>
Date: Fri, 12 Jun 2009 23:11:58 +1000

Yet another data representation / schema / RPC:
   http://people.apache.org/~cutting/avro/

--
Mark Nottingham     http://www.mnot.net/

---

From: Tim Bray <[email-masked]>
Date: Fri, 12 Jun 2009 09:06:31 -0700

>  http://people.apache.org/~cutting/avro/

Has a bit of an ASN.1 feel to it.  But I'm sure Doug knows that
unfortunate history... -Tim

> _______________________________________________
> Apps-Discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 12 Jun 2009 14:27:19 -0500

> unfortunate history... -Tim

It looks like JSON-based syntax with a PER-like encoding, plus always
sending the schema with the data, so that one can always decode encoded
data (much as one always could do that with TLV encodings like BER/DER/
CER, only better since the schema gives you more info than the BER tags
ever did).

I can see why everyone wants their own syntax (ASN.1 is famously
difficult to parse, and JSON is easier to deal with than XML), but if
you're going to use a PER-like encoding, why not just use PER?  (Or XDR,
which is very much like a subset of PER with 4-octet alignment.)

It's like we are condemned to re-invent the wheel.

Nico
--

---

From: Alessandro Vesely <[email-masked]>
Date: Sun, 14 Jun 2009 16:47:17 +0200

> which is very much like a subset of PER with 4-octet alignment.)

A similar question has been answered (for Google's protocol buffers) 
in http://markmail.org/message/i7vkkeh3bycmwh4l . It seems one needs 
new names and specifications to refine previous ideas. (And this 
new-vs-updated dilemma has a special flavor on a IETF list.)

> It's like we are condemned to re-invent the wheel.

Looks like, but isn't. Nowadays it's easy to write applications that 
are trivially portable to different machines and OSes. Yet, it's not 
obvious how to port ideas across different dynamic environments. Given 
that the differences between avro and, say, XDR are undistinguished, 
what are the analogous of a source and a compiler that would allow to 
obtain both objects from the same idea?

Considering that we've been knowing that there is a gap between the 
physical word and our "metaphysical" minds since the days of Plato, 
one may say we haven't gone very far during the past millennia. 
However, computers have been invented as a proof of concept to show 
that thinking is a physical process, and the steps in their 
realization may suggest that our progress rate is growing...

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 16 Jun 2009 13:00:29 -0500

> new-vs-updated dilemma has a special flavor on a IETF list.)

That post has a lousy answer to the question it purports to answer.

AVRO sends the schema on the wire with every message (it could just send
a URL), and that is an answer to the issue in
http://markmail.org/message/i7vkkeh3bycmwh4l -- inventing a new encoding
is not a requirement for solving the problem that existing encodings
don't send enough information about the schema.

AVRO should have used PER with a JSON-based schema (instead of plain
ASN.1).  Instead AVRO has a new PER-like encoding that's not actually
PER (re-invention).  Re-inventing some things is easier to justify than
others.  ASN.1's syntax is hard to parse, therefore re-inventing it
based on XML (done) or JSON (now also done) makes sense (and would have
done the world a real favor!).  But re-inventing the encoding rules
doesn't really, not when so many encoding schemes have been tried over
the years...

...S-expressions, BER-like TLV, PER, XDR, NDR, XML, JSON, DataDumper,
... -- endless variations these three fundamental ones: S-expressions,
TLV and packed encoding.

> obvious how to port ideas across different dynamic environments. Given 

I don't follow this.

> obtain both objects from the same idea?

Why do you need both?  Why not re-use?  Anyways, the answer to your
question, in the ASN.1 world, of course, is ECN (enconding control
notation).  No, you don't want to use it -- it will do you no good.

> one may say we haven't gone very far during the past millennia. 

A silly argument for justifying re-invention, that.  The real reason we
re-invent, I suspect, is time pressures [+ ignorance [+ laziness]].

You have a deadline and a problem you should suspect others have solved
already.  What do you do?  You could solve it yourself in the time it
takes to write the code and docs, you could spend a sizeable portion of
that time (quite possibly more!) to find out how others have solved it
[over and over again], then adapt their solutions in the remaining time
(quite possibly more!).  (Adapting others' solutions can take a lot of
work since those will likely have impedance mismatches to your
development environment -- a C API but not a JavaScript one, a code
generator that imposes C struct layouts on you, and so on and on).

If you're a capable programmer the obviously simpler at first!) and much
faster approach is to write your own solution and to heck with re-use,
even if you might later come to wish you'd used off-the-shelf tools.

I think that's the explanation for re-invention.  Can I tell the
re-inventing programmer they're wrong?  Not really, or not necessarily.
Is re-invention a bad thing?  Not always -- sometimes the end result is
better than the predecessors.

Overall though, most re-invention is probably lousy, and the inescapable
conclusion, if you'll grant that premise, is that it is presumptuous to
re-invent the wheel without first researching previous inventions of it.
A software engineer should always do some research.

> realization may suggest that our progress rate is growing...

Practical computers were invented to solve practical problems, not to
model the world, or any philosophies.  Platonic object-oriented meta-
schema/schema/data models come fairly late in the development of
computers (after FORTRAN and LISP).  If your point is that we have the
same variety of philosophies in the world of computers as outside it, I
think I'll concede that point :)

Nico
--

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 16 Jun 2009 20:35:57 +0200

On Jun 16, 2009, at 20:00, Nicolas Williams wrote:

> AVRO should have used PER

What do you *like* about PER?
Honest question, since I have forgotten most of what I once knew about  
PER.

I seem to remember that the decision to adopt PER created much of the  
interop issues that finally killed H.323.
And I remember that one particularly ugly technical issue was the way  
PER handles extensibility.
But I'm sure willing to learn about PER's good side.

Gruesse, Carsten

---

From: "Randy Presuhn" <[email-masked]>
Date: Tue, 16 Jun 2009 11:54:00 -0700

Hi -

> From: "Carsten Bormann" <[email-masked]>
> To: "Nicolas Williams" <[email-masked]>
> Cc: <[email-masked]>; "Alessandro Vesely" <[email-masked]>
> Sent: Tuesday, June 16, 2009 11:35 AM
> Subject: Re: FYI: Apache Avro
...
> But I'm sure willing to learn about PER's good side.
...

For a protocol that doesn't need lots of open-ended extension
capabilities or object-identifier-like things, it offers:

(1) compact encoding
(2) lower processing cost

(2) often comes as a surprise, because the rules can seem rather
arcane.  But the elimination of redundant information in the
encoding means fewer error cases to check for, simplifying
the code path and eliminating lots of tests, as well as simply
reducing the number of bits that need to be shoveled around.

For a protocol where all the fields and ranges are carved out
in advance, this could be helpful.
But for something like SNMP, where most of the bandwidth
goes to object identifiers, it's reall no help at all.

Randy

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 16 Jun 2009 13:46:48 -0500

> Honest question, since I have forgotten most of what I once knew about  

It's not so much that I like it, but that AVRO's encoding is so similar
to it.  Why stop at being similar?

For example, the "zig zag" encoding of integers is sooo, odd.  Why??
Why not any of the heretofore tried and true encodings of integers?

> interop issues that finally killed H.323.

I've no knowledge of that.  And even if something like that happened it
may say nothing about the applicability of PER to AVRO -- what were the
interop issues about?

> PER handles extensibility.

I think it handles it quite nicely, actually.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 16 Jun 2009 14:16:24 -0500

> I think it handles it quite nicely, actually.

I should add, PER handles extensibility thusly (from memory): by adding
a single bit to the encoding of a value that indicates whether the type
was extended, and if it was there's a length field so that decoders that
don't know the extension can skip it.  Similarly, extensible constraints
on simple types are encoded as a single bit followed by the value in
constrained or unconstrained encoding according to the bit's value.
Extensible constraints on SEQUENCE OF are dealt with similarly.

Where BER encodes CHOICE implicitly via BER's tagging, PER indicates
which CHOICE arm was taken with a small integer that counts CHOICE arms
from 0 (ignores explicit and implicit tags altogether).  Presence/
absence of OPTIONAL or DEFAULT fields of a SEQUENCE is encoded as a
bitmask with as many bits as optional/default fields.  Etcetera.  All
this is well documented.  The ITU-T ASN.1 specs are very readable (more
so even than the various books on the topic, IMO).  AVRO's documentation
does not match the ITU-T ASN.1 specs in completeness or readability (it
didn't when I checked).

I.e., PER encodes extensions, presence/absence, in as few bits as
possible, as few as one bit when there's no extension/extended
constraint, or one bit plus length or variable-length value encoding
when there is.  Pretty clever.

XDR is very similar to PER, but much less compact, and without support
for certain ASN.1 features, though these can typically be modelled atop
XDR anyways (e.g., optional fields -> use of '*' in XDR, extensibility
markers -> opaque<> field or discriminated union with a single, default
void arm, ...).  AVRO's encoding is similarly similar to PER.

Nico
--

---

From: Alessandro Vesely <[email-masked]>
Date: Wed, 17 Jun 2009 10:55:38 +0200

Nicolas Williams wrote:
>> [The] differences between avro and, say, XDR are undistinguished
> 
> Why do you need both?  Why not re-use?

For practical issues, e.g. backward compatibility.

> Anyways, the answer to your

I.e. it's not quite practical or general.

> re-invent, I suspect, is time pressures [+ ignorance [+ laziness]].

That's correct, but is POV-specific.

> even if you might later come to wish you'd used off-the-shelf tools.

Agreed.

> better than the predecessors.

Yeah, someone wrote the code is good only after it has been rewritten 
at least three times.

> A software engineer should always do some research.

 From the other POV, existing solutions are not yet so good. E.g. not 
general or practical enough. Possibly, not sufficiently advertised.

>> that thinking is a physical process
> 
> computers (after FORTRAN and LISP).

Pascal, Babbage, Turing, and others came before that. Of course, the 
practical problem is what makes pressure, rises funds, and requires 
realistic schedules. (Babbage, for one, never actually built his 
analytic computer.) However, practical solutions would never have 
ventured into if the way hadn't been paved already.

The point in realizing what steps we took at what times was just to 
reassure that the wheel is actually a spiral, in the sense that there 
is some progress on each round of the loop. Correctly guessing where 
progress is directed may help.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 17 Jun 2009 11:09:11 -0500

> For practical issues, e.g. backward compatibility.

What does a new wheel have to be compatible with?

> Yeah, someone wrote the code is good only after it has been rewritten 

Supposing that's always true, then it should generally be better to fix
existing wheels to do what you want than to invent ones, except when the
existing wheels have had too many fixes and so don't work as well as new
ones might.

But also, consider that there are two levels of re-invention here:
implementation, and encoding specification.  If you can't find any tools
that produce a, say, PER encoder/decoder the way you want it, then you
can build a new tool to do that -- why build a new encoding
specification along the way??

> general or practical enough. Possibly, not sufficiently advertised.

But are you referring to implementation or specification?  (see above)

> progress is directed may help.

Not every addition is necessarily an improvement.  One should not
dismiss re-use on the theory that what comes next will be better than
what came before.  I fully expect _existing_ implementations of various
things to be useless to you, but not the protocols.

Consider this: something better than HTTP is certainly feasible, but
would you reject out of hand any suggestion that you use HTTP?  As you
say, practical considerations would almost certainly drive you to
embrace HTTP.  The world of encoding doesn't have a clear winner,
therefore it's easy to be unaware of the choices and invent new ones.

Whoever wrote AVRO's encoder should think about what justifications they
would have had for the choice they made, but with full knowledge of the
choices they really had.  It's possible that they'd conclude that
PER/XDR/NDR/... were each somewhat wide of the mark, it's almost certain
that they'd conclude that existing tools for those were wide of the
mark.  But I'm interested in the former, in why the existing encodings
were all inappropriate (if nothing else, so we can learn from that).

If it turns out that there are no good reasons for inventing a new
encoding, oh well, it'll only be one more in the huge pile that we have
already.  And knowing that will not help us much because there's no way
to transmit that knowledge to the next person in a position to re-invent
this wheel.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 17 Jun 2009 11:15:31 -0500

> reducing the number of bits that need to be shoveled around.

Indeed.  (2) is a surprise mostly because of the rules (which are not so
much arcane as detailed).  But if you look at protocols like IP, TCP,
..., ones with fixed-width fields and bags-of-options (often simple
enough to be described with ASCII art, and everyone agrees that their
encodings are simple to process), they are very similar in layout to the
PER encoding of an ASN.1 schema description of those protocols.  That's
because PER was designed to produce that sort of protocol.

> goes to object identifiers, it's reall no help at all.

Perhaps.  But my point isn't: retrofit PER everywhere because PER is soo
cool.  My point is: don't re-invent PER if that's what you need.  To me
it looks like AVRO's encoding has similar properties to PER and
gratouitous differences from it (i.e., it looks pointless).

Nico
--