Subject: [netmod] identifier length survey

From: Andy Bierman <[email-masked]>
Date: Sat, 28 Mar 2009 18:25:57 -0700

Hi,

There seems to be 3 options wrt/ identifier length
in the YANG language:

   1) 1 .. infinity  (no rule at all)
   2) 1 .. N         (MUST for every module:
                      "identifiers MUST be 1 to N chars long")
   3) 1 .. N         (MUST only for standard modules:
                      current text: "tools MUST accept 1 to N chars")

Which do you prefer and why?

If the result is (2) or (3), then the WG needs to pick a value for N,
which will hopefully be 63.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sun, 29 Mar 2009 08:08:27 +0200

>                       current text: "tools MUST accept 1 to N chars")

3) is actually "1..infinity but at least N must be supported"

For me, 2) means an identifier>N is a hard error while 3) means an
implementation can support longer identifiers but will generate a
warning if they are used.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: "Bert Wijnen \(IETF\)" <[email-masked]>
Date: Sat, 28 Mar 2009 23:22:46 -0700

In SNMP, we specified that there could be no more than 128 OID subids.
In the SMI, I don;t think we set a limit.
But the fact that the base protocol defined alimit; that fact has caused us to have
to deal with things like InetAddress as an INDEX ... becaus etheroretically that can
be more than 128.

Whatever we do, I would hope that we do not "hardcode" any limit into the base
protocol/spec. Rather, if we do limits, I would make them as current "rules" or some
such that we ould (if needed) relax in the future.

With MIB descriptors, we have no SMI defined limit. We did have originally a limit of 32
for IETF std track MIB modules. We later extended that to 64. And if needed we can
extend it to more.

I think I would prefer a similar approach in YANG.

Bert
  ----- Original Message ----- 
  From: Andy Bierman 
  To: NETMOD Working Group 
  Sent: Saturday, March 28, 2009 6:25 PM
  Subject: [netmod] identifier length survey

  Hi,

  Andy

  _______________________________________________
  netmod mailing list
  [email-masked]
  https://www.ietf.org/mailman/listinfo/netmod

---

From: Phil Shafer <[email-masked]>
Date: Sun, 29 Mar 2009 09:59:28 -0400

>Which do you prefer and why?

I prefer (1) since it does not impose an artificial limit.  This
option follows the path of XML, which has no limit on the length
of XML element names, XML Namespace strings, etc.

Thanks,
 Phil

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sun, 29 Mar 2009 16:11:54 +0200

On Sun, Mar 29, 2009 at 08:22:46AM +0200, Bert Wijnen (IETF) wrote:

> In SNMP, we specified that there could be no more than 128 OID
> subids.  In the SMI, I don;t think we set a limit.  But the fact
> that the base protocol defined alimit; that fact has caused us to
> have to deal with things like InetAddress as an INDEX ... becaus
> etheroretically that can be more than 128.

The OID limit is not comparable to YANG identifiers; if at all, a
sub-identifier would be comparable to a YANG identifier. And
YANG/NETCONF does not require to encode key values into the name.

> Whatever we do, I would hope that we do not "hardcode" any limit
> into the base protocol/spec. Rather, if we do limits, I would make
> them as current "rules" or some such that we ould (if needed) relax
> in the future.
>
> With MIB descriptors, we have no SMI defined limit. We did have
> originally a limit of 32 for IETF std track MIB modules. We later
> extended that to 64. And if needed we can extend it to more.

RFC 2578:

   For all descriptors appearing in an information module, the
   descriptor shall be unique and mnemonic, and shall not exceed 64
   characters in length.  (However, descriptors longer than 32
   characters are not recommended.)

I think we do have an SMI defined limit and I am not sure how easy it
is to extend (an RFC overwriting this part part of RFC 2578 would be
needed I think).

There are notable differences between YANG and the SMI here. YANG
identifiers show up as XML elements names while SMI descriptors have
no impact on the wire. In both cases, identifiers / descriptors are
translated to source code and there are usually restrictions (in C99
for example, only 63 characters are significant - this is where the 63
comes from). Of course, compilers that translate to other formats may
have to do some name mangling if the target identifier length is
shorter than some N.

/js

---

From: Andy Bierman <[email-masked]>
Date: Sun, 29 Mar 2009 08:42:39 -0700

> have to do some name mangling if the target identifier length is

I still do not understand why 63 characters is not long enough.
In 21 years and 100,000s of MIB objects, not one person
has ever wanted an SMIv2 object descriptor that even reached 63
characters long.

Now some people think YANG needs to support infinite length descriptors?
What is the use-case?

Here's another real limit for IETF MIBs: 72.
Identifiers cannot be quoted, and only quoted strings
can be split across multiple lines, and an RFC line cannot
exceed 72 characters.

Andy

---

From: "Bert Wijnen \(IETF\)" <[email-masked]>
Date: Sun, 29 Mar 2009 09:07:31 -0700

I know OIDs are not relevant.
I was trying to get across that I prefer to leave the formal spec flexible
(so 1..infinity) and then define guidances (aka: no longer than 63 is
recommended).

Bert
  ----- Original Message ----- 
  From: Juergen Schoenwaelder 
  To: Bert Wijnen (IETF) 
  Cc: Andy Bierman ; NETMOD Working Group 
  Sent: Sunday, March 29, 2009 7:11 AM
  Subject: Re: [netmod] identifier length survey

  RFC 2578:

  /js

---

From: Andy Bierman <[email-masked]>
Date: Sun, 29 Mar 2009 09:17:38 -0700

> (so 1..infinity) and then define guidances (aka: no longer than 63 is

Are you suggesting a change to the current text?
Is anybody?  If not, then this tar-pit is played out, right?

Andy

---

From: Andy Bierman <[email-masked]>
Date: Sun, 29 Mar 2009 09:24:01 -0700

> warning if they are used.

For me, the problem with (3) is that it is a rule written
in terms of what a tool MUST accept, but every other rule
in the YANG spec is specified in terms of what a YANG author MUST write.
That is why I wanted to change it in the first place.
If the rule was formulated the proper way, it would
be much easier to understand.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Sun, 29 Mar 2009 09:37:50 -0700 (PDT)

> can be split across multiple lines, and an RFC line cannot

I just want to point out that identifiers can be quoted.  YANG
keywords cannot be quoted, by any argument to a keyword can be
quoted.  YANG has a very simple syntax:

  statement = keyword [argument] (";" / "{" *statement "}")

So this is legal:

  leaf 'sys' + 'Name' { ... }

/martin

---

From: Phil Shafer <[email-masked]>
Date: Sun, 29 Mar 2009 15:28:24 -0400

>Is anybody?  If not, then this tar-pit is played out, right?

You raised this as an interoperability issue at the WG meeting.  I
agree with you.  The current text should be changed, since it allows
one to write YANG modules that compiler with one valid tool set,
but can fail (years later) with another perfectly valid tool set.
This is a serious (though minor) issue and should be resolved.

The question is: do we need to limit identifier length?  If you
have a convincing argument, please share it.  My opinion is that
there's no purpose served by a limit, and no having one matches
what XML does.  While there are many components of XML that YANG
avoids, we have a convincing reason for avoiding them.  Lacking a
justification, a limit is just a CLR.

Thanks,
 Phil

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sun, 29 Mar 2009 22:04:51 +0200

On Sun, Mar 29, 2009 at 09:28:24PM +0200, Phil Shafer wrote:

> The current text should be changed, since it allows one to write
> YANG modules that compiler with one valid tool set, but can fail
> (years later) with another perfectly valid tool set.

Here is the text:

   Implementations MUST support identifiers up to 63 characters in
   length. 

I do not understand how you reach the conclusion quoted above from
this sentence.

/js

---

From: Andy Bierman <[email-masked]>
Date: Sun, 29 Mar 2009 15:18:23 -0700

> I do not understand how you reach the conclusion quoted above from

I agree with you.

But there is no sentence that says:

   Implementations MAY support identifiers longer than 63 characters
   in length.

Perhaps that is what Phil means.

I am not sure what adding that sentence would mean to the standard.
I am fine with the current text, as-is.

I agree with the comment that Wes made at the meeting,
which is that tool makers need to know how what size an identifier can
be, in order to create a conforming implementation.
For many practical reasons, the number needs to
be a lot closer to 63 than infinity.

I cannot think of a sentence any more clear than the one
Juergen wrote to accomplish that.

It means that if you use an identifier in a YANG module
that is 63 characters or less, and your tool does not
accept it, your tool is broken.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Sun, 29 Mar 2009 15:53:39 -0700

> justification, a limit is just a CLR.

It is not just a CLR.

Saying that implementations MUST support infinite length
identifiers is a non-starter.

Saying that implementations can pick whatever identifier
length they want and YANG module writers will have no
idea what to expect across tool-vendors is a non-starter.

There are no OIDs in NETCONF.
The descriptors are stored in the agent,
and show up in lots of PDUs.
Code is generated using these descriptors.
The memory impact from these descriptors is already
non-trivial at 63 chars.

For these 3 reasons alone, there needs to be a finite limit specified.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 30 Mar 2009 08:16:49 +0200

> Which do you prefer and why?

I don't have a strong opinion and tend to support 1), but I am also fine
with the existing formulation - I believe no sane module designer would
ever try to exceed the 63 char limit, and it can be also stressed in the
guidelines document.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Phil Shafer <[email-masked]>
Date: Mon, 30 Mar 2009 08:54:39 -0400

>identifiers is a non-starter.

This is the part I don't follow you on.  Many technologies are able
to use XML without limit the length of the elements they use.  XSLT,
for example, does not say that one can only write stylesheets for
XML documents where the element names do not exceed 63 characters.
XSD editors don't fail when editing schemas for languages that use
element names longer than 63 characters.  XML parsers are perfectly
happy when handling these long element names.  The developers of
all those tools would certainly question the mental abilities of
someone that makes a 2048 character long identifier, but they don't
stop working when faced with one.  Similarly, I can't see why one
would, but I can't see why we can't handle it.

In general, identifier length will be self regulating.  If you see
someone publish a YANG module with a 2048 character long identifier,
you will question their sanity and perhaps decide not to implement
such a module.  You may decide to work with the author to make a
module that's sane.  But you shouldn't need to go to the author and
say that you love their module but can't use it because your toolset
only supports 63 character identifiers.

If there's no strong reasoning behind the rule, it's a CLR.

>There are no OIDs in NETCONF.

Completely agree ;^)

>and show up in lots of PDUs.

True, but not speed over the wire is not the goal of XML-based
protocols.

>Code is generated using these descriptors.

These are code generators for XML schema, etc.

>The memory impact from these descriptors is already

63 characters is pretty trivial as far as memory footprint.

More importantly, only those who implement modules with obnoxiously
long identifiers pay the price.  If a device implements only sane
modules, there's no price to be paid.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 30 Mar 2009 07:07:37 -0700

>> identifiers is a non-starter.

I don't know how to implement infinite length identifiers. Do you?
How is it that memory resources of no concern in this case,
but they are super-critical when it comes to making sure
the list keys are first?  It's just memory.  Deal with it.

So every implementation is going to pick a limit,
but every one will be different.  Great standard.

> only supports 63 character identifiers.

Self-regulating does not work in standards.
We cannot have the standard say that an implementation
is conforming if it support 1 byte identifiers.
That's what you want by removing the rule entirely.

> If there's no strong reasoning behind the rule, it's a CLR.

Huh?

The reason is to protect the YANG writers!
They need to know if it is safe to use a particular
identifier value.  The standard currently says
"You are safe if you use 63 chars or less."
That is far from a CLR.  It is critical to interoperability
and deployment.

> 63 characters is pretty trivial as far as memory footprint.

times 10,000 objects it is not trivial...

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 30 Mar 2009 10:00:41 -0400

>I do not understand how you reach the conclusion quoted above from

If toolsets are allowed to support differing identifier lengths,
some tools will happily eat long idenfitiers and others will not.
Modules written for one toolset will not compile on another, even
though both toolsets are following the spec.

Consider the following real world scenario: I have a module with a
64 character leaf name.  I publish it.  The entire world rallies
behind this standard, and life is good.  Then you need to run my
module on your device.  But when you compile the module, it fails
with a ">63" error.  The rest of the world is running the open
source YANG libraries that don't give an error or warning for my
leaf, but you've bought a toolset that won't compile it.  You send
me email, but I tell you (a) use the open source tools, and (b) the
module is published and working, so I'm not about to change it.  I
tap my finger tips on my helmet while whispering to my teammates
"Fetchez le vache!".

If I can't count on tools supporting long identifiers, then I
can't use long identifiers and the 63 minimum maximum becomes
the defacto maximum.  If that's the case, we should say this in
the draft, fixing the max at 63 characters.  I don't see a strong
reason for this, so I'd prefer to remove the limit completely.

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 30 Mar 2009 16:14:26 +0200

> but every one will be different.  Great standard.

Not necessarily, the memory for the identifier strings can be allocated
dynamically without setting any arbitrary limit on their length.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Phil Shafer <[email-masked]>
Date: Mon, 30 Mar 2009 10:19:52 -0400

>I don't know how to implement infinite length identifiers. Do you?

Read in the identifier, malloc sufficient space for it, add it to
your dictionary.

>the list keys are first?  It's just memory.  Deal with it.

Assuming you are comparing this to buffering an entire netconf
database, the difference is that the length of an element name is
likely to be small but the size of a database is known to be large.
I have customers with 100MB configuration files, and don't want to
buffer this amount of data.  If someone goes to me with a module
that has a 100MB identifier, I can refuse to consider their module.

>That's what you want by removing the rule entirely.

Nonsense.  My comment was we should have unlimited identifiers and
that tools should support this.  People who write modules with long
identifiers are self regulating, in that people will ignore them
and/or call them names.  Imagine finding a C library where every
variable has an insanely long name.  You'll assume the author is
an idiot and move on.  This is regardless of the ability of your
compiler to compile that function, but from a desire not to have
code that looks like:

        if (my_really_long_function_name_that_is_silly(
            my_really_long_enum_name_that_is_silly,
            my_other_really_long_enum_name_that_is_silly)
            ==
            my_really_long_error_code_name_that_is_silly) { ... }

No one will adopt such code, so the use of long identifiers is
self regulating.

>That is far from a CLR.  It is critical to interoperability

If the length has no limit, writers are also protected.

>times 10,000 objects it is not trivial...

Why would you have 10,000 copies of a string?  Put it in a dictionary
and refer to that one entry.  Take a look at libxml2 for an example.
They do this for namespaces as well, IIRC, so that namespace
comparisons aren't strcmp() calls, but simple pointer equality
tests.  String ops are expensive; avoid them where possible.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 30 Mar 2009 07:30:26 -0700

> dynamically without setting any arbitrary limit on their length.

Well, I need to use malloc(), which will fail if
the requested amount is big enough.  I don't
have a magic malloc() function like you.

But the standard allows this just fine.
You can support any identifier length you want, >= 63.

Since 64 length identifiers have been around for awhile in SMIv2,
the YANG limit of 63 is actually lowering the implementation
burden by 1 byte.

Since nobody has a use case for identifiers over 64 chars
right now,  the only logical motivation one could have for
removing this rule is if they wanted to support
a limit lower than 63 in their implementation.  Not good.

I vote for leaving the text as-is, and not removing the 63 limit from YANG.
Are you in favor of keeping or removing it?

Andy

---

From: Andy Bierman <[email-masked]>
Date: Mon, 30 Mar 2009 07:32:24 -0700

....
> If the length has no limit, writers are also protected.

This is the critical reason the rule needs to stay in the draft.
Please explain how writers are better protected by removing it.

Andy

---

From: Phil Shafer <[email-masked]>
Date: Mon, 30 Mar 2009 10:50:54 -0400

>Please explain how writers are better protected by removing it.

If identifiers are not limited to a small number by a CLR, tools
have to support any length.  Your tools that support any length
will support the same length as my tools that support any length.
Writers need not worry that a module which compiles fine with one
tool set will fail with another tool set.

Thanks,
 Phil

---

From: Phil Shafer <[email-masked]>
Date: Mon, 30 Mar 2009 11:00:57 -0400

>have a magic malloc() function like you.

If is we limit the identifier size to the limit at which malloc
will fail, would that be suitable?  This limit is on the order of
megas, right?  What about 1m?  Or 4m?  Too large still?  64K?  16K?
1K?  Is there something magic about 63?  Does your non-magic malloc
only return that amount?  ;^)

We also carry namespaces in our content; should we limit them to
63 characters?

>a limit lower than 63 in their implementation.  Not good.

I don't think anyone has suggesting lowing the limit.

We've been getting rid of CLRs by saying "why not allow this?", and
this is one more spot where there's no real reason to keep an ancient
SNMP restriction, dating from the dark ages when 63 bytes was huge.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 30 Mar 2009 08:14:55 -0700

> tool set will fail with another tool set.

I strongly disagree with this argument.
I do not believe any implementations of NETCONF
will ever handle arbitrarily long identifiers.
I will write a test case to see if yous will
handle a '4 billion trillion byte string' :-)

Nobody has provided a single use-case demonstrating
why setting the bar set at 63 is too low.

There is a huge difference between a mere encoding
standard like XML which does not specify anything
to do with the content, and one like NETCONF/YANG,
which relies on common content handling capabilities
for its value.

YANG writers need to know where the bar is set.
Infinite length identifiers in YANG are not possible,
so every NETCONF/YANG implementation will pick
some limit.  That is fine, as long as the number
is >= 63.  I see no reason to change that semantics in the draft.

Andy

---

From: Balazs Lengyel <[email-masked]>
Date: Mon, 30 Mar 2009 16:54:19 +0200

Hello,
IMHO the current text is OK. It tells you must support 63 (it is mandatory to support that).
I think that is enough to guarantee interoperability.
Mandating or assuming you can use unlimited long identifiers is risky. It will decrease the 
chances of interoperability, as in practice, some implementations may not support it.

And yes, a warning from YANG tools for longer identifiers would be nice.
Balazs

---

From: Phil Shafer <[email-masked]>
Date: Mon, 30 Mar 2009 11:48:06 -0400

>handle a '4 billion trillion byte string' :-)

All implementations are limited by available memory,  but we don't
make max-elements mandatory for all lists.  Will you write a test
case for a billion trillion list members?

C compilers are limited by memory, but there's no limit on the size
of C files.  Editors are limited by memory but there's not fixes
limit on editor files.  My inbox is limited by memory, but there's
no limit (or rate limit) on incoming email.

>why setting the bar set at 63 is too low.

Try the other side: Why limit this?

I know you have a scheme in your implementation for encoding your
stuff in a database that tops out at something like 900 bytes, and
I know that's why this issue is dear to you, but I don't see that
as a convincing driver for a spec.  900/63 == 14, so does it follow
that we limit depth of YANG modules to 14 levels?

In any case, you and I have debated this issue enough.  I'm going
to step back and see what the rest of the WG says.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Mon, 30 Mar 2009 09:06:12 -0700

> case for a billion trillion list members?

This is different.
We are defining a data modeling language.
In NETCONF, the agent simply sends 'resource-denied'
when it runs out of memory.

If anything, the correct analogy here is min-elements, not max.

> that we limit depth of YANG modules to 14 levels?

I am quite happy with 63 as the minimum everybody
must support.  This limit of 14 levels is only
for the pre-loaded SQL database supporting
the online netconf central WEB pages.  It has
nothing to do with the standard.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 01 Apr 2009 08:01:28 +0200 (CEST)

> Which do you prefer and why?

I think Phil had a good explanation why (3) leads to interoperability
issues.  The current text implies that if you want to write an
interoperable module, you better use less than 63 characters.  The
question is why we want the standard to allow for
non-interoperability?

Regarding a limit as in (2), IMO this is pretty arbitrary.  Why do we
have a limit on this particular thing, but not on namespace uri
length; not on the 'contact', 'description' clause data; not on the
number of leafs in a container; not on the number of features; not on
the nested levels in the hierarchy; not on anything else!  What makes
the identifier special?

The one explanation we have seen is from Juergen, and that is that we
want to support code generators that use the YANG identifiers as
identifiers in code.  While I agree with this idea, I think that such
a code generator (like ours) will have to handle some special cases
anyway, e.g. an identifier in C cannot have a '-' character in it, and
identifiers longer than 63 characters will sure be a corner case.

So, my order of preference would be (1), (2), (3).

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 31 Mar 2009 23:18:47 -0700

> So, my order of preference would be (1), (2), (3).

I strongly object to (1), and prefer to leave the text as-is.

  1) YANG writers MUST know the minimum safe length value that
     can be used on all YANG tools
  2) no YANG/NETCONF implementation can possibly support infinite
     length identifiers, and letting implementors pick any arbitrary limit
     is not inter-operable
  3) No use-case has ever been presented for why the length of 63
     (or 64 established by SNMP 20 years ago) is causing any problems
     whatsoever that need to be solved.  The limit of 63/64 is not
     arbitrary. It is time-proven to be necessary and sufficient
     for efficient tools interoperability
  4) The current rule does not prevent anyone from supporting infinite
     length identifiers if they want.  The IETF does not need them.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Wed, 01 Apr 2009 08:29:33 +0200 (CEST)

>      length identifiers, and letting implementors pick any arbitrary limit

Why don't you think the same rule is important for the other
constructs I mentioned?  What makes the identifier special?

/martin

---

From: Andy Bierman <[email-masked]>
Date: Tue, 31 Mar 2009 23:44:04 -0700

>> the nested levels in the hierarchy; not on anything else!  What makes

The difference is between data stored in the agent and data that is not.

Good catch on the namespace-URI.
We need a similar rule for that (for the agent)

   Implementations MUST support namespace URIs at least 1023 characters
   in length.

The other clauses are not stored in the agent so they do not matter.

It is simply not inter-operable to let YANG compilers accept YANG
identifiers based on whether malloc() worked or not for that particular
instance.  YANG writers need to know that any agent that cannot
handle 63 character names is broken.

>> identifiers longer than 63 characters will sure be a corner case.

This is not the point.
Your mangled names are irrelevant.
What matters is that all tools accept 63 char names,
and any IETF YANG module can have identifiers up to that limit.

Andy