Subject: Re: [core] progressing ietf-core-yang-cbor and ietf-core-sid

From: Ivaylo Petrov <[email-masked]>
Date: Wed, 8 Jan 2020 11:23:21 +0100

Hello,

I am working on a new version of the SID draft that would address the
comments related to 63bit SIDs and looking to clarify how are .sid files
passed to IANA. Those are the only comments that I have received and I
believe they are not blocking the WGLC. What I heard during the last IETF
was that this draft is WGLC material, https://youtu.be/YIHoiHhz2Ys?t=1280
and I was expecting the CoRE chairs to go ahead with this. If you believe
there is some misunderstanding on my side and there are other issues that
need to be resolved before the WGLC, please let me know.

Best regards,
Ivaylo

CONFIDENTIALITY NOTICE: This email may contain confidential and privileged
material for the sole use of the intended recipient(s).

It is strictly forbidden to share any part of this message with any third
party, without the written consent of the sender.

If you have received this message by mistake, please notify the sender
immediately by email and delete the message and any file attached to it.
Thank you!

On Mon, Dec 30, 2019 at 11:08 AM Peter van der Stok <[email-masked]>
wrote:

>
> I have not seen any progress on:
>   1) https://datatracker.ietf.org/doc/draft-ietf-core-yang-cbor/
>   2) https://datatracker.ietf.org/doc/draft-ietf-core-sid/
>
> At IETF105, it seemed that the documents were unstuck, and would progress
> to WGLC soon.  Is there something holding this up other than author time?
>
>
>
> I also want to express my concern about the VERY slow progress of these
> documents.
> It is quite possible and understandable that the authors are occupied with
> too many other tasks.
>
> An additional author may be the answer to unstick the progress.
>
> Greetings,
>
> Peter
>
>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 8 Jan 2020 12:00:56 +0100

Hi Ivaylo,

The minutes indeed say:

* draft-ietf-core-yang-cbor-11, draft-ietf-core-yang-library-00,
  draft-ietf-core-comi-08, draft-ietf-core-sid-07: Now ready for WGLC,
  to be sent to CoRE as well as netconf/netmod (with the discussion to
  happen on CoRE).  (One comment on the numeric range of SIDs still to
  be taken care of.)

The parenthesis may have been a bit cryptic, but the intention was indeed to have I-D versions with all known issues (including the 63-bit one) addressed.  Is there a reason this will need more time?

Grüße, Carsten

> On 2020-01-08, at 11:23, Ivaylo Petrov <[email-masked]> wrote:
> 
> Hello,
> 
> I am working on a new version of the SID draft that would address the comments related to 63bit SIDs and looking to clarify how are .sid files passed to IANA. Those are the only comments that I have received and I believe they are not blocking the WGLC. What I heard during the last IETF was that this draft is WGLC material, https://youtu.be/YIHoiHhz2Ys?t=1280 and I was expecting the CoRE chairs to go ahead with this. If you believe there is some misunderstanding on my side and there are other issues that need to be resolved before the WGLC, please let me know.
> 
> Best regards,
> Ivaylo
> 
> CONFIDENTIALITY NOTICE: This email may contain confidential and privileged material for the sole use of the intended recipient(s).
> It is strictly forbidden to share any part of this message with any third party, without the written consent of the sender.
> If you have received this message by mistake, please notify the sender immediately by email and delete the message and any file attached to it. Thank you!
> 
> 
> On Mon, Dec 30, 2019 at 11:08 AM Peter van der Stok <[email-masked]> wrote:
>> 
>> I also want to express my concern about the VERY slow progress of these documents.
>> It is quite possible and understandable that the authors are occupied with too many other tasks.
>> 
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: Ivaylo Petrov <[email-masked]>
Date: Wed, 8 Jan 2020 23:24:30 +0100

Hello Carsten,

I was not aware that this is the blocking point and that is why I had put
much less priority to this modification. I should have read more carefully
the minutes, my bad. I have uploaded a new version now that addresses
this issue.

Best regards,
Ivaylo

On Wed, Jan 8, 2020 at 12:00 PM Carsten Bormann <[email-masked]> wrote:

> The parenthesis may have been a bit cryptic, but the intention was indeed
> to have I-D versions with all known issues (including the 63-bit one)
> addressed.  Is there a reason this will need more time?
>
> Grüße, Carsten
>
>
> > CONFIDENTIALITY NOTICE: This email may contain confidential and
> privileged material for the sole use of the intended recipient(s).
> > It is strictly forbidden to share any part of this message with any
> third party, without the written consent of the sender.

---

From: Ivaylo Petrov <[email-masked]>
Date: Thu, 9 Jan 2020 00:00:02 +0100

Hello Michael,

See my comment below.

Best regards,
Ivaylo

On Wed, Dec 25, 2019 at 12:50 AM Michael Richardson <[email-masked]>
wrote:

>
> {trying to clear out my inbox}
>
> Michel Veillette <[email-masked]> laments:
>     > The file "[email-masked]" is constructed
> with
>     > an "import ietf-voucher" and a "uses v:voucher-artifact-grouping".
> For
>     > this reason, part of the "yang-data voucher-constrained-artifact" is
>     > defined in "ietf-voucher" with SIDs assigned within the scope of this
>     > module.
>
>     > This example raises an interesting point. "ietf-voucher" have been
>     > defined without considering a possible constrained implementation and
>     > have no SID list or .sid file included in RFC 8366. SIDs has been
>     > assigned after the fact, as allowed by
>     > https://tools.ietf.org/html/draft-ietf-core-sid-07#section-3. If we
>     > include a SID list or .sid file for
>     > "[email-masked]", part of the SIDs will
> be in
>     > a RFC, the rest won't. To avoid such situation, we should rely
> entirely
>     > on an IANA registry for all .sid files defined by RFCs.
>
> I am still uncertain how this is going to work.
>

IP: Let me try to summarize how the sid range obtaining and allowaction
process is currently defined and please let me know if it is still not
clear enough or if you have any concerns. After that I share some thoughts
for module updates.

* In order for a draft author to obtain a SID range from IANA prior to
publication, the person needs to convince the chairs or AD that this early
allocation is needed as described in RFC7120. After that this request can
be forwarded to the designated experts that will approve it and it will be
reflected on the IANA registry.
If the author does not want to go through the trouble of early allocation
there are two cases:
  - the RFC is already published - then as it is specified in sec 6.3.2 ,
the experts will check that there is an RFC and the registration will be
done upon request.
  - the RFC is being processed by IANA right before publication - then the
same procedure is triggered by IANA as if the RFC is already published and
another party requests the range.
* Once the author has the range, he can generate the .sid file. Once there
is a published RFC or IANA is processing a document that is ready for
publication, the experts will be contacted and after that the .sid file
will be added to "IETF SID Registry”. I can see how it could be difficult
to generate the .sid file if you don't have the range, but for me this can
be achieved with appropriate instructions to the RFC editor.

I am not sure to correctly understand the citation in your email, so please
forgive me if I have completely missing the point, but for me it should
either be covered by what I have written so far or it should be related to
updating a YANG module. For me it is simpler if a module is updated to have
its .sid file updated as well (provided it has one).

If this is done, the situation where a module M1 is updated and it is used
by another module M2 would not cause issues as it is should match the case
where the SIDs are generated for the first time. On the other hand, if
someone else generates a .sid file that includes the extra values that come
from the new version of module M1, this is not going to cause real issues
as there could be two SID values that represent the same YANG item, which
means that the .sid file for M1 can still be updated.

> IP: I believe that this is resolved now.

> I think that allocations were made for ietf-anima-constrained-voucher, but
> I
> guess that was in the github version only.
>
> --
> Michael Richardson <[email-masked]>, Sandelman Software Works
>  -= IPv6 IoT consulting =-
>
>
>
>

---

From: Michael Richardson <[email-masked]>
Date: Thu, 09 Jan 2020 22:14:41 -0500

{AD+Chair action requested below}

Ivaylo Petrov <[email-masked]> wrote:
    >> "[email-masked]", part of the SIDs  will be in
    >> a RFC, the rest won't. To avoid such situation, we should rely entirely

    mcr> I am still uncertain how this is going to work.

I think that the major issue is that I am still not convinced that IANA is
prepared to operate pyang itself.  Conversations at IETF105 and IETF106 did
not convince me that they (IANA) completely understood what they are being
asked to do.
For instance, I'm not convinced that there is a support process for IANA to
get bugs fixed in the sid.py module.

(I would be pleased to be convinced otherwise)

    ivaylo> IP: Let me try to summarize how the sid range obtaining and
    ivaylo> allowaction process is currently defined and please let me know
    ivaylo> if it is still not clear enough or if you have any
    ivaylo> concerns. After that I share some thoughts for module updates.

Yes, I understand the early allocation process.

    ivaylo> - the RFC is already published - then as it is specified in sec 6.3.2 ,
    ivaylo> the experts will check that there is an RFC and the registration will
    ivaylo> be done upon request.

Early Allocations are done for a limited period of time, IANA is allowed to
reclaim the space if the document does not proceed.

If when an author asks for early allocation for a YANG module that references
other modules which are already published, then there are multiple things that could
happen:

1) IANA could do a permanent allocation for the referenced modules, entering
   the sid file in as well.
   {this may cause a recursion quite deep initially}

2) IANA could do an early allocation for the referenced modules, marking them
   as such, and potentially reclaiming.

(1) has the problem that it may cause a bunch of pollution.
(2) has the problem that the referenced allocation will need to be reference
    counted for all/any documents that reference it.

I suspect that the problem will be short-lived pain, and that doing (1) is
the right thing, and that we might want to take the hit in WG and Expert
Review in a proactive manner.  (I am happy to help here)

But, there is also:

(3) a document could ask for an early SID allocation, referencing a YANG
    module which is in another WG, or worse, not yet adopted, and therefore
    not subject to Early Adoption rules!
    This may be a problem we do not to fix (Just get the document adopted),
    but I think that we should be clear about this.

My recommendation is that all YANG modules, upon being *ADOPTED* by a WG,
have a SID allocation done.  Yes, the document will get revised, but the
SID system is designed to deal with this.

I think that doing this requires an IESG and IANA decision.
I think that getting through this before March is important.

The CORE WG has had a series of virtual interims (I see none scheduled, I
assume that is an oversight), and maybe we could invite IANA and a couple of
IESG ADs to confer and get clarity here.

    > If this is done, the situation where a module M1 is updated and it is
    > used by another module M2 would not cause issues as it is should match
    > the case where the SIDs are generated for the first time. On the other
    > hand, if someone else generates a .sid file that includes the extra
    > values that come from the new version of module M1, this is not going
    > to cause real issues as there could be two SID values that represent
    > the same YANG item, which means that the .sid file for M1 can still be
    > updated.

Agreed!

---

From: Ivaylo Petrov <[email-masked]>
Date: Fri, 10 Jan 2020 16:37:09 +0100

Hi Michael,

Thank you for the detailed explanation! I understand your concerns
now. Please find my comments below.

> (I would be pleased to be convinced otherwise)

I remember that during the discussion with IANA we considered that
the authors and the expert will be responsible for the running of the
pyang tool and IANA will only take the .sid file and store it. The .sid
files will be provided to IANA when there is a request to register the
.sid file - however that is done. Do you see an error in our reasoning
other that the fact that we might be burdening the authors too much?
Does this seem acceptable to you?

> SID system is designed to deal with this.

Yes, that issue does not seem obvious. Currently I don't see a
blocking issue with allocating SIDs for all YANG modules once
a draft is adopted,  so maybe this is truly our best way to avoid
problems.

--
Best regards,
Ivaylo

---

From: Michael Richardson <[email-masked]>
Date: Sat, 11 Jan 2020 19:30:13 -0500

Ivaylo Petrov <[email-masked]> wrote:
    >> I think that the major issue is that I am still not convinced that
    >> IANA is prepared to operate pyang itself.  Conversations at IETF105
    >> and IETF106 did not convince me that they (IANA) completely understood
    >> what they are being asked to do.  For instance, I'm not convinced that
    >> there is a support process for IANA to get bugs fixed in the sid.py
    >> module.
    >>

    > I remember that during the discussion with IANA we considered that the
    > authors and the expert will be responsible for the running of the pyang
    > tool and IANA will only take the .sid file and store it. The .sid files
    > will be provided to IANA when there is a request to register the .sid
    > file - however that is done. Do you see an error in our reasoning other
    > that the fact that we might be burdening the authors too much?  Does
    > this seem acceptable to you?

okay, so the expert reviewer needs to be able to run pyang --generate-sid-file, etc.
I don't know if IANA has a process to accept files online, but I guess
attachments to tickets will work.

    >> (3) a document could ask for an early SID allocation, referencing a
    >> YANG module which is in another WG, or worse, not yet adopted, and
    >> therefore not subject to Early Adoption rules!  This may be a problem
    >> we do not to fix (Just get the document adopted), but I think that we
    >> should be clear about this.
    >>
    >>
    >> My recommendation is that all YANG modules, upon being *ADOPTED* by a
    >> WG, have a SID allocation done.  Yes, the document will get revised,
    >> but the SID system is designed to deal with this.

    > Yes, that issue does not seem obvious. Currently I don't see a blocking
    > issue with allocating SIDs for all YANG modules once a draft is
    > adopted, so maybe this is truly our best way to avoid problems.

okay, if we are in agreement, then I think that we need text in the SID
document's IANA Considerations that says this.

---

From: Ivaylo Petrov <[email-masked]>
Date: Mon, 13 Jan 2020 11:35:30 +0100

OK, a question to the chairs below.

> document's IANA Considerations that says this.

Do you see any issues adding such a text and do you want me to add it
before you
can start a WGLC or how do you want this to happen?

--
Best regards,
Ivaylo

---

From: Michael Richardson <[email-masked]>
Date: Mon, 13 Jan 2020 13:11:51 -0500

I think you should just ahead and add the text.
The document can change during WGLC.

Some edits I suggest:

}   The reason not to use 64-bit
}   using SIDs:       unsigned integers is that otherwise protocols doing
}   arithmetical
}   operations with the SIDs could be very difficult to implement.

Change to:
  The use of 63-bit unsigned integers allows SIDs to be manipulated more easily
  on architectures that might otherwise lack native 64-bit unsigned arithmetic.
  The loss a single bit of precision is not significant given the size of the
  space.

Maybe you pointed me at github.com of the XML before, but I lost the link, so
I'll do this the old fashioned way. For the IANA section, I suggest the following:

6.4.1.  Structure
(no change)

6.4.2.  Allocation policy
(one pointed added at end)

   The allocation policy is Expert review.  The Expert MUST ensure that
   the following conditions are met:

   o  The ".sid" file has a valid structure:
      *  The ".sid" file MUST be a valid JSON file following the
         structure of the module defined in RFCXXXX 

   o  The ".sid" file allocates individual SIDs ONLY in the SID Ranges
         for this YANG module (as allocated in the IETF SID Range
         Registry):

      *  All SIDs in this ".sid" file MUST be within the ranges
               allocated to this YANG module in the "IETF SID Range
               Registry".

   o  If another ".sid" file has already allocated SIDs for this YANG
         module (e.g.  for older or newer versions of the YANG module), the
         YANG items are assigned the same SIDs as in the the other
         ".sid" file.

   o  SIDs never change.

   o  IANA is expected to store the SID files and make them available
      along with the associated YANG files in the YANG Module Names registry.

ADD:
6.4.3.  Recursive Allocation of SID Range at Document Adoption

   Due to the difficulty in changing SID values during IETF document
   processing, it is expected that most documents will ask for SID
   allocations using Early Allocations [BCP100].   The details of the Early
   Allocation should be included in any Working Group Adoption call.

   During the early use of SIDs, many existing, previously published YANG
   modules will not have SID allocations.  For an allocation to be
   useful the included YANG modules may also need to have SID allocations
   made.

   The Expert Reviewer who performs the (Early) Allocation analysis will need
   to go through the list of included YANG modules and perform SID
   allocations for those modules as well.

   If the document is a published
   RFC, then the allocation is permanent. 
   The Expert Reviewer provides the generated SID file to IANA for
   registration.    
   This process may be someone busy during a bootstrap period (there are over
   100 YANG modules to date, none of which have SID allocations), but should
   quiet down once needed entries are allocated. 

   If the document is an unprocessed Internet-Draft adopted in a WG, then
   an Early Allocation is performed for this document as well.   
   Early Allocations require approval by an IESG Area Director. 
   An early allocation which requires additional allocations will list the
   other allocations in it's description, and will be cross-posted to the
   any other working group mailing lists.

   A YANG module which references a module in an document which has not yet
   been adopted by any working group will be unable to perform an Early
   Allocation for that other document until it is adopted by a working group.
   As described in [BCP100], an AD Sponsored document acts as if it had a
   working group.  The approving AD may also exempt a document from this
   policy by agreeing to AD Sponsor the document.
   
   Critically, the original document should never get through the IETF
   process and then be surprised to be referencing a document whose progress
   is not certain.
  
   A previously SID-allocated YANG module which changes it's references to
   include a YANG module for which there is no SID allocation needs to repeat
   the Early Allocation process.

   Early Allocations are made with a one-year period, after which they are
   expired.  [BCP100] indicates that at most one renewal may be made.
   For the SID allocation a far more lenient stance is desired.
   1) An extension of a referencing documents Early Allocation should update any
      referenced Early Allocations to expire no sooner than the referencing
      document.
   2) The [BCP100] mechanism allows the IESG to provide a second renewal,
      and such an event may prompt some thought about how the collection
      of documents are being processed.
   
   This is driven by the very generous size of the SID space and the often complex
   and deep depencies of YANG modules.  Often a core module with many
   dependancies will undergo extensive review, delaying the publication
   of other documents.

   [BCP100] also says
      Note that if a document is submitted for review to the IESG and at
      the time of submission some early allocations are valid (not
      expired), these allocations should not be expired while the document
      is under IESG consideration or waiting in the RFC Editor's queue
      after approval by the IESG.

---

From: Michael Richardson <[email-masked]>
Date: Mon, 13 Jan 2020 13:15:31 -0500

Ivaylo,
In section 6.3.3, you have no allocation for constrained-voucher.
You had given us 2500:2550 --- constrained-voucher-request.
     And 2400 -- 2440:2500 --- constrained-voucher.

I thought that this was going to be in the initial allocation.
Maybe this causes some problem, and we need to do this some other way?

We can't do an Early Allocation until this is published.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Ivaylo Petrov <[email-masked]>
Date: Wed, 15 Jan 2020 13:56:10 +0100

Hello Michael,

Thank you very much for the proposed text! I have applied the changes that
you proposed with slight modifications where I felt the need. You can find
the proposed version in attachment and the changes here [1]. We are using
markdown to build the draft, therefore there is no XML version that you can
directly edit, but here you would find the one we are using in case you
want to propose further modifications [2]. If the proposed changes are good
for you, I will publish -09 right away.

Please notice that the change you proposed for sec 6.4.2. seemed out of
place there and I modified slightly sec 6.4.1 to address what I believe was
your concern. Please let me know if further changes are required.

Best regards,
Ivaylo

[1]:
https://github.com/core-wg/yang-cbor/commit/05611792ff852438072bf13e2309c34ba5b8b159
[2]: https://github.com/core-wg/yang-cbor/tree/sid-09

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 15 Jan 2020 14:34:44 +0100

(No chair hat.)

>  The loss a single bit of precision is not significant given the size of the

s/use of/limitation to/     (we have been using them before)
s/architectures/platforms/  (Java is our main problem here, no?)
s/precision/range/
s/space/remaining space/

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 15 Jan 2020 14:32:38 +0100

Hi Ivaylo,

Sorry for being slow to reply.

> On 2020-01-13, at 11:35, Ivaylo Petrov <[email-masked]> wrote:
> 

If the text doesn’t come out of the blue but reflects discussion here on the WG list, that is exactly what we do when making documents ready for WGLC.  So please do go ahead.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 15 Jan 2020 14:37:04 +0100

>   o  SIDs never change.

That may require a little bit of expansion.  SIDs do change while I’m editing the document on my laptop and not telling anyone else what I’m doing.  But at some time, they freeze.

What is the specific event that makes a SID allocation immutable, even if the underlying identifier goes away (and maybe comes back later)?

What is the way we keep this memory?

Grüße, Carsten

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 15 Jan 2020 15:01:00 +0000

> -----Original Message-----
> From: core <[email-masked]> On Behalf Of Carsten Bormann
> Sent: 15 January 2020 13:35
> To: Michael Richardson <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [core] progressing ietf-core-yang-cbor and ietf-core-sid
> 
> (No chair hat.)
> 
> >  The use of 63-bit unsigned integers allows SIDs to be manipulated
> > more easily  on architectures that might otherwise lack native 64-bit
> unsigned arithmetic.
> >  The loss a single bit of precision is not significant given the size
> > of the  space.
> 
[RW] 
I think that it would be awkward for most languages/CPU architectures, anywhere where you could end up with a negative number that is less than 2^63.

Yes, it is possible to write code in C or Java to handle this, but my hunch is that implementations could easily get this wrong (e.g. by just using signed int64 as the type to hold the difference between parent and child SIDs.)

Thanks,
Rob

> s/precision/range/ s/space/remaining space/
> 
> Grüße, Carsten
>

---

From: Michael Richardson <[email-masked]>
Date: Wed, 15 Jan 2020 10:44:59 -0500

Ivaylo Petrov <[email-masked]> wrote:
    > Thank you very much for the proposed text! I have applied the changes
    > that you proposed with slight modifications where I felt the need. You
    > can find the proposed version in attachment and the changes here [1].
    > We are using markdown to build the draft, therefore there is no XML

Markdown even better than XML :-)

    > place there and I modified slightly sec 6.4.1 to address what I believe
    > was your concern. Please let me know if further changes are required.

okay.

But, what about:
    >     Ivaylo, In section 6.3.3, you have no allocation for
    > constrained-voucher.  You had given us 2500:2550 ---
    > constrained-voucher-request.  And 2400 -- 2440:2500 ---
    > constrained-voucher.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 15 Jan 2020 16:46:28 +0100

Yeah, but here the problem is in the head of the implementor and not in the CPU architecture :-)
I was not suggesting the text behind the substitutes to be included, this was just explanation why I’d like to have the change.

Grüße, Carsten

> On 2020-01-15, at 16:01, Rob Wilton (rwilton) <[email-masked]> wrote:
> 
> 
>

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 15 Jan 2020 16:40:46 +0000

> From: Carsten Bormann <[email-masked]>

> Sent: 15 January 2020 15:46

> To: Rob Wilton (rwilton) <[email-masked]>

> Cc: Michael Richardson <[email-masked]>; [email-masked]

> Yeah, but here the problem is in the head of the implementor and not in

> the CPU architecture :-)

I'm not sure that I agree, 😉

I might be wrong, but most CPU architectures don't support 64 bit negative values (unless you move to 128 bit signed numbers with the corresponding performance overhead).  Of course, implementors can work around the architecture limitations by emulating 64 bit negative numbers using unsigned 64 bit numbers ...

In fact, I was wondering whether it is really a good thing that CBOR allows a 64 bit negative integer to be expressed.  It means that the CBOR encoding can easily carry values that most implementations cannot easily generate or represent.

So, I looked at the first available C implementation example at https://cbor.io/impls.html, and it can't handle negative 64 bit numbers, overflowing them into a long variable.  I wonder what proportion of CBOR implementations do handle these correctly ...  Of course, I'm not suggesting changing the spec, just a retrospective observation.

> I was not suggesting the text behind the

> substitutes to be included, this was just explanation why I’d like to have

> the change.

[RW]

Yes, but I think that "architectures" is probably the correct term here, at least until we end up with CPUs with a 65 bit signed datapath, although I have to confess that I doubt that the choice of word really matters. 😊

Thanks,
Rob

> > On 2020-01-15, at 16:01, Rob Wilton (rwilton) <[email-masked]<mailto:[email-masked]>> wrote:

> >> From: core <[email-masked]<mailto:[email-masked]>> On Behalf Of Carsten Bormann

> >> To: Michael Richardson <[email-masked]<mailto:[email-masked]>>

> >> Cc: [email-masked]<mailto:[email-masked]>

> >> On 2020-01-13, at 19:11, Michael Richardson <[email-masked]<mailto:[email-masked]>>

> >>> more easily  on architectures that might otherwise lack native

> >>> 64-bit

> > I think that it would be awkward for most languages/CPU architectures,

> anywhere where you could end up with a negative number that is less than

> 2^63.

> > Yes, it is possible to write code in C or Java to handle this, but my

> > hunch is that implementations could easily get this wrong (e.g. by

> > just using signed int64 as the type to hold the difference between

> > parent and child SIDs.)

> >> [email-masked]<mailto:[email-masked]>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 15 Jan 2020 18:07:20 +0100

OK, let’s get pedantic then.

The operation that would be needed here without the 63-bit restriction is adding a 65-bit signed (delta) to a 64-bit unsigned (context SID), with the constraint that the result (actual SID) fits into a 64-bit unsigned.  Unless you want to do error checking (which probably is a good idea), this can be done entirely with 64-bit unsigned numbers, simply ignoring the modulo wraparounds.  Error checking then just requires a few magnitude checks, which do require some scribbling on the back of an envelope to get right.  All CPU architectures of the last 40 years (OK, maybe since the iAPX 432, which was 1981) that I know of make it not hard to do these things.  Platforms like the JVM do.

CBOR’s "65-bit signed" (64-bit unsigned + sign bit in mt) integer coverage is indeed surprising initially (it is until you start to embrace bignums).  From the C language, implementers should be used to the fact that as soon as you go to "signed 64-bit" (63-bit unsigned + sign bit expressed as two’s complement), there are all kinds of problems.  Of course, not all C courses teach this…  (The main problem being that 64-bit is big enough that many implementers never reach the boundaries of this space.)

So the problem here really is in the platform or BKAC (between keyboard and chair), not in the architecture...

Grüße, Carsten

> On 2020-01-15, at 17:40, Rob Wilton (rwilton) <[email-masked]> wrote:
> 
>  
>

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 15 Jan 2020 18:00:38 +0000

> Sent: 15 January 2020 17:07

[RW]

I am not being pedantic. 😉

CPU architectures really do not natively support negative 64 bit integers, and you can't pretend that they do, so let's just avoid them, because we don't need them and they will only cause bugs/mistakes.

> The operation that would be needed here without the 63-bit restriction is

> adding a 65-bit signed (delta) to a 64-bit unsigned (context SID), with

> the constraint that the result (actual SID) fits into a 64-bit unsigned.

> Unless you want to do error checking (which probably is a good idea), this

> can be done entirely with 64-bit unsigned numbers, simply ignoring the

> modulo wraparounds.  Error checking then just requires a few magnitude

> checks, which do require some scribbling on the back of an envelope to get

> right.

[RW]

I've not said that one can't encode this, just that it becomes more tricky and I predict that people will get it wrong, or likely assume that SIDs are only 63 bits.

E.g. when constructing the SID delta (in C) you can't just subtract the child SID from the parent SID and convert to a cbor signed/unsigned value, and expect the right thing to happen.  You have further annoyances if you want to store the SID deltas in heap data structures for any reason.  Perhaps nobody will even want/need to do this, but it seems plausible that they might.

> All CPU architectures of the last 40 years (OK, maybe since the

> iAPX 432, which was 1981) that I know of make it not hard to do these

> things.

[RW]

I respectively disagree.

Having seen the hard to debug bugs when smart experienced software engineers were not careful enough with C's handling of signed and unsigned integers definitely puts me into the camp that Java got this right, and that it is unsigned arithmetic that folks struggle to consistently get right.

> Platforms like the JVM do.

> CBOR’s "65-bit signed" (64-bit unsigned + sign bit in mt) integer coverage

> is indeed surprising initially (it is until you start to embrace bignums).

> From the C language, implementers should be used to the fact that as soon

> as you go to "signed 64-bit" (63-bit unsigned + sign bit expressed as

> two’s complement), there are all kinds of problems.  Of course, not all C

> courses teach this…  (The main problem being that 64-bit is big enough

> that many implementers never reach the boundaries of this space.)

[RW]

Okay, but I'm of the opinion that signed types are the sane ones, and it is unsigned types that do surprising things (normally with unanticipated underflow due to subtraction and modulo arithmetic).

> So the problem here really is in the platform or BKAC (between keyboard

> and chair), not in the architecture...

[RW]

As per above, I think that we disagree on this point.  But really not sure it is worth spending more time on ...

Kind regards,
Rob

> > On 2020-01-15, at 17:40, Rob Wilton (rwilton) <[email-masked]<mailto:[email-masked]>> wrote:

> > > From: Carsten Bormann <[email-masked]<mailto:[email-masked]>>

> > > To: Rob Wilton (rwilton) <[email-masked]<mailto:[email-masked]>>

> > > Cc: Michael Richardson <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>

> > > Subject: Re: [core] progressing ietf-core-yang-cbor and

> > > ietf-core-sid

> > > Yeah, but here the problem is in the head of the implementor and not

> > > in the CPU architecture :-)

> > I might be wrong, but most CPU architectures don't support 64 bit

> negative values (unless you move to 128 bit signed numbers with the

> corresponding performance overhead).  Of course, implementors can work

> around the architecture limitations by emulating 64 bit negative numbers

> using unsigned 64 bit numbers ...

> > In fact, I was wondering whether it is really a good thing that CBOR

> allows a 64 bit negative integer to be expressed.  It means that the CBOR

> encoding can easily carry values that most implementations cannot easily

> generate or represent.

> > So, I looked at the first available C implementation example at

> https://cbor.io/impls.html, and it can't handle negative 64 bit numbers,

> overflowing them into a long variable.  I wonder what proportion of CBOR

> implementations do handle these correctly ...  Of course, I'm not

> suggesting changing the spec, just a retrospective observation.

> > > I was not suggesting the text behind the substitutes to be included,

> > > this was just explanation why I’d like to have the change.

> > Yes, but I think that "architectures" is probably the correct term

> > here, at least until we end up with CPUs with a 65 bit signed

> > datapath, although I have to confess that I doubt that the choice of

> > word really matters. 😊

> > > > On 2020-01-15, at 16:01, Rob Wilton (rwilton) <[email-masked]<mailto:[email-masked]>>

> > > >> Subject: Re: [core] progressing ietf-core-yang-cbor and

> > > >> ietf-core-sid

> > > >> On 2020-01-13, at 19:11, Michael Richardson

> > > >> <[email-masked]<mailto:[email-masked]>>

> > > >>> The use of 63-bit unsigned integers allows SIDs to be

> > > >>> manipulated more easily  on architectures that might otherwise

> > > >>> lack native 64-bit

> > > >>> The loss a single bit of precision is not significant given the

> > > >>> size of the  space.

> > > > I think that it would be awkward for most languages/CPU

> > > > architectures,

> > > anywhere where you could end up with a negative number that is less

> > > than 2^63.

> > > > Yes, it is possible to write code in C or Java to handle this, but

> > > > my hunch is that implementations could easily get this wrong (e.g.

> > > > by just using signed int64 as the type to hold the difference

> > > > between parent and child SIDs.)

---

From: Michael Richardson <[email-masked]>
Date: Wed, 15 Jan 2020 13:28:04 -0500

> That may require a little bit of expansion.  SIDs do change while I’m
    > editing the document on my laptop and not telling anyone else what I’m
    > doing.  But at some time, they freeze.

Actually, if things go well on your laptop, they don't.
You might add/delete leafs and so you might roll through the SIDs quickly,
and then, because you haven't published anything, decided to reset, but
that's your business :-)

    > What is the specific event that makes a SID allocation immutable, even
    > if the underlying identifier goes away (and maybe comes back later)?

    > What is the way we keep this memory?

There is the .sid file which records the allocations.
You use --generate-sid-file (giving your allocation) to initialize the
process, which creates a .sid file.  You do this only once.

You use --update-sid-file normally to allocate SID values from a YANG file.
It keeps track not to re-use elements after the leaf is deleted, and makes
sure to re-use numbers when the leaf has not changed.  Deleting and then
adding a leaf with the *same name* should wind up with the SID being reused.

The SID file is some JSON.
Here is an example:
  https://github.com/anima-wg/constrained-voucher/blob/master/ietf-constrained-voucher-request%402019-09-01.sid

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 15 Jan 2020 22:32:05 +0100

> and then, because you haven't published anything, decided to reset, but

Exactly.

My question really was if we can state unambiguously when it stops being my business.

> adding a leaf with the *same name* should wind up with the SID being reused.

So when the development of a standard is completed, we hand IANA a SID file with dead SIDs in it?  Or how do we maintain the memory of those dead SIDs?  Are there boundaries through which a SID file can go that does remove dead SIDs?

>  https://github.com/anima-wg/constrained-voucher/blob/master/ietf-constrained-voucher-request%402019-09-01.sid

Yes.  The file format is fine, the question is whether we give enough guidance on how to handle the evolution of “the” SID file for a YANG module.

I think Andy Bierman has reminded us often enough that this is hard to get right.
Sorry for playing a bad copy of Andy today.

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Wed, 15 Jan 2020 16:56:36 -0500

Carsten Bormann <[email-masked]> wrote:
    >> You use --update-sid-file normally to allocate SID values from a YANG
    >> file.  It keeps track not to re-use elements after the leaf is
    >> deleted, and makes sure to re-use numbers when the leaf has not
    >> changed.  Deleting and then adding a leaf with the *same name* should
    >> wind up with the SID being reused.

    > So when the development of a standard is completed, we hand IANA a SID
    > file with dead SIDs in it?  Or how do we maintain the memory of those
    > dead SIDs?  Are there boundaries through which a SID file can go that
    > does remove dead SIDs?

Yes, if we evolved the design during WG processing, then we could well have
dead SIDs.  We could, at WGLC or IESG approval, edit the SID file and mark
them as useable again.
(If there were no running code, we could reset the SID file entirely at that point)
The SID file maintains the memory of the dead values for only as long as we
want it to.

    >> The SID file is some JSON.  Here is an example:

    > Yes.  The file format is fine, the question is whether we give enough
    > guidance on how to handle the evolution of “the” SID file for a YANG
    > module.

Thus, I think, a need to get early IANA review of the details.

    > I think Andy Bierman has reminded us often enough that this is hard to
    > get right.  Sorry for playing a bad copy of Andy today.

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 16 Jan 2020 00:04:38 +0100

> The SID file maintains the memory of the dead values for only as long as we

This paragraph is exactly reflecting my nightmares about this.
We could do a lot of things, if we wanted to.
What are our unambiguous, not-to-be-missed instructions that make this fool-proof?

I would expect that we get rid of dead SIDs about as often as we revoke TCP port number assignments, i.e., essentially never.  But this requires getting around the cognitive dissonance of just having “wasted” a SID value (*):
With very strongly worded instructions, I’d assume.

Grüße, Carsten

(*) (Note that there are way more SIDs than there are TCP ports.)

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 Jan 2020 15:14:12 -0800

On Wed, Jan 15, 2020 at 1:32 PM Carsten Bormann <[email-masked]> wrote:

> file.
> file with dead SIDs in it?  Or how do we maintain the memory of those dead
> SIDs?  Are there boundaries through which a SID file can go that does
> remove dead SIDs?
>
> guidance on how to handle the evolution of “the” SID file for a YANG module.
>
:-)

There are so many ways to mess up things with SIDs...
I hope the pyang regression tests for  SIDs are ready for the task.
There are multiple YANG lint programs and they don't all agree or get
everything right.
There are no such programs for SID (e.g. to verify everything that is
supposed to
have a SID assigned actually got an assignment).

As a hypothetical, let's assume there are undetected bugs in a SID file
or across the entire set of SID files. How do they get corrected?
The SID file itself contains no metadata indicating it is a corrected
revision of a previous SID file.

This has already proven to be a problem for YANG, and it has the
deviation-stmt
which can be used in some cases to correct broken YANG in another YANG file.

A SID file is just a JSON instance file.
There are no defined mechanisms to correct a SID file from IANA or anywhere
else.
Perhaps a "sid-file-revision" field in the SID file would help.

Grüße, Carsten
Andy

---

From: Michael Richardson <[email-masked]>
Date: Wed, 15 Jan 2020 19:08:42 -0500

Carsten Bormann <[email-masked]> wrote:
    >>> editing the document on my laptop and not telling anyone else what
    >>> I’m doing.  But at some time, they freeze.
    >>
    >> Actually, if things go well on your laptop, they don't.  You might
    >> add/delete leafs and so you might roll through the SIDs quickly, and
    >> then, because you haven't published anything, decided to reset, but
    >> that's your business :-)

    > My question really was if we can state unambiguously when it stops

Unfortunately, "my" in ambiguous here :-)
{/me considers demonstrating my abilities with German dative pronouns...}

I think that you mean "the document authors", but it could be that you mean
the WG's problem, or the CORE WG's problem.

I would like to suggest that WGs doing things with SID should ask for an
allocation at document adoption time.  This is the earlist that it can become
anyone's problem.

I think that a WGLC consideration would be whether to:
  1) reset the SID allocations for the document in question
  2) mark any deleted leafs as available

There is potentially recursive implication of this.
In an ideal world, documents will be proposed in a bottom-up order, but if
that does not occur, then this could cause this decision to cycle.

    > guidance on how to handle the evolution of “the” SID file for a YANG

I think we shouldn't try to make up all the rules ahead of time.
We just need to some running code first.

---

From: Michael Richardson <[email-masked]>
Date: Wed, 15 Jan 2020 19:12:03 -0500

Andy Bierman <[email-masked]> wrote:
    > or across the entire set of SID files. How do they get corrected?  The
    > SID file itself contains no metadata indicating it is a corrected

I guess revising the YANG date is too big a hammer.

    > deviation-stmt which can be used in some cases to correct broken YANG
    > in another YANG file.

    > A SID file is just a JSON instance file.  There are no defined
    > mechanisms to correct a SID file from IANA or anywhere else.  Perhaps a
    > "sid-file-revision" field in the SID file would help.

I agree, that would be worthwhile.

---

From: Michael Richardson <[email-masked]>
Date: Wed, 15 Jan 2020 19:17:19 -0500

Carsten Bormann <[email-masked]> wrote:
    >> Yes, if we evolved the design during WG processing, then we could well
    >> have dead SIDs.  We could, at WGLC or IESG approval, edit the SID file
    >> and mark them as useable again.  (If there were no running code, we
    >> could reset the SID file entirely at that point) The SID file
    >> maintains the memory of the dead values for only as long as we want it
    >> to.

    > This paragraph is exactly reflecting my nightmares about this.  We
    > could do a lot of things, if we wanted to.  What are our unambiguous,
    > not-to-be-missed instructions that make this fool-proof?

We will screw up, but the fools are often brilliant in their ability to get
around fool-proofing :-)

    > I would expect that we get rid of dead SIDs about as often as we revoke
    > TCP port number assignments, i.e., essentially never.  But this
    > requires getting around the cognitive dissonance of just having
    > “wasted” a SID value (*): With very strongly worded instructions, I’d
    > assume.

    > (*) (Note that there are way more SIDs than there are TCP ports.)

The time when we are most likely to create dead SID values is during the
development of running code between WG Adoption and WGLC.
I think it is reasonable thing to decide, with some communication among
implementers to reset/re-allocate.

Revisions to the YANG which deprecate leafs and therefore create deprecated
SID values are different: deployed code might still be using them.
(If we are revising published YANG, then certainly can never reset the SID process!)

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 Jan 2020 16:27:56 -0800

On Wed, Jan 15, 2020 at 4:12 PM Michael Richardson <[email-masked]>
wrote:

> The

That needs to stay the same, because the YANG module is not changing and
this field needs to
match the YANG file.

> I agree, that would be worthwhile.

This would make it possible to correct a SID file and still support the
broken version(s)
that might be deployed already. IANA would only need to maintain the latest
sid-file-revision.
I suggest a simple integer, not a datestamp.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 Jan 2020 16:53:51 -0800

> (If we are revising published YANG, then certainly can never reset the SID

IMO there should be no concept of conformance status for a SID file.
It would be unwise to recover SIDs of obsolete YANG definitions.
It is safer to just burn the wasted range and keep range assignments small.

Do SID files need to be identified by a 3-tuple {module-name,
module-revision, sid-file-revision }

   [email-masked]  -> { foo, 2020-01-15, 1 }

If the sid-file-revision is missing then it defaults to '1'.
Hopefully SID files will be correct and updated revisions will not be
needed.

I think a robust CORECONF client needs to know the 3-tuples of all the SID
files used by a server.
It is not enough just to know module@revision-date.  We want to be able to
rely on centralized (common) SID files and not have to retrieve each one
from each server (because they doctored the real SID file or replaced it
with their own).

Andy

---

From: Ivaylo Petrov <[email-masked]>
Date: Thu, 16 Jan 2020 11:28:24 +0100

Hello,

Thank you very much for the discussion. It brings a lot of interesting
perspectives on the problem how to make sure SIDs are most usable for
people and make it difficult to make mistakes.

I believe there are a few important points that need to be agreed upon
before we accept how to fix anything. Here are my assumptions:
1. we don't want to waste too much SIDs while creating SID modules
2. we want SIDs to be as stable as possible
3. we don't want to unpleasantly surprise people with the way SIDs
need to be used as compared to only using YANG modules.

My question is what happens currently with an implementation that uses
a version of YANG module that has been present inside a draft, but
have differences with the version in the final RFC that is published
from the draft? I could not see multiple revisions of YANG modules
associated with the same RFC in the IANA page. I also don't see update
of the revision information for a module that has been changed in
different versions of a draft. Therefore my impression is that people
that start using YANG modules that are not fully finished either are
ready to update their implementation later when the modules are
finalized or they control both ends and can handle the fact that they
use YANG module version that could be incompatible with other
implementations. Coming from assumption 3 then is my question - do we
need to make anything different? I don't think the SID file needs to
be published as available on IANA's web page before the publication of
an RFC. Doing so would open a huge potential for problems a number of
which have already been mentioned in this thread. For me this does not
contradict assumption 2 as people should be aware of the possible
stability issues before publication of the RFC and that should the
expected behaviour from their point of view considering it is already
the case with YANG modules. Then going back to early allocation, what
is done during early allocation is simply assigning ranges, not
registration of the SIDs themselves. Those ranges are used to generate
the .sid files in the draft and people can use the values from the
draft at their own risk if they so decide, but they should be aware
that those values can change. I suspect that I am missing something
here, but if I am not, then SID files do not need to include dead
SIDs. There might be SIDs that are present in one published version of
a module and not in the next one, but we can not delete them as the
older revision of the YANG module might still be in use by some
implementations. Note that this is not a waste in this case, so we are
not violating assumption 1.

I am not sure I see how a version of the .sid file will help here. My
concern is that it will make finding the SID file that one wants more
difficult - one needs to know both the yang version and a sid file
version + the sid in order to be able to understand it. Currently one
can find the sid file that covers a range and from inside it figure
out which revision of which module is used and that is
straight-forward. Having multiple .sid files also means that
verification of SIDs will be that much more difficult as it will be
spread possibly over a number of files.

Andy, you have an interesting point about what will happen if there is
a problem with our .sid file generation tool. Could you give examples
of the types of errors that would cause significant issues? My general
answer is - if we have published the .sid file, some SIDs might be
lost. This is better than the alternative of modifying the meaning of
a SID leading to possible ambiguity. Fixing such errors might require
updating some .sid files and doing firmware update, which would not be
a pleasant thing. However for me one vital part of the role of the
Experts is to check that the output of the tool makes sense and to
guard against big allocation issues. A possibly interesting thing to
do would be to create another version of the SID allocation tool. That
way having two independent implementations would greatly reduce the
risk of issues if they produce the same output. Another safeguard here
is the fact that SID ranges are independent, therefore if the tool
messes up something, it should not be possible to do so in a way that
affects other ranges. I currently don't see how a problem - big, but
subtle, in the generation software can create a situation that would
make me want to change the SID architecture.

Best regards,
Ivaylo

On Thu, Jan 16, 2020 at 1:54 AM Andy Bierman <[email-masked]> wrote:
>
>
>
> On Wed, Jan 15, 2020 at 4:17 PM Michael Richardson <[email-masked]> wrote:
>>
>>
> Do SID files need to be identified by a 3-tuple {module-name, module-revision, sid-file-revision }
>
> Hopefully SID files will be correct and updated revisions will not be needed.
>
> I think a robust CORECONF client needs to know the 3-tuples of all the SID files used by a server.
> from each server (because they doctored the real SID file or replaced it with their own).
>
>
> Andy
>
>
>
>>
>> --

---

From: Andy Bierman <[email-masked]>
Date: Thu, 16 Jan 2020 03:35:53 -0800

> need to be used as compared to only using YANG modules.
YANG module development has proven to be a messy process.
It is the exception, not the rule, that a module stays stable from draft-00
to RFC.

> from the draft?

The implementor is on their own, just like now for YANG,
and just like MIB modules for 30+ years.  Vendors know that
implementing a work-in-progress is dangerous.  WGs should know
that publishing an RFC with zero implementation experience for the
technology in the draft is going to produce an inferior result.

Only the RFC versions of MIB or YANG modules is ever recorded by IANA.
The work-in-progress versions are extracted from drafts.
The YangModels github repo stores these modules.

> I could not see multiple revisions of YANG modules
Relying on the YANG to SID algorithm to generate the SID file
assumes that every tool will generate the SID file correctly and
every tool is perfect and never has a new bug or a regression bug.
It is more realistic that SID interoperability will be achieved using
actual SID files
downloaded from a common repo.  A SID file version allows for
corrections to the YANG to SID translation.  The YANG module revision
is not changing.  It is not at fault.  Multiple SID files for the same YANG
revision
should be distinguishable.

> make me want to change the SID architecture.
It would be naive to think that pyang cannot possibly have a bug in it,
or that YANG complexity in real modules is low enough to simply inspect
the file output like your constrained voucher example. How does the output
look
for ietf-bgp.yang (or any of the recent routing modules).

When a YANG module has a bug in it, a new revision with a new revision date
is
published to fix it. IMO it would be wise to provide a mechanism to fix a
bug in
a SID file.

Multiple independent implementations are always good for finding bugs in
the tools.
There are so many ways to change a YANG module, I suspect that the code
that updates a SID file (as opposed to resetting the SID translation each
time)
has the most opportunity for bugs.

Best regards,

Andy

---

From: Andy Bierman <[email-masked]>
Date: Thu, 16 Jan 2020 04:50:04 -0800

> need to be used as compared to only using YANG modules.

These goals are not going to be easy to achieve.
The biggest challenge is maintaining existing SIDs from draft to draft.

There is a significant amount of metadata needed to generate a SID file,
starting with all the previous YANG revisions or SID files, and when/if
the SID files have been reset vs. updated with preserved numbers.

SID assignments depend on statement order.
YANG identifiers do not depend on statement order at all.
Most definitions are actually conceptual, and only the
conceptual expansion of submodules, groupings, augments, and uses
can be numbered. This leads to all sorts of opportunities for bugs.
Some changes are difficult to detect (like 2 augments that get combined)
which have zero impact on YANG identifiers but a huge impact on SIDs.

My biggest concern about SID generation is that it depends on all
the revisions of all the imported modules.  The SID file for module X
is dependent on the groupings expanded from module Y and Z
(and Z imports A, which imports B, etc.)

If any of these imported modules change at all, then the SID file for the
target module
can change, even though that module did not change at all. The date that
the SID file is generated can change the outcome (!)  How does the SID tool
make sure the exact same set of imported modules is used?  How are these
dependencies documented in the SID file so generation of version N = 1 can
detect
object changes correctly?

Andy

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 16 Jan 2020 18:47:19 +0100

> the SID files have been reset vs. updated with preserved numbers.

Well, specifically, the current YANG module and the previous SID file; not the entire history of the universe.

The need for the previous SID file means that there needs to be a single lineage; SID files cannot be independently progressed.  This calls for central entity (or a blockchain :-).
What we need to define unambiguously is who that entity is.
One we are at the IANA, the Designated Expert (DE) can play this role.
But we need to define the process leading up to this.
(Yes, that process will evolve, and we probably also need to allow exceptions.
But there must be strong guide rails that work for the vast majority of cases.)

Adding a version number to the SID file could indeed help with accidents/disasters.
But it could also lead to people thinking they can get away with concurrent lineages; thinking that version numbers will fix any fallout.
That would be a disaster on its own, most likely leading to permanent universe splits.
So I’m not so sure we want to open that Pandora’s box.

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Thu, 16 Jan 2020 18:17:10 -0500

mcr> Revisions to the YANG which deprecate leafs and therefore create
    mcr> deprecated
    mcr> SID values are different: deployed code might still be using them.
    mcr> (If we are revising published YANG, then certainly can never reset
    mcr> the SID process!)

Andy Bierman <[email-masked]> wrote:
    andy> IMO there should be no concept of conformance status for a SID file.
    andy> It would be unwise to recover SIDs of obsolete YANG definitions.
    andy> It is safer to just burn the wasted range and keep range assignments
    andy> small.

I think that we are in agreement here.

    andy> Do SID files need to be identified by a 3-tuple {module-name,
    andy> module-revision, sid-file-revision }

    andy> [email-masked] -> { foo, 2020-01-15, 1 }

seems right to me.

    andy> I think a robust CORECONF client needs to know the 3-tuples of all the
    andy> SID files used by a server.
    andy> It is not enough just to know module@revision-date. We want to be able
    andy> to
    andy> rely on centralized (common) SID files and not have to retrieve each
    andy> one
    andy> from each server (because they doctored the real SID file or replaced
    andy> it with their own).

I think that you are saying that SID requires changes to the YANG module meta
data which I think is available in some YANG (meta-)module?
If we all agree that SID values can be deprecated, but never reused (And I do
not hear anyone saying otherwise), then the highest sid-file-revision is
always the correct one.

If we revise the module, and we get a new module-revision, I do not think
that we reset the sid file, nor do we reset the sid-file-revision.  Do you
concur?

---

From: Michael Richardson <[email-masked]>
Date: Thu, 16 Jan 2020 18:34:42 -0500

{Trying to answer several emails in this thread at once.}

    >> before we accept how to fix anything. Here are my assumptions:

"Petrov's Three Laws of SID allocation":
[in the spirit of: https://en.wikipedia.org/wiki/Three_Laws_of_Robotics]

    >> 1. we don't want to waste too much SIDs while creating SID modules

I would have preferred if you hadn't made this the first point.

    >> need to be used as compared to only using YANG modules.

    andy> YANG module development has proven to be a messy process.
    andy> It is the exception, not the rule, that a module stays stable from draft-00
    andy> to RFC.

So, again, I advocate that at WGLC, that the WG consider if they wish to
remove and/or renumber any SID values that have been allocated since WG Adoption.
(Of course, we might being doing a BIS document, so there is history before)

I see Carsten's point about the sid-file-revision could lead to people being
attached to particular lineages.  I don't think that is as likely; but I'm
not going to die on this hill.  It's JSON, we could add it later if we needed
it, but it would be better to specify what it is.  I am happy with an integer
that increments.  I think that the SID writer should probably insert it's
name, revision, the date it ran, who ran it, the phase of the moon, and the
current cost of a .org domain, into the file if it changes anything.

    >> have differences with the version in the final RFC that is published

    andy> The implementor is on their own, just like now for YANG,
    andy> and just like MIB modules for 30+ years.  Vendors know that
    andy> implementing a work-in-progress is dangerous.  WGs should know
    andy> that publishing an RFC with zero implementation experience for the
    andy> technology in the draft is going to produce an inferior result.

I think that you intended to identify a tussle here, but maybe it was rather subtle?

On the one hand: work-in-progress can be dangerous to implement.
On the other hand: having no implementation feedback produces inferior
       results.

The discussion we are having here is how to enhance the stability of the spec
in order to reduce cost/risk of early implementations, while at the same time
allowing the WG to regret (and reverse) early decisions.

I think that the flow I have suggested works: allocate at adoption time,
remove dead-wood at WGLC, and make renumbering a WGLC consideration.

    andy> Only the RFC versions of MIB or YANG modules is ever recorded by IANA.
    andy> The work-in-progress versions are extracted from drafts.
    andy> The YangModels github repo stores these modules.

Andy, should sid files be included in <CODE BEGINS> then along with YANG
modules?  We don't say anything about that, I think.  I hadn't thought about this.

---

From: Andy Bierman <[email-masked]>
Date: Thu, 16 Jan 2020 16:40:45 -0800

On Thu, Jan 16, 2020, 3:17 PM Michael Richardson <[email-masked]>
wrote:

> I think only the last sid file and the new module revision are needed,
> like Carsten said.
>

The sid revision would start at 1 for each module, revision date.
I still think it is useful to be able to correct a sid file with errors.
All code has bugs and there will be bugs found that will require edits to
the sid file.

Andy

--

---

From: Andy Bierman <[email-masked]>
Date: Thu, 16 Jan 2020 16:47:34 -0800

On Thu, Jan 16, 2020, 3:34 PM Michael Richardson <[email-masked]>
wrote:

> (Of course, we might being doing a BIS document, so there is history before

I agree with this plan.

> I think that you intended to identify a tussle here, but maybe it was

It is a delicate balance.
Vendors are not supposed to complain when the draft changes and running
code has to be rewritten.
.

> modules?  We don't say anything about that, I think.  I hadn't thought

Yes. They are code components

Andy

---

From: Andy Bierman <[email-masked]>
Date: Sat, 18 Jan 2020 21:36:49 -0800

On Thu, Jan 16, 2020 at 9:47 AM Carsten Bormann <[email-masked]> wrote:

> The need for the previous SID file means that there needs to be a single
> lineage; SID files cannot be independently progressed.  This calls for
> central entity (or a blockchain :-).
> But it could also lead to people thinking they can get away with
> concurrent lineages; thinking that version numbers will fix any fallout.

We already opened Pandora's box by attempting to create a 1-flat-number
globally unique, multi-module,
multi-owner object identifier. I think YANG is quite different than the MAC
address example of success
in this endeavour because YANG modules need the numbers assigned in the
development phase,
not the manufacturing phase.

YANG has many features that make it easier on the writer and harder on the
tool maker,
such as no order dependencies, but is has some guardrails. No matter how
the writer may
possibly alter or refactor a YANG module, nothing unintentional can change
the YANG identifier for
any defined data node.  SID has no such guardrails.

I think there is a practical solution for deployment, which does not
require the use of the
centrally maintained SID files, which is not realistic for 4 reasons:
  - no way to apply changes to an incorrect SID file (without a
sid-file-identifier)
  - no way for vendors to apply deviations on any YANG modules, e,.g
deviation { not-supported; }
  - the SID file generation depends on the specific revisions of all
modules imported
    and submodules included by the target YANG module
  - vendors are already failing to deploy YANG modules from a central
source model.
    The NETCONF and RESTCONF protocols have robust mechanisms to discover
and
    retrieve the actual YANG files used by the server, and vendors rely on
this feature.
    Some vendors need model branching and other complexity that make SID
assignment even harder
    (see
https://tools.ietf.org/html/draft-ietf-netmod-yang-versioning-reqs-02)

CORECONF (and SID in general) needs a SID file retrieval mechanism.
Perhaps based on YANG Data Instance Files
https://tools.ietf.org/html/draft-ietf-netmod-yang-instance-file-format-06
A server will make the URI of this resource known somehow, which can be
used if needed,
before a client starts using SIDs.

In addition, or instead of listing YANG modules, the data is SID files,
maybe 1 super-SID-file JSON object.
It is not realistic that a client will be able to use all global SID files
based on the YANG module name
and revision date, especially in the development phase. The IANA SID file
may be useful after the YANG
module is very stable.

More solution details:
  - The WG or vendor can maintain an informal public archive of SID files.
  - The YangModels repo update process should include SID files.
https://github.com/YangModels/yang
  - Vendors still need the public SID file for the starting point, or maybe
use it unchanged
  - The SID files need <CODE BEGINS> support and probably some tool changes
in the ID-submission process
  - IANA will only archive the RFC version, just like YANG modules.

Grüße, Carsten

Andy

---

From: Ivaylo Petrov <[email-masked]>
Date: Tue, 21 Jan 2020 11:47:32 +0100

Hello Michael,

See my answers below

> I would have preferred if you hadn't made this the first point.

It was unordered list, I just wanted to be able to refer to the
different points.
I should have made that clear.

> (Of course, we might being doing a BIS document, so there is history before)

Sounds good to me.

> current cost of a .org domain, into the file if it changes anything.

The cost of .org domain could indeed be useful :) While I can surely agree that
we should allow name, tool version and date to be inserted, I don't think this
information should be mandatory. For some YANG modules it might be fine for
 people to write by hand the .sid files (although I don't think this
is ever a good idea).

> modules?  We don't say anything about that, I think.  I hadn't thought about this.

I agree with that.

--
Best regards,
Ivaylo

---

From: Ivaylo Petrov <[email-masked]>
Date: Tue, 21 Jan 2020 11:55:27 +0100

Hello Andy,

Find my answers below

On Sun, Jan 19, 2020 at 6:37 AM Andy Bierman <[email-masked]> wrote:
> We already opened Pandora's box by attempting to create a 1-flat-number globally unique, multi-module,
> multi-owner object identifier. I think YANG is quite different than the MAC address example of success
> in this endeavour because YANG modules need the numbers assigned in the development phase,
> YANG has many features that make it easier on the writer and harder on the tool maker,
> such as no order dependencies, but is has some guardrails. No matter how the writer may
> possibly alter or refactor a YANG module, nothing unintentional can change the YANG identifier for

[ivaylo]:
It seems that you are implying that refactoring a YANG without changing the
YANG identifiers and the paths will cause changes to the .sid file. Could you
provide an example as I am not sure I see why this needs to be the case.
Note that I added the restriction of not changing paths as otherwise I
believe the
NETCONF representation will also change, so CORECONF will be no different.

>
> I think there is a practical solution for deployment, which does not require the use of the
>   - no way to apply changes to an incorrect SID file (without a sid-file-identifier)

[ivaylo]:
I believe currently the way to achieve this is to have a new version
of the YANG module,
so that a new version of the SID file can be generated. Having a
global version of .sid
file that is an integer that progresses lineary seems as a good
solution to me. It should
also be able to accommodate your concern here I believe. As Michael
said, the latest
version of the .sid file should be able to handle both all clients of
previous versions of
the YANG module and the latest one. As there could be multiple versions for one
version of a YANG module, errors could be corrected. Does this sound
good to you?

>   - no way for vendors to apply deviations on any YANG modules, e,.g deviation { not-supported; }

[ivaylo]:
I am not sure I see why that is the case. In sec 3 of
draft-ietf-core-yang-library we have

* 'feature' and 'deviation' are implemented using a SID (i.e. an
unsigned integer).

While I agree that more details can be added here, I don't see any
reason deviations
not to be supported.

>   - the SID file generation depends on the specific revisions of all modules imported

[ivaylo]:
I agree, but I believe this is a good thing.  As is said sec 5.1.1 of RFC6020

` modules need to be imported using specific revisions`.

Could you provide more details of the issue you see?

>   - vendors are already failing to deploy YANG modules from a central source model.
>     The NETCONF and RESTCONF protocols have robust mechanisms to discover and
>     retrieve the actual YANG files used by the server, and vendors rely on this feature.
>     Some vendors need model branching and other complexity that make SID assignment even harder
>     (see https://tools.ietf.org/html/draft-ietf-netmod-yang-versioning-reqs-02)
>

[ivaylo]:
I believe that the envisioned .sid file retrieval mechanism is:
1. You find the sid that you are interested in
2. You check in which mega range it is and who is the operator of that
mega range in
the IANA registry.
3. You contact that mega range operator through a given protocol to
obtain further details.

Presently we believed as there are not going to be too many mega range
registries and
we have no experience operating them it might be unwise to specify the
protocol that
they need to implement in order for clients to be able to query them.
Specifying such a
protocol might be something we do if this is a really big concern for people.
Would that retrieval mechanism cover the cases that you were thinking
of? Are there other
cases that you believe it should cover?

> Perhaps based on YANG Data Instance Files https://tools.ietf.org/html/draft-ietf-netmod-yang-instance-file-format-06
> A server will make the URI of this resource known somehow, which can be used if needed,
> In addition, or instead of listing YANG modules, the data is SID files, maybe 1 super-SID-file JSON object.
> It is not realistic that a client will be able to use all global SID files based on the YANG module name
> and revision date, especially in the development phase. The IANA SID file may be useful after the YANG
> module is very stable.
>
> More solution details:
>   - The YangModels repo update process should include SID files.  https://github.com/YangModels/yang
>   - Vendors still need the public SID file for the starting point, or maybe use it unchanged
>   - The SID files need <CODE BEGINS> support and probably some tool changes in the ID-submission process

--
Best regards,
Ivaylo

---

From: Andy Bierman <[email-masked]>
Date: Tue, 21 Jan 2020 11:29:00 -0800

On Tue, Jan 21, 2020 at 2:55 AM Ivaylo Petrov <[email-masked]> wrote:

> >> The need for the previous SID file means that there needs to be a
> single lineage; SID files cannot be independently progressed.  This calls
> for central entity (or a blockchain :-).

YANG modules under development do not follow any of the "module update"
rules
so there are no restrictions that can be relied on in real YANG development.
This requires the tools and developers to be more careful, especially
since YANG tools have no existing reason to be careful in these areas.

> version of a YANG module, errors could be corrected. Does this sound

This will not work at all.
There is no reason to update the YANG module.
What would change? Just the revision-date?

Even if we ignore all possibility that a SID file can have a flaw in it,
there is still the problem caused when imported modules change.
Reusable groupings in a common module are all the rage.
It is very possible that the target module may never even change after
release 1

   module foo {
        ...
        import A { prefix a; }
        revision 2019-01-01;

        container top {
           uses a:grouping1;
       }
  }

The SID file contents for /top are not determined by module foo.
Over time, module A will keep changing and module foo will never change.
All of the unspecified revisions imported by module foo determine the
expansion of 'grouping1.
This approach only works within a server implementation because all the
module revisions
are known to the client via the YANG library.

> ` modules need to be imported using specific revisions`.

This is not how YANG is used 95% of the time.
Even if I import-by-revision, it is very common for that module to
have its own imports, and I have no control over the revisions used there.

> 3. You contact that mega range operator through a given protocol to

No. I am interested in loading the SID data from each individual server, as
needed.

I understand some people want to hard-wire SIDs into their client or server
and they will not have the YANG identifiers.  I am not interested in this
deployment option for YANG-Push so the SID values will get added at runtime.
The client will consume or use cached the SID files at
session-startup-time, just like the YANG modules.

> cases that you believe it should cover?

I think vendors will continue to fill in the gaps with proprietary
mechanisms.
The SID file format will get augmented, etc. to support real deployment
scenarios.
This could be considered a protocol issue and can be ignored for SID files
in general.

How the client gets the YANG and SID files could be considered an
implementation detail.
The IANA repo is there but there is no requirement to use it.

Andy

---

From: Ivaylo Petrov <[email-masked]>
Date: Wed, 22 Jan 2020 12:44:18 +0100

Thank you for your clarification, I think we are getting closer to
understanding the difference of viewpoints. Find my comments below.

On Tue, Jan 21, 2020 at 8:29 PM Andy Bierman <[email-masked]> wrote:
> YANG modules under development do not follow any of the "module update" rules

[ivaylo]:
If there are no restrictions to what a YANG module change can bring, I
am afraid we can not provide much help for .sid files. The suggestion
that any implementer of pre-WGLC version of the draft might need to
adjust her implementation due to SIDs change seems acceptable to me and
I don't see how we can provide anything better.

> It is very possible that the target module may never even change after release 1
>
>    module foo {
>         ...
>         import A { prefix a; }
>         revision 2019-01-01;
>
>         container top {
>            uses a:grouping1;
>        }
>   }
>
> All of the unspecified revisions imported by module foo determine the expansion of 'grouping1.
> This approach only works within a server implementation because all the module revisions

[ivaylo]
I think the difference of viewpoints here came from my assumption that
all yang modules import specific versions of other modules. You point
out in your email that in practice this is not necessarily the case. If
we want to be able to handle this as I believe we are, I agree that we
might need to add information in the SID file about what version of each
of the dependent modules had been used to allocate any given SID as
otherwise we might have serious problems figuring this out at runtime,
if this is at all possible.

Storing this information can be solved by having many independent
versions of .sid files - one for a given set of YANG modules and
revisions or we can have each SID number allocation contain more
information than just the path to that node - namely the versions of all
the YANG modules and the revision information.  In both cases it appears
that this will complicate how SIDs allocation.

Andy, do you agree that I have captured the essence of your concern
correctly? Are there any thoughts whether we want to solve this problem
and which is the best approach (even if it is not any of the ones
already mentioned)?

I might be missing some historical background here, but my personal
preference is to go with the second approach as it would allow for
linear versioning of the .sid files, which I believe is lighter on the
constrained device. It seems that your preference, Andy, is to go with
the first one and have SIDs be more contextually dependent.

> No. I am interested in loading the SID data from each individual server, as needed.
>
> The client will consume or use cached the SID files at session-startup-time, just like the YANG modules.

[ivaylo]:
You are correct that I was looking much more from the perspective of
hard-wiring the SIDs on constrained devices and assume that
introspection will be done as needed using the central IANA registry or
other server with such information. If you want to know what .sid file
is present for each server, wouldn't the mechanism from
draft-ietf-core-yang-library solve that? Or maybe you would like to have
a more compact reply that gives only the YANG module name, version and
version the SID file that can be found on a server - be it internal or
external one. If that is the need you have, we can see which is the best
approach to resolve it.

> I think vendors will continue to fill in the gaps with proprietary mechanisms.
> The SID file format will get augmented, etc. to support real deployment scenarios.
> This could be considered a protocol issue and can be ignored for SID files in general.
>
> How the client gets the YANG and SID files could be considered an implementation detail.

[ivaylo]:
I assume that one of the participants is a constrained device, which
might not be exactly what you are considering, but in that case if the
constrained device is a client, it is very unlikely for it to try to
discover at runtime what are the SIDs that a server will support. It
will generally send CORECONF requests that contain some SIDs and assume
that the server or a proxy in between will be able to understand their
meaning. That is where the IANA repository and a protocol for
inspection how a SID maps can be useful.

Then assuming that the constrained device is the server, a client can
discover the supported SIDs either through introspection protocol, which
will be heavy, but might be required in some cases or more likely
through provisioning/bootstrap.

It seems to me that you are considering also the question if none of the
endpoints is really constrained. In that case I believe the client will
still be able to use the same introspection protocol as before and find
the YANG modules looking at the SID file.

--
Best regards,
Ivaylo

---

From: Andy Bierman <[email-masked]>
Date: Wed, 22 Jan 2020 09:52:43 -0800

On Wed, Jan 22, 2020 at 3:44 AM Ivaylo Petrov <[email-masked]> wrote:

> >> >> Well, specifically, the current YANG module and the previous SID
> file; not the entire history of the universe.
> >> >>
> >> > We already opened Pandora's box by attempting to create a
> 1-flat-number globally unique, multi-module,

I think that WGs will want to regenerate the files during development, even
though
it often takes the IETF 2 - 5 years to complete a YANG module.  Nodes get
renamed a lot
and every time that is done it creates new SIDs and leaves behind
irrelevant SIDs.
It might takes 5x - 10x the required range by the time the WG is done.

It is not even clear that I-Ds (or RFCs) can include SID files, since they
rely on
the imports used to generate the file.  I agree that there is not much that
can (or should)
be done to pretend that work-in-progress has any stability.

> the first one and have SIDs be more contextually dependent.

I don't see how the 2nd approach can scale to even 20 YANG modules, let
alone 100s.
If there are multiple variants of every SID file, then there is no point in
an "official" SID file
at all. Only the mappings for a specific server implementation can be
used.  Even products
from the same vendor will use different revisions of the same imported
module.

It is quite possible that common SID files will be practical after
development is done
for a module and it is quite stable. There will probably be a mix of
deployment strategies
depending on module maturity and implementation details.

> >> ` modules need to be imported using specific revisions`.

The YANG import-by-revision mechanism is fatally flawed and has been mostly
worthless
since Day One.  Perhaps this will be fixed someday but until then, most
designers do
not specify the exact revision that must be used.

> external one. If that is the need you have, we can see which is the best

You can hardwire SIDs based on proprietary vendor information or IANA
information.
I don't see how IANA can maintain any SID files though, since the entire
set of imported modules
depends on the SID file generation date and modules available to the tools
used by by
authors submitting files to IANA

> >> > In addition, or instead of listing YANG modules, the data is SID
> files, maybe 1 super-SID-file JSON object.

Andy

---

From: Ivaylo Petrov <[email-masked]>
Date: Thu, 23 Jan 2020 11:14:05 +0100

Find my answers below

> be done to pretend that work-in-progress has any stability.

[ivaylo]:
Maybe I failed to provide enough details and I might have been
misunderstood here, sorry if this is the case, but my idea was that
before WG adoption all bets are off for stability of a .sid file. Once a
draft is adopted, the authors should make sure they don't regenerate the
.sid file from scratch for every little change. While renaming could be
done by changing the .sid file (automatically or not) making the old SID
value point to the new name, that might cause other problems. Unless
someone believes this is something we should explicitly allow, I would
prefer to let the authors decide if this would be safe in they case or
not. Once WGLC is done, the .sid file should not reallocate SID values
unless this is discussed on the mailing list. I will add text about this
in the draft this week and publish a new version.

> depending on module maturity and implementation details.

[ivaylo]:
After some more consideration, I agree that it adds a lot of complexity
for something that I am not sure we want to support as a first class
citizen. I agree that we should have support for handling multiple
versions of the dependencies, where I am not that sure is that we should
make this as effortless as possible if that would add that much
complexity. A simpler approach that would fit that bill for me is that
each .sid file version N has all the SIDs present in version N-1 without
changes for any N>0 (assuming that 0 is the initial version). The .sid file
shall also have a field with the list of versions of all the modules that
were used to generate the current version of that .sid file. That way
only traversing the .sid file versions one can discover what versions of
each YANG module were used for the generation of each SID. At the same
time we don't add too much more information to each .sid file. As
Carsten suggested, the latest version always wins.

The drawback I see is that we might carry around a number of SIDs
that are no longer needed. While in some situations that might be
unnecessary, I believe there is no way to avoid it without risking
ambiguity or requesting a client to do even more work.

> not specify the exact revision that must be used.

[ivaylo]:
I believe my latest proposal will handle appropriately that problem.

> authors submitting files to IANA

[ivaylo]:
The proposal to keep track of the YANG module versions used to generate
any given .sid file handles this concern I believe.

--
Best regards,
Ivaylo