Subject: [i2rs] Review of draft-ietf-i2rs-ephemeral-state-11

From: Robert Wilton <[email-masked]>
Date: Mon, 27 Jun 2016 16:30:03 +0100

Hi,

I've reviewed draft-ietf-i2rs-ephemeral-state-11 and given a few minor 
comments below.  Generally I think that I understand the requirements 
stated in this document.

Minors comments:

1) Ephemeral-REQ-01 (page 5):

    Ephemeral-REQ-01: I2RS requires ephemeral state; i.e. state that does
    not persist across reboots.  If state must be restored, it should be
    done solely by replay actions from the I2RS client via the I2RS
    agent.

The architecture document indicates that the ephemeral state would (or 
is that may) also be lost on other circumstances such as process restart 
of the I2RS agent.  Does this need to be clarified in the requirement?  E.g.

    Ephemeral-REQ-01: I2RS requires ephemeral state; i.e. state that MUST
    NOT persist across reboots of the device or I2RS Agent subsystem. If
    state must be restored, it should be done solely by replay actions
    from the I2RS client via the I2RS agent.

2) Hierarchy: (page 5)
Based on the previous description, I would possibly split 
Ephemeral-REQ-06 up into the following requirements:

Old (from ephemeral-state:10):

    Ephemeral-REQ-06: The ability to augment an object with appropriate
    YANG structures that have the property of being ephemeral.  An object
    defined as any one of the following: yang module, submodule or
    components of submodule, or schema node.

Old (from ephemeral-state-11):

    Ephemeral-REQ-06: The ability to augment Yang schema nodes with
    additional Yang Schema nodes that have the property of being
    ephemeral.

Proposed:

    Ephemeral-REQ-06:
  
    1. The ability to define a YANG module or submodule schema that
    only contains data nodes with the property of being ephemeral.

    2. The ability to augment a YANG data model with additional YANG
    schema nodes that have the property of being ephemeral.

3) Ephemeral-REQ-07: (page 6):

    Ephemeral-REQ-07: Ephemeral configuration state could override
    overlapping local configuration state, or vice-versa.
    Implementations MUST provide a mechanism to choose which takes
    precedence.  This mechanism MUST include local configuration (policy)
    and MAY be provided via the I2RS protocol mechanisms.

I note that this requirement doesn't specify the scope of whether the 
override mechanism should operate globally or on a per data node basis.  
I'm not sure whether this needs to be clarified - since the text in the 
architecture document makes it pretty clear that a global level 
resolution is sufficient.

4) Ephemeral-REQ-08: (page 6):
Similar to Juergen's comments, I'm concerned about the 
writable/non-writable requirement.

    Ephemeral-REQ-08: Yang MUST have a way to indicate in a data model
    that schema nodes have the following properties: ephemeral, writable/
    not-writable, and status/configuration.

I'm somewhat adverse to writable operational state, and hence I would 
prefer if this requirement was watered down to something like:

    Ephemeral-REQ-08: In addition to config true/false, there MUST be a
    way to indicate that YANG schema nodes represent ephemeral state.
    It is desirable to allow for, and have to way to indicate, config
    false YANG schema nodes that are writable operational state.

5) Ephemeral-Req-12, page 7:
Presumably the requirement is that the notification must indicate the 
node that we involved in the collision?  I.e. it isn't sufficient to 
just signal to the client that there has been a collision with some of 
their configuration?

    Ephemeral-REQ-12: When a collision occurs as two clients are trying
    to write the same data node, this collision is considered an error
    and priorities were created to give a deterministic result.  When
    there is a collision, a notification MUST BE sent to the original
    client to give the original client a chance to deal with the issues
    surrounding the collision.  The original client may need to fix their
    state.

Should this be made explicit?  E.g. perhaps:

    Ephemeral-REQ-12: When a collision occurs as two clients are trying
    to write the same data node, this collision is considered an error
    and priorities were created to give a deterministic result.  When
    there is a collision, a notification (indicating which data node the
    collision occurred on) MUST BE sent to the original client to give
    the original client a chance to deal with the issues surrounding
    the collision.  The original client may need to fix their state.

6) Ephemeral-REQ-14, page 7:
I would suggest potentially rewording this to make the requirement and 
leeway on the solution more explicit.

    Ephemeral-REQ-14: If two clients have the same priority, the
    architecture says the first one wins.  The I2RS protocol has this
    requirement to prevent oscillations between clients.  If one uses the
    last wins scenario, you may oscillate.  That was our opinion, but a
    design which prevents oscillation is the key point.

Proposed alternative text:

    Ephemeral-REQ-14: A deterministic conflict resolution mechanism MUST
    be provided to handle the error scenario that two clients, with
    the same priority, update the same configuration data node.  The I2RS
    architecture gives one way that this could be achieved, by
    specifying that the first update wins.  Other solutions, that prevent
    oscillation of the config data node, are also acceptable.

Cosmetic comments:

7) Section 1. Introduction.  The requirements are no longer version 
specific, hence perhaps update the following text from:

    1.  select features from YANG, NETCONF, and RESTCONF per version of
        the I2RS protocol (See sections 4, 5, and 6)

    2.  propose additions to YANG, NETCONF, and RESTCONF per version of
        the I2RS protocol for key functions (ephemeral state, protocol
        security, publication/subscription service, traceability),

to:

    1.  select features from YANG, NETCONF, and RESTCONF for the initial
        I2RS protocol version (See sections 4, 5, and 6).

    2.  propose additions to YANG, NETCONF, and RESTCONF for the initial
        I2RS protocol version for key functions (ephemeral state, protocol
        security, publication/subscription service, traceability).

8) Section 1. introduction.  I'm not sure that this 3rd bullet is 
relevant, and possibly could be removed, although equally it doesn't 
seem to do any harm:

    3.  suggest protocol strawman as ideas for the NETCONF, RESTCONF, and
        YANG changes.

9) The document uses a mix of "Yang" and "YANG", probably should just 
use "YANG".

10) Section 3. Ephemeral State Requirements:

The document refers to "ephemeral configured state" here, but elsewhere 
(e.g. 3.4) "ephemeral configuration" or "ephemeral configuration state" 
is used.  It might be helpful for use of these terms to be consistent, I 
would suggest "ephemeral configuration" is sufficient.

11) Ephemeral-REQ-13, page 7:

Minor omission in the last sentence.

   Ephemeral-REQ-13: The requirement to support multi-headed control is
    required for collisions and the priority resolution of collisions.
    Multi-headed control is not tied to ephemeral state.  I2RS is not
    mandating how AAA supports priority.  Mechanisms which prevent
    collisions of two clients trying the same node of data are the focus.

Proposed:

   Ephemeral-REQ-13: The requirement to support multi-headed control is
    required for collisions and the priority resolution of collisions.
    Multi-headed control is not tied to ephemeral state.  I2RS is not
    mandating how AAA supports priority.  Mechanisms which prevent
    collisions of two clients trying to modify the same node of data
    are the focus.

12) Ephemeral-REQ-15, page 7:
I would suggest that it might be better to refer to "Ephemeral state" 
rather than the I2RS ephemeral data-store.

    Ephemeral-REQ-15: Section 7.9 of the [I-D.ietf-i2rs-architecture]
    states the I2RS architecture does not include multi-message atomicity
    and roll-back mechanisms.  I2RS notes multiple operations in one or
    more messages handling can handle errors within the set of operations
    in many ways.  No multi-message commands SHOULD cause errors to be
    inserted into the I2RS ephemeral data-store.

Proposed:

    Ephemeral-REQ-15: Section 7.9 of the [I-D.ietf-i2rs-architecture]
    states the I2RS architecture does not include multi-message atomicity
    and roll-back mechanisms.  I2RS notes multiple operations in one or
    more messages handling can handle errors within the set of operations
    in many ways.  No multi-message commands SHOULD cause errors to be
    inserted into the ephemeral state.

Thanks,
Rob

---

From: "Susan Hares" <[email-masked]>
Date: Thu, 30 Jun 2016 12:15:29 -0400

Robert: 

Thank you for your comments.  See resolution of comments below.  Version -12 will be sent to the list to handle most of these comments. 

Sue 

From: i2rs [mailto:[email-masked]] On Behalf Of Robert Wilton
Sent: Monday, June 27, 2016 11:30 AM
To: [email-masked]
Subject: [i2rs] Review of draft-ietf-i2rs-ephemeral-state-11

Hi,

I've reviewed draft-ietf-i2rs-ephemeral-state-11 and given a few minor comments below.  Generally I think that I understand the requirements stated in this document.

Minors comments:

The architecture document indicates that the ephemeral state would (or is that may) also be lost on other circumstances such as process restart of the I2RS agent.  Does this need to be clarified in the requirement?  E.g.

Sue: In our previous discussions,  other people suggest that “reboots” covered hardware and software reboot of the I2RS agent.   And that the specific nature of the reboot was too-much information for the requirement. 

2) Hierarchy: (page 5)
Based on the previous description, I would possibly split Ephemeral-REQ-06 up into the following requirements: 

Proposed:

   Ephemeral-REQ-06:

I will accept this change and release in it version-12.  

I note that this requirement doesn't specify the scope of whether the override mechanism should operate globally or on a per data node basis.  I'm not sure whether this needs to be clarified - since the text in the architecture document makes it pretty clear that a global level resolution is sufficient.

Sue:  Again, we were cautioned to not specify the design, but only provide a result.  The global is sufficient. 

4) Ephemeral-REQ-08: (page 6):
Similar to Juergen's comments, I'm concerned about the writable/non-writable requirement.

I'm somewhat adverse to writable operational state, and hence I would prefer if this requirement was watered down to something like:

Sue: You are Juergen are concerned about writeable/non-writeable.   Martin is concerned about status/configuration.  The I2RS authors believe we are running into the different operational state models.  Each time we change this requirement we get another response.   The requirement will stay for now.  I suspect the we will be working on the design of the solution after we have settled on the operational state models. 

5) Ephemeral-Req-12, page 7:
Presumably the requirement is that the notification must indicate the node that we involved in the collision?  I.e. it isn't sufficient to just signal to the client that there has been a collision with some of their configuration? 

Sue:  For some data models, it may be important to provide more than just that.  How about: 

  When there is a collision, a notification  (which includes an indication of the data node the collision occurred on)

MUST BE sent to the original client to give the original client a chance to deal with the issues surrounding the collision.  

6) Ephemeral-REQ-14, page 7:
I would suggest potentially rewording this to make the requirement and leeway on the solution more explicit. 

Sue: Accepted – this will be released with version 2. 

Cosmetic comments:

7) Section 1. Introduction.  The requirements are no longer version specific, hence perhaps update the following text from:

 

to:

Sue: We will probably have a subsequent version of the I2RS protocol.  I would prefer to leave this as stated so this is clear.  

8) Section 1. introduction.  I'm not sure that this 3rd bullet is relevant, and possibly could be removed, although equally it doesn't seem to do any harm:

Sue: There is a protocol strawman that will be submitted based on experience.  

9) The document uses a mix of "Yang" and "YANG", probably should just use "YANG".

Fixed in version 12. 

10) Section 3. Ephemeral State Requirements:

>The document refers to "ephemeral configured state" here, but elsewhere (e.g. 3.4) "ephemeral configuration" or "ephemeral configuration state" is used.  It might be helpful for use of these terms to be consistent, I would suggest "ephemeral configuration" is >sufficient.

Sue: Ephemeral state is defined as ephemeral configuration state and operational state in the 

 <https://tools.ietf.org/html/draft-ietf-i2rs-ephemeral-state-11#section-3> 3.  Ephemeral State Requirements

   In requirements Ephemeral-REQ-01 to Ephemeral-05, Ephemeral state is
   defined as potentially including both ephemeral configured state and
   operational state.

Proposed:

Sue: Thank you.  Version -12 will have this change. 

12) Ephemeral-REQ-15, page 7:
I would suggest that it might be better to refer to "Ephemeral state" rather than the I2RS ephemeral data-store.

Proposed:

Accepted change for -15. 

Thanks,
Rob

---

From: Robert Wilton <[email-masked]>
Date: Thu, 30 Jun 2016 17:39:13 +0100

Hi Sue,

All fine with me.  A couple of comments inline ...

On 30/06/2016 17:15, Susan Hares wrote:
>
> Robert:
>
> Thank you for your comments.  See resolution of comments below. 
> Version -12 will be sent to the list to handle most of these comments.
>
> Sue
>
> *From:*i2rs [mailto:[email-masked]] *On Behalf Of *Robert Wilton
> *Sent:* Monday, June 27, 2016 11:30 AM
> *To:* [email-masked]
> *Subject:* [i2rs] Review of draft-ietf-i2rs-ephemeral-state-11
>
> Hi,
>
> is that may) also be lost on other circumstances such as process 
> restart of the I2RS agent.  Does this need to be clarified in the 
> requirement?  E.g.
>
> Sue: In our previous discussions,  other people suggest that “reboots” 
> covered hardware and software reboot of the I2RS agent.   And that the 
> specific nature of the reboot was too-much information for the 
> requirement.
>
Rob: OK.

> override mechanism should operate globally or on a per data node 
> basis.  I'm not sure whether this needs to be clarified - since the 
> text in the architecture document makes it pretty clear that a global 
> level resolution is sufficient.
>
> Sue:  Again, we were cautioned to not specify the design, but only 
> provide a result.  The global is sufficient.
>
Rob: OK

> Sue: You are Juergen are concerned about writeable/non-writeable.  
>  Martin is concerned about status/configuration.  The I2RS authors 
> believe we are running into the different operational state models. 
>  Each time we change this requirement we get another response.   The 
> requirement will stay for now.  I suspect the we will be working on 
> the design of the solution after we have settled on the operational 
> state models.
>
Rob:
My actual concern is that what I have proposed as a datastore solution 
cannot meet this requirement, because I think that only configuration 
should be writable, and hence the writable/non-writable property is 
implicit.

>   When there is a collision, a notification  (which includes an 
> indication of the data node the collision occurred on)
>
> MUST BE sent to the original client to give the original client a 
> chance to deal with the issues surrounding the collision.
>
Rob: Yes, OK.

> Sue: We will probably have a subsequent version of the I2RS protocol. 
>  I would prefer to leave this as stated so this is clear.
>
> >The document refers to "ephemeral configured state" here, but 
> elsewhere (e.g. 3.4) "ephemeral configuration" or "ephemeral 
> configuration state" is used.  It might be helpful for use of these 
> terms to be consistent, I would suggest "ephemeral configuration" is 
> >sufficient.
>
Rob: It was only a minor nit, I was suggesting consistency of the term, 
perhaps change "ephemeral configuration"and "ephemeral configuration 
state"  to "ephemeral configured state".

Thanks,
Rob

>
>     3
>     <https://tools.ietf.org/html/draft-ietf-i2rs-ephemeral-state-11#section-3>. 
>     Ephemeral State Requirements
>

---

From: "Susan Hares" <[email-masked]>
Date: Thu, 30 Jun 2016 12:54:15 -0400

Robert:

Version-12 is uploaded.   

On Ephemeral-REQ-08: 

>Sue: You are Juergen are concerned about writeable/non-writeable.   Martin is concerned about status/configuration.  The I2RS authors believe we are running into the different operational state models.  Each time we change this requirement we get another >response.   The requirement will stay for now.  I suspect the we will be working on the design of the solution after we have settled on the operational state models.

>Rob:
>My actual concern is that what I have proposed as a datastore solution cannot meet this requirement, because I think that only configuration should be writable, and hence the writable/non-writable property is implicit.

We have the case of the ephemeral topology data base that conflicts this work.   We will need to work this out at IETF 96 in Berlin. 

Sue 

From: Robert Wilton [mailto:[email-masked]] 
Sent: Thursday, June 30, 2016 12:39 PM
To: Susan Hares; [email-masked]
Subject: Re: [i2rs] Review of draft-ietf-i2rs-ephemeral-state-11

Hi Sue,

Robert: 

Sue 

Hi,

Minors comments:

Rob: OK.

2) Hierarchy: (page 5)

Proposed:

   Ephemeral-REQ-06:

Rob: OK

Rob:
My actual concern is that what I have proposed as a datastore solution cannot meet this requirement, because I think that only configuration should be writable, and hence the writable/non-writable property is implicit.

Rob: Yes, OK.

Cosmetic comments:

 

to:

Fixed in version 12. 

Rob: It was only a minor nit, I was suggesting consistency of the term, perhaps change "ephemeral configuration"and "ephemeral configuration state"  to "ephemeral configured state".

Thanks,
Rob

 

Proposed:

Proposed:

Accepted change for -15. 

Thanks,
Rob

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Thu, 30 Jun 2016 18:52:15 +0000

Hi Susan,

My responses below are based on v12.

Section 9:
Pub/Sub Requirements Expanded for Ephemeral State -- Subscribing to Ephemeral state means that there must be some stream of Ephemeral information against which the Subscription can be applied.   This will be a different stream than for startup config, and might be a different stream from running config. In addition, it is possible that there will not be a single stream with includes both Operational and Config data. (Likely these are all different from any OpState stream too.)    With this in mind:

Pub-Sub-REQ-01:
Comment #1
Must the subscription service support operational and ephemeral configuration data within a single subscription?  Or can two separate subscriptions be used?  Applying a single subscription against a single stream is where things are going in the NETCONF at this point.
Comment #2
Isn’t all data in Operational data stores ephemeral?  I see the mailing list discussions on this terminology topic for the requirements earlier in the document.  This is where you said: “Ephemeral state is defined as "ephemeral configuration state" and operational state.”  But the terminology fix doesn’t seem to have made it down here.

Pub-Sub-REQ-02:
Comment #3:
Building off Comment #2, is there such a thing as Ephemeral data in Operational data?  Or are you talking filtering so that only changes in Operational data are sent.   I believe you are intending the second.   In that case there might need to be special types of filters needed.  Filters such as a count of the number of objects under a tree (e.g., number of routes), or range based filters for an operational value.

Section 2, Bullets 5 & 9 and Section 7:
Priority Collision.  Could you explicitly say when the priority of one write will prevent a lower priority write from occurring?  Is it the completion of and response to some POST?  I don’t know what time-of-use means.

Eric

From: i2rs [mailto:[email-masked]] On Behalf Of Susan Hares
Sent: Thursday, June 30, 2016 12:54 PM
To: Robert Wilton -X (rwilton - ENSOFT LIMITED at Cisco); [email-masked]
Subject: Re: [i2rs] Review of draft-ietf-i2rs-ephemeral-state-11

Robert:

Version-12 is uploaded.

Sue

From: Robert Wilton [mailto:[email-masked]]
Sent: Thursday, June 30, 2016 12:39 PM
To: Susan Hares; [email-masked]<mailto:[email-masked]>
Subject: Re: [i2rs] Review of draft-ietf-i2rs-ephemeral-state-11

Hi Sue,

Sue

From: i2rs [mailto:[email-masked]] On Behalf Of Robert Wilton
Sent: Monday, June 27, 2016 11:30 AM
To: [email-masked]<mailto:[email-masked]>
Subject: [i2rs] Review of draft-ietf-i2rs-ephemeral-state-11

Hi,

Minors comments:

   agent.

2) Hierarchy: (page 5)

   ephemeral.

Proposed:

   Ephemeral-REQ-06:

   state.

Cosmetic comments:

to:

       YANG changes.

Thanks,
Rob

3<https://tools.ietf.org/html/draft-ietf-i2rs-ephemeral-state-11#section-3>.  Ephemeral State Requirements

   operational state.

Proposed:

   are the focus.

Accepted change for -15.

Thanks,
Rob

---

From: "Susan Hares" <[email-masked]>
Date: Thu, 30 Jun 2016 15:12:34 -0400

Eric: 

Thank you for your comments:  See comments below.

Sue 

From: i2rs [mailto:[email-masked]] On Behalf Of Eric Voit (evoit)
Sent: Thursday, June 30, 2016 2:52 PM
To: Susan Hares
Cc: [email-masked]
Subject: Re: [i2rs] Review of draft-ietf-i2rs-ephemeral-state-11

Hi Susan,

>Pub/Sub Requirements Expanded for Ephemeral State -- Subscribing to Ephemeral state means that there must be some stream of Ephemeral information against which the Subscription can be applied.   This will be a different stream than for startup config, and might be a >different stream from running config. In addition, it is possible that there will not be a single stream with includes both Operational and Config data. (Likely these are all different from any OpState stream too.)    With this in mind:

>Must the subscription service support operational and ephemeral configuration data within a single subscription?  Or can two separate subscriptions be used?  Applying a single subscription against a…

No.  It can be two separate subscriptions.  

>Isn’t all data in Operational data stores ephemeral?  I see the mailing list discussions on this terminology topic for the requirements earlier in the document.  This is where you said: “Ephemeral state is defined as "ephemeral configuration state" and operational state.”  But the >terminology fix doesn’t seem to have made it down here.

Sue: We thought this was more specific.  I can change it to “Ephemeral state”.  

Pub-Sub-REQ-02: 

Comment #3:

>Building off Comment #2, is there such a thing as Ephemeral data in Operational data?  Or are you talking filtering so that only changes in Operational data are sent.   I believe you are intending the second.   In that case there might need to be special types of filters >needed.  Filters such as a count of the number of objects under a tree (e.g., number of routes), or range based filters for an operational value.

Sue: We have ephemeral data models with operational state.  Technically, the data is both ephemeral (under an ephemeral model) and operational state.   The difficulty is that this conflicts with some data models. 

In RESTCONF,  the updating process will check the priority of the existing write.  This check will occur at least the completion of the POST for an over-write of an existing post.  Since the RESTCONF posts are not simultaneous (but rather 1 at a time), the check will occur a single write.   

In NETCONF, this will be at edit-config stage. Both of these stages were defined as “time of use” – because it was when the update was to occur. 

Do you have an alternate suggestion? 

Eric

Robert:

Version-12 is uploaded.   

Sue 

Hi Sue,

Robert: 

Sue 

Hi,

Minors comments:

Rob: OK.

2) Hierarchy: (page 5)

Proposed:

   Ephemeral-REQ-06:

Rob: OK

Rob:

Rob: Yes, OK.

Cosmetic comments:

 

to:

Fixed in version 12. 

Thanks,
Rob

 

Proposed:

Proposed:

Accepted change for -15. 

Thanks,
Rob

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Thu, 30 Jun 2016 19:38:25 +0000

From: Susan Hares, June 30, 2016 3:13 PM

Eric:

Sue

From: i2rs [mailto:[email-masked]] On Behalf Of Eric Voit (evoit)
Sent: Thursday, June 30, 2016 2:52 PM
To: Susan Hares
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [i2rs] Review of draft-ietf-i2rs-ephemeral-state-11

Sue: We thought this was more specific.  I can change it to “Ephemeral state”.
Eric: consistency with higher requirements makes sense.

Pub-Sub-REQ-02:

Sue: We have ephemeral data models with operational state.  Technically, the data is both ephemeral (under an ephemeral model) and operational state.   The difficulty is that this conflicts with some data models.
Eric: I understand now.   As an aside comment, the types of filtering needed will be varied and complex.  They will need to be specified later, and should directly correspond with the filters which we need to support via in draft-ietf-netconf-yang-push.

Eric: I think it is possible that RESTCONF and NETCONF posts can come in concurrently from alternative sources.  I think detecting this is one of the purposes of ETag in draft-ietf-netconf-restconf-13 section 3.4.1.2.   It might be worth have the requirement state that priority must be considered for both completed ephemeral posts, as well as posts in progress.

/Eric

Eric

---

From: "Susan Hares" <[email-masked]>
Date: Thu, 30 Jun 2016 15:46:47 -0400

Eric: 

For version -13, the following  changes will be made: 

   Pub-Sub-REQ-01: The Subscription Service MUST support 

   subscriptions against ephemeral state in operational data stores, configuration data stores or both.

   Pub-Sub-REQ-02: The Subscription Service MUST support filtering 

   so that subscribed updates under a target node might publish only 

   ephemeral state in operational data or configuration data, or publish both 

   ephemeral and operational data.

Ephemeral-REQ-12:

Will have the following text after it 

   Note:RESTCONF and NETCONF posts can come in concurrently from alternative sources 

   (see  ETag in <xref target="draft-ietf-netconf-restconf"></xref> section 3.4.1.2 usage).

   Therefore the collision detection and comparison of priority needs to occur both

   for both type of updates (POST or edit-config) at the point of comparison. 

Does this resolve your comments? 

Sue 

From: i2rs [mailto:[email-masked]] On Behalf Of Eric Voit (evoit)
Sent: Thursday, June 30, 2016 3:38 PM
To: Susan Hares
Cc: [email-masked]
Subject: Re: [i2rs] Review of draft-ietf-i2rs-ephemeral-state-11

Eric: 

Sue 

 

 

Pub-Sub-REQ-02: 

/Eric

Eric

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Thu, 30 Jun 2016 19:55:08 +0000

Works for  me

Eric:

Ephemeral-REQ-12:

Sue

Sue

Pub-Sub-REQ-02:

/Eric

Eric