Subject: Re: [dnssd] Genart last call review of draft-ietf-dnssd-push-20

From: Tom Pusateri <[email-masked]>
Date: Tue, 2 Jul 2019 14:36:16 -0400

> On Jun 28, 2019, at 4:03 PM, Robert Sparks via Datatracker <[email-masked]> wrote:
> 
> Reviewer: Robert Sparks
> Review result: Ready with Issues
> 
> I am the assigned Gen-ART reviewer for this draft. The General Area
> Review Team (Gen-ART) reviews all IETF documents being processed
> by the IESG for the IETF Chair.  Please treat these comments just
> like any other last call comments.
> 
> For more information, please see the FAQ at
> 
> <https://trac.ietf.org/trac/gen/wiki/GenArtfaq>.
> 
> Document: draft-ietf-dnssd-push-20
> Reviewer: Robert Sparks
> Review Date: 2019-06-28
> IETF LC End Date: 2019-07-05
> IESG Telechat date: Not scheduled for a telechat
> 
> Summary: Ready for publication as a Proposed Standard but with an Issue to
> consider before publication,
> 
> Issue:
> 
> The discussion of recursive resolvers in section 6.1 may need additional
> consideration. In particular, the recommendation to pass a received error code
> along to a client has, I think, some unintended consequences for the client. If
> the recursive server receives a NOTIMP, for example, passing that to the client
> tells the client the wrong thing about the server it is connected to. Perhaps
> it would be better for the recursive server to return SERVFAIL in this
> circumstance? (Similar to what it would do if it couldn't connect to the next
> server as described at the bottom of page 10).

Let me think about this some more before I respond.

> 
> Nits:
> 
> Page 5, Section 3, 3rd paragraph, last sentence: NOT REQUIRED is not a
> 2119/8174 keyword. I suggest using lowercase 'not required' in this sentence.

Fixed.

> 
> Page 7, Section 4, 3rd paragraph: The first sentence alludes to concerns about
> anonymous subscriptions, saying TCP alleviates those concerns. As written this
> is pretty vague. Can you expand on what you mean by an anonymous subscription
> in this context?

Now says:

"Connection setup over TCP ensures return reachability and alleviates concerns of state overload at the server which is a potential problem with connection-less protocols using spoofed source addresses. All subscribers are guaranteed to be reachable by the server by virtue of the TCP three-way handshake.”

> 
> Page 10, Section 6.1, first sentence: Suggest s/first step in DNS Push/first
> step in a DNS Push/
> 

Fixed.

> Page 15, last paragraph: Why MUST the server immediately terminate a connection
> in this situation? Just accepting the request seems safe - having subscription
> requests show up for the same name seems nearly idempotent, and only one PUSH
> would result from having multiple such subscriptions. Is this close an attempt
> to avoid resource denial attacks buy some node subscribing many times to the
> same thing? That feels extreme, especially since tearing down the connection
> would cancel other subscriptions the client already has established on that
> connection.

The discussion about case insensitivity here is a side note. The main point is that if you receive a subscription for something you already have a subscription for, the two sides are out of sync. There is no protocol mechanism built in to regain synchronization and the best way is then to close the connection and try again. The most common reason that the two ends are out of sync is likely software bugs and if this keeps occurring, the administrator will begin to see a pattern of connections closing and can report the problem.

> 
> Page 16, second paragraph: I suggest replacing the second sentence with
> something like "A name in a SUBSCRIBE message that matches only a literal CNAME
> in the zone will only receive notifications of changes to the CNAME (assuming
> the subscription asks for that type), and nothing else."
> 

The SUBSCRIBE contains the record type not just a name. The point is that a subscription of a CNAME is only to the CNAME record and not to the record it points to. How about:

"Aliasing is not supported. That is, a CNAME in a SUBSCRIBE message matches only a literal CNAME record in the zone, and not to any records referenced by the owner name."

> Page 23, top of page: Since section 4 restricts this protocol to TLS over TCP,
> the "(or equivalent for other protocols)" phrase should be removed.

Good catch. I removed all instances of "(or equivalent for other protocols)”

Thanks!

Tom

---

From: David Schinazi <[email-masked]>
Date: Tue, 2 Jul 2019 12:36:09 -0700

Hi Tom,

If the protocol is restricted to TLS over TCP, it should send a TLS
close_notify, not a TCP RST.
TLS close_notify is cryptographically guaranteed to originate from the peer,
whereas TCP RST can be injected by an on-path entity to cause truncation
attacks.

Thanks,
David

---

From: Tom Pusateri <[email-masked]>
Date: Tue, 2 Jul 2019 15:42:12 -0400

Thanks David.

The only other transport I could see being relevant in the future is QUIC. But there are lots of unknowns about DNS over QUIC at present so I think we can safely assume that would require a new PUSH draft to specify the behavior.

So I think your suggestion is a good one.

Tom

> On Jul 2, 2019, at 3:36 PM, David Schinazi <[email-masked]> wrote:
> 
> Hi Tom,
> 
> If the protocol is restricted to TLS over TCP, it should send a TLS close_notify, not a TCP RST.
> whereas TCP RST can be injected by an on-path entity to cause truncation attacks.
> 
> Thanks,
> David
> 
>

---

From: Robert Sparks <[email-masked]>
Date: Tue, 2 Jul 2019 14:55:42 -0500

inline

> "Connection setup over TCP ensures return reachability and alleviates concerns of state overload at the server which is a potential problem with connection-less protocols using spoofed source addre…
wfm
> The discussion about case insensitivity here is a side note. The main point is that if you receive a subscription for something you already have a subscription for, the two sides are out of sync. T…
At a high level, we're talking about what to do when something violates 
protocol. I'm not going to argue for a change, but I find issuing an 
error when asked to make things like X when they are already like X 
suspect. The idea that you can treat this as a symptom of being out of 
sync and that a reset would fix it would be more compelling if you 
weren't basing this on TLS over TCP - I can only see it being a problem 
that must recur and you're choosing to have it be noisy and thrashy to 
expose the problem. I guess that's ok, though I worry about the intended 
deployment environments not really having an administrator to notice 
there's a problem.
> "Aliasing is not supported. That is, a CNAME in a SUBSCRIBE message matches only a literal CNAME record in the zone, and not to any records referenced by the owner name."
Sure

---

From: Tom Pusateri <[email-masked]>
Date: Tue, 2 Jul 2019 17:24:10 -0400

> server as described at the bottom of page 10).

The only time NOTIMP is returned is in the case where DSO stateful operations (over which DNS PUSH runs) is not supported. This error code is due to the fact that DSO uses a different Opcode than the standard Query/Response Opcode. DSO defines a new opcode. The effect of this is that NOTIMP is the correct response if a server doesn’t support the new Opcode.

Similarly, if DSO is supported but DNS Push TLV type is not supported, we return a new RCODE for DSO type not implemented DSOTYPENI.

These are both required by existing specifications.

If a client begins the connection with a DSO Keepalive to a resolver and the resolver accepts the connection, subsequent SUBSCRIBE operations that return NOTIMP will be obvious that the authoritative server does not support DSO.

However, if the client begins with a SUBSCRIBE and receives NOTIMP from the resolver, it’s not clear if the NOTIMP was originated by the resolver or the authoritative server.

Right now, we allow either DSO TLV to setup the state. In the case that the client begins with a SUBSCRIBE and the resolver returns NOTIMP, the client should attempt a connection with the authoritative server directly as described in 6.1. If the authoritative server returns NOTIMP, then PUSH notifications are not possible.

While we could be more explicit about every error case, I don’t think the document says anything wrong here.

Thanks,
Tom

---

From: Robert Sparks <[email-masked]>
Date: Wed, 3 Jul 2019 09:45:32 -0500

> However, if the client begins with a SUBSCRIBE and receives NOTIMP from the resolver, it’s not clear if the NOTIMP was originated by the resolver or the authoritative server.

And I think that's a problem.

What does a NOTIMP mean to the client? Most of the draft says "The 
server doesn't implement DSO". It doesn't say "doesn't implement DSO for 
this particular set of bits in this query". Section 6.2.2 says the 
client should assume a retry delay of 1 hour before talking to the 
server (the resolver) again.

Now, other parts of the document imply "for this particular set of bits" 
- in the overview, near the bottom of page 5, it says to use NOTIMP 
(actually it says NOTIMPL, maybe those are different things and I'm 
confused?) if a message is received for a class other than "IN" and the 
server has only implemented push for "IN". Again, that "assume a retry 
delay" kicks in.

> Right now, we allow either DSO TLV to setup the state. In the case that the client begins with a SUBSCRIBE and the resolver returns NOTIMP, the client should attempt a connection with the authorita…
But there's fate sharing, at least as I'm reading the document now. That 
resolver could have been authoritative (or could reach an authoritative 
server) for some other subscription. But you've told the client that 
it's not ok to talk to this resolver for any subscription (for an hour 
at least).
> While we could be more explicit about every error case, I don’t think the document says anything wrong here.
And I used NOTIMP as an example. I think there may be fate sharing 
issues with other response codes as well.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 3 Jul 2019 11:12:11 -0400

On Jul 3, 2019, at 10:45 AM, Robert Sparks <[email-masked]> wrote:
> What does a NOTIMP mean to the client? Most of the draft says "The server doesn't implement DSO". It doesn't say "doesn't implement DSO for this particular set of bits in this query". Section 6.2.2 says the client should assume a retry delay of 1 hour before talking to the server (the resolver) again.
> 
> Now, other parts of the document imply "for this particular set of bits" - in the overview, near the bottom of page 5, it says to use NOTIMP (actually it says NOTIMPL, maybe those are different things and I'm confused?) if a message is received for a class other than "IN" and the server has only implemented push for "IN". Again, that "assume a retry delay" kicks in.

Robert, first, thanks for doing a really thorough review of this document.  This is much appreciated.   This particular insight is one that I think is really important.   I think your initial take on this is correct: if a resolver receives NOTIMP or DSONI from upstream, what this means is “fail,” not “not implemented,” from the perspective of the resolver.   The resolver knows what the client is asking, and can’t do it.  That’s SERVFAIL, not NOTIMP or DSONI.

I think your subsequent point about terminating connections is also correct: we do not want a billion broken clients hammering on servers.  However, the DSO document actually specifies how to deal with this: the server can tell the client to shut up for a period of time, and this is explicitly recommended for situations like this.   The advantage of failing in cases like this is that it causes the implementation to not work, which then motivates the implementor to fix it.

And thanks for the advice about how to terminate TLS connections—I had missed that nuance.  Are TLS implementations actually able to do this (to reject RST packets)?

---

From: Tom Pusateri <[email-masked]>
Date: Thu, 4 Jul 2019 12:52:25 -0400

> On Jul 3, 2019, at 11:12 AM, Ted Lemon <[email-masked]> wrote:
> 
> On Jul 3, 2019, at 10:45 AM, Robert Sparks <[email-masked] <mailto:[email-masked]>> wrote:

a. If the client sends SUBSCRIBE or KEEPALIVE to the resolver and the resolver doesn’t support DSO, it will send NOTIMP whether you want it to or not because it doesn’t know about the opcode.

b. If the client sends SUBSCRIBE and gets back DSONI from the resolver because it supports DSO but not PUSH Notifications, the resolver is following the DSO spec.

c. If the client sends KEEPALIVE and successfully establishes a DSO connection to the resolver and then sends SUBSCRIBE and the resolver gets back NOTIMP from the authoritative, and returns this to the client, then the client knows exactly that the problem is that the authoritative doesn’t support DSO.

To solve the ambiguity:

1. The client should only send SUBSCRIBE to set up the DSO session if it has gone through the discovery process and wants to talk to the authoritative directly. If it wants to try to subscribe to push notifications through a resolver, then it should use KEEPALIVE first to establish the DSO connection with the resolver.

2. If the resolver supports SUBSCRIBE but the authoritative doesn’t, the resolver should not send DSONI back to the client because the client can’t tell the difference between the resolver not supporting SUBSCRIBE and the authoritative not supporting SUBSCRIBE. In this case, the resolver should return SERVFAIL. This should inform the client that the authoritative doesn’t support SUBSCRIBE. If there are multiple authoritative servers supporting _dns-push._tcp, the resolver may want to try all of them before returning SERVFAIL.

> And thanks for the advice about how to terminate TLS connections—I had missed that nuance.  Are TLS implementations actually able to do this (to reject RST packets)?

This was actually a comment from David Schinazi (and a good one). I’ve adjusted the working copy on github but there’s still one section I’m wrestling with regarding TCP RST.

Thanks,
Tom

---

From: Ted Lemon <[email-masked]>
Date: Thu, 4 Jul 2019 14:13:16 -0400

> 1. The client should only send SUBSCRIBE to set up the DSO session if it has gone through the discovery process and wants to talk to the authoritative directly. If it wants to try to subscribe to p…

This creates a new semantic connection between KEEPALIVE and SUBSCRIBE.  This means that DSO implementations now have to track more state.  I don’t think there’s a benefit to doing this.  Why do you think it’s necessary?

> 2. If the resolver supports SUBSCRIBE but the authoritative doesn’t, the resolver should not send DSONI back to the client because the client can’t tell the difference between the resolver not supp…

There’s no need to mention DSONI here.  Just say what the right behavior is.  If you mention DSONI here, someone might read this to suggest that in some case DSONI would make sense.

SERVFAIL just means that the server is unable to support the requested behavior. It doesn’t signal why. Unless there’s something the client would do differently, I don’t think it’s necessary for the client to know why the request failed.

---

From: Tom Pusateri <[email-masked]>
Date: Thu, 4 Jul 2019 15:32:52 -0400

> On Jul 4, 2019, at 2:13 PM, Ted Lemon <[email-masked]> wrote:
> 
> On Jul 4, 2019, at 12:52 PM, Tom Pusateri <[email-masked] <mailto:[email-masked]>> wrote:

Do I already have a session or not with my resolver is not a lot of extra state. Especially when it’s over TLS and you are likely to keep the session up for some time and then let it expire. This state is necessary regardless.

I was trying to give the client enough information to determine WHY it failed. If we determine this isn’t important, we can just let the client try to figure it out but it will be more work for the client.

> SERVFAIL just means that the server is unable to support the requested behavior. It doesn’t signal why. Unless there’s something the client would do differently, I don’t think it’s necessary for th…

It’s likely that resolvers aren’t going to support this before authoritative servers are and clients will quickly learn their resolver isn’t capable and go directly to the authoritative for some period of time.

Tom

---

From: Ted Lemon <[email-masked]>
Date: Thu, 4 Jul 2019 15:54:04 -0400

> I was trying to give the client enough information to determine WHY it failed. If we determine this isn’t important, we can just let the client try to figure it out but it will be more work for the…

The client is a computer program, not a person, so there is no chance that it will be able to figure out what went wrong! :)

Seriously, though, what’s the strategy the client should follow in this case?  I think we generally say “try again in an hour” but I’m not sure if we said that explicitly here or just in the DSO document.

> It’s likely that resolvers aren’t going to support this before authoritative servers are and clients will quickly learn their resolver isn’t capable and go directly to the authoritative for some pe…

I think that how resolver support for this will work is an open question right now, which will probably have to be addressed in a follow-on document.  At present, the implementation I’ve done doesn’t even attempt the local resolver, because I couldn’t figure out how to implement that.  I’m assuming that in most cases there’s no particular benefit to using the local resolver, because the auth server will also be local.

---

From: Tom Pusateri <[email-masked]>
Date: Thu, 4 Jul 2019 16:09:09 -0400

Maybe we could have a conference call and hash this out. Anyone is welcome. 

I’m ok with removing resolver support for now to simplify this and tackle resolvers in another document. But if there’s a good solution we’re all happy with, we can fix it now. 

Tom

> On Jul 4, 2019, at 3:54 PM, Ted Lemon <[email-masked]> wrote:
> 
> _______________________________________________
> dnssd mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnssd

---

From: Ted Lemon <[email-masked]>
Date: Thu, 4 Jul 2019 17:30:39 -0400

> I’m ok with removing resolver support for now to simplify this and tackle resolvers in another document. But if there’s a good solution we’re all happy with, we can fix it now. 

I think we’ve specified the parts of resolver support that we understand, and it would be a shame to take those out.   I don’t think we need to make this any more complicated than that—we’d talked about the resolver support issue during WGLC, and were satisfied.   Simply addressing the point Robert raised should be sufficient.

---

From: David Schinazi <[email-masked]>
Date: Fri, 5 Jul 2019 17:45:11 -0700

On Thu, Jul 4, 2019 at 9:52 AM Tom Pusateri <[email-masked]> wrote:

> And thanks for the advice about how to terminate TLS connections—I had
> missed that nuance.  Are TLS implementations actually able to do this (to
> reject RST packets)?
>
>
> This was actually a comment from David Schinazi (and a good one). I’ve
> adjusted the working copy on github but there’s still one section I’m
> wrestling with regarding TCP RST.
>

On most operating systems today, TCP is in the kernel and TLS is in
user-space, so most implementations of TLS over TCP do not have the ability
to discard TCP RSTs that were injected by attackers. However, the TLS
close_notify alert allows endpoints to be able to tell the difference
between a connection that was closed gracefully by the peer and one that
was forcefully terminated (possibly by an attacker). So upon receipt of a
TCP RST without prior TLS close_notify, the connection will die, but the
application will know that the latest message could have been truncated
halfway through transmission and should therefore be discarded.

David

---

From: "Christopher Wood" <[email-masked]>
Date: Sat, 06 Jul 2019 06:50:36 -0700

On Fri, Jul 5, 2019, at 5:45 PM, David Schinazi wrote:
> user-space, so most implementations of TLS over TCP do not have the 
> ability to discard TCP RSTs that were injected by attackers. However, 
> the TLS close_notify alert allows endpoints to be able to tell the 
> difference between a connection that was closed gracefully by the peer 
> and one that was forcefully terminated (possibly by an attacker). So 
> upon receipt of a TCP RST without prior TLS close_notify, the 
> connection will die, but the application will know that the latest 
> message could have been truncated halfway through transmission and 
> should therefore be discarded.

+1 (though some applications might choose to use the partial message)

---

From: Stuart Cheshire <[email-masked]>
Date: Mon, 08 Jul 2019 15:53:32 -0700

> whereas TCP RST can be injected by an on-path entity to cause truncation attacks.

In TCP we use FIN for a graceful close, and RST for an abortive close. The former is normal operation; the latter means your code has a bug you need to fix.

Is there an appropriate equivalent in TLS? It would be good to differentiate normal operation from a fatal protocol error that causes a forcible termination.

I see in the TLS 1.3 spec, RFC 8446, Section 6.2. “Error Alerts” says:

   Whenever an implementation encounters a fatal error condition, it
   SHOULD send an appropriate fatal alert and MUST close the connection
   without sending or receiving any additional data.

<https://tools.ietf.org/html/rfc8446#section-6.2>

Are any of these error alerts appropriate to perform this abortive disconnect, like perhaps the decode_error code?

   decode_error:  A message could not be decoded because some field was
      out of the specified range or the length of the message was
      incorrect.  This alert is used for errors where the message does
      not conform to the formal protocol syntax.  This alert should
      never be observed in communication between proper implementations,
      except when messages were corrupted in the network.

Or are these TLS error alerts reserved for TLS-layer error conditions?

Stuart Cheshire

---

From: David Schinazi <[email-masked]>
Date: Mon, 8 Jul 2019 16:05:18 -0700

In general the "TLS Alerts" error codes are specific to the operation of
TLS itself, not the application running over TLS.

If you want to send a graceful close, the tool of choice is close_notify.
If you detect an unrecoverable error and want to abort the connection, I
see two options:
(1) forcibly terminate the connection at the DNS layer by sending a DNS
error message followed by a TLS close_notify
(2) forcibly terminate the connection at the TCP layer by sending a RST

As a client sending, I don't see much value in (1) since all the server can
do in either case is free the resources associated with this connection.
As a server sending, I suspect (1) is best unless you were unable to parse
anything in which case (2) makes sense.

David

On Mon, Jul 8, 2019 at 3:53 PM Stuart Cheshire <[email-masked]> wrote:

> In TCP we use FIN for a graceful close, and RST for an abortive close. The
> former is normal operation; the latter means your code has a bug you need
> to fix.
>
> Is there an appropriate equivalent in TLS? It would be good to
> differentiate normal operation from a fatal protocol error that causes a
> forcible termination.
>
> Are any of these error alerts appropriate to perform this abortive
> disconnect, like perhaps the decode_error code?
>

---

From: Tom Pusateri <[email-masked]>
Date: Mon, 8 Jul 2019 19:12:24 -0400

If we detect a PUSH protocol error and don’t want the client to immediately reconnect and repeat the same error, we can send a retry delay TLV with a long delay and then close gracefully with a TLS close_notify in lieu of a TCP RST. 

Tom

> On Jul 8, 2019, at 7:05 PM, David Schinazi <[email-masked]> wrote:
> 
> In general the "TLS Alerts" error codes are specific to the operation of TLS itself, not the application running over TLS.
> 
> If you detect an unrecoverable error and want to abort the connection, I see two options:
> (1) forcibly terminate the connection at the DNS layer by sending a DNS error message followed by a TLS close_notify
> As a client sending, I don't see much value in (1) since all the server can do in either case is free the resources associated with this connection.
> As a server sending, I suspect (1) is best unless you were unable to parse anything in which case (2) makes sense.
> 
> David
>

---

From: Stuart Cheshire <[email-masked]>
Date: Tue, 09 Jul 2019 19:22:50 -0700

> As a server sending, I suspect (1) is best unless you were unable to parse anything in which case (2) makes sense.

This is a great candidate for some serious discussion in Montréal.

The draft *used* to say to respond to fatal errors by forcibly aborting the connection with a TCP RST. This is consistent with RFC 8490, DNS Stateful Operations, the underlying technology used by draft-ietf-dnssd-push.

I believe it was actually you who suggested using TLS close_notify:

> From: David Schinazi <[email-masked]>
> Subject: Re: [dnssd] Genart last call review of draft-ietf-dnssd-push-20
> Date: 2 July 2019 at 12:36:09 PDT
> To: Tom Pusateri <[email-masked]>
> Cc: Robert Sparks <[email-masked]>, [email-masked], DNSSD <[email-masked]>
> Resent-From: [email-masked]
> Resent-To: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], [email-masked], Tim Wicinski <[email-masked]>, [email-masked]
> 
> Hi Tom,
> 

I suspect we have a miscommunication going on here.

> the "(or equivalent for other protocols)" phrase should be removed.

This is a fine observation.

You then suggested changing TCP RST to TLS close_notify, not realizing (a) this is only for fatal errors, and (b) the precedent already set by RFC 8490.

We have in fact updated the document, but I think this was too hasty, and we should revert it back to the way it was before.

If not, we at least need to have a thorough DNSSD Working Group discussion about this before making a last-minute change to the protocol.

Stuart Cheshire

---

From: David Schinazi <[email-masked]>
Date: Wed, 10 Jul 2019 10:32:53 -0700

Thanks Stuart, that makes sense to me - I hadn't loaded the entire context
back into memory... Apologies.

Basically a "graceful" close should always use a TLS close_notify, but any
catastrophic failure can use TCP RST.

David

On Tue, Jul 9, 2019 at 7:22 PM Stuart Cheshire <[email-masked]> wrote:

> > As a client sending, I don't see much value in (1) since all the server
> can do in either case is free the resources associated with this connection.
> > As a server sending, I suspect (1) is best unless you were unable to
> parse anything in which case (2) makes sense.
>
> The draft *used* to say to respond to fatal errors by forcibly aborting
> the connection with a TCP RST. This is consistent with RFC 8490, DNS
> Stateful Operations, the underlying technology used by
> draft-ietf-dnssd-push.
>
> > Cc: Robert Sparks <[email-masked]>,
> [email-masked], DNSSD <[email-masked]>
> > Resent-To: [email-masked], [email-masked],
> [email-masked], [email-masked], [email-masked],
> [email-masked], Tim Wicinski <[email-masked]>, [email-masked]
> >
> > Hi Tom,
> >
> You then suggested changing TCP RST to TLS close_notify, not realizing (a)
> this is only for fatal errors, and (b) the precedent already set by RFC
> 8490.
>
> We have in fact updated the document, but I think this was too hasty, and
> we should revert it back to the way it was before.
>
> If not, we at least need to have a thorough DNSSD Working Group discussion
> about this before making a last-minute change to the protocol.
>
> Stuart Cheshire
>
>

---

From: Tom Pusateri <[email-masked]>
Date: Wed, 10 Jul 2019 15:56:44 -0400

I pointed out earlier that now that we have a RETRY DELAY TLV in DNS Stateful Operations, we can do better than just a TCP RST by not only signaling that a fatal error occurred but also, inform the client not to try reconnecting for a long time.

This means we would always close with a TLS close_notify and if a fatal error occurs, we set the RETRY DELAY. This would keep the broken client from connecting right away again and then following the same error path.

So we should discuss this more before restoring the old behavior.

Tom

> On Jul 10, 2019, at 1:32 PM, David Schinazi <[email-masked]> wrote:
> 
> Thanks Stuart, that makes sense to me - I hadn't loaded the entire context back into memory... Apologies.
> 
> Basically a "graceful" close should always use a TLS close_notify, but any catastrophic failure can use TCP RST.
> 
> David
>

---

From: Ted Lemon <[email-masked]>
Date: Thu, 11 Jul 2019 13:19:05 -0400

> If not, we at least need to have a thorough DNSSD Working Group discussion about this before making a last-minute change to the protocol.

To add some further nuance from a discussion that Stuart and I had today on this, there are actually several different cases where connection closes are done, and how they should be done is something we should talk about.

I think in all cases where the client is closing the connection, there’s a case to be made that we don’t want to use close_notify.   It’s true that an attacker can kill our DNS Push connection in this case by forging an RST to the server.   We should discuss whether this is a serious concern that we need to take into account.   If it is, then using close_notify would protect against this iff the server ignores TCP RSTs for active TLS sessions. 

But the main argument for using close_notify in this case is that we want to be able to resume.   This will not be the case if the client closed the connection because of a protocol error.   It will be the case when the client is closing the connection due to inactivity.

There is a case where the server closes the connection when the client sends a duplicate subscribe.   That’s because this is a protocol error: the client is broken, and cannot be expected to take corrective action.   Then the question is, do we close the connection down with a retry-delay to make the client go away, or do we just send an RST?

Argument in favor of sending retry-delay:
if the client implements it, it will shut up for a while.

Arguments against:
If the client doesn’t implement it, it won’t shut up, so we haven’t gained anything
Making things “sort of work” when the client is broken isn’t all that helpful—we actually want the behavior in this case to be dysfunctional, so that it is noticed and fixed.

I think that the working group should consider these issues and come to a consensus.

My own personal opinion is that we should always do close_notify, because if we can assume this, then an attacker can’t kill the connection by sending an RST, if that behavior is implemented in the TLS/TCP stack.   My one doubt about this is that if we are going through a NAT, will the NAT drop its mapping when it sees the RST?   If so, then close_notify doesn’t protect against this attack for a majority of current users.   It still might be worth doing for IPv6, of course.

As to whether we should use retry-delay, I have really mixed feelings about this.   I want implementations to be visibly broken when they are broken, but I don’t want to have to operate a server that has to deal with broken clients.   The question is whether forcibly disconnecting will actually cause implementors to take action, or whether it will not be noticed and contribute to dysfunction.

My personal experience is that breaking badly is actually conducive to improvement, so that’s the direction I’m leaning at the moment.

---

From: Tom Pusateri <[email-masked]>
Date: Thu, 11 Jul 2019 14:20:32 -0400

If a client implements PUSH, it implements DSO which means it implements KEEPALIVE and RETRY DELAY.

That doesn’t mean it will honor every part and it might retry before the delay expires.

But the server sent the retry delay and knows the timeout value and so the server can filter this client for that period of time regardless of whether the client honors it or not. In fact, a server SHOULD do the filtering because the RETRY DELAY is really saying, I’m not going to listen to you until after this timeout.

Also, even if the client closes because of an error, that doesn’t preclude it from using TLS session resumption for the next subscription.

So I’m in favor of always using close_notify and sending a RETRY DELAY for critical errors when needed.

But I think it would be helpful to outline the actual errors that could occur on either end and verify this works in every case. Sending as much information to the other side as possible is helpful for determining bugs. TCP RST signaling doesn’t convey much information.

Tom

> On Jul 11, 2019, at 1:19 PM, Ted Lemon <[email-masked]> wrote:
>

---

From: David Schinazi <[email-masked]>
Date: Thu, 11 Jul 2019 14:44:20 -0700

To reiterate my earlier point, TLS close_notify DOES NOT protect against an
attacker closing the connection by sending a FIN or RST.
The benefit of close_notify is to protect against truncation attacks: if an
attacker sends a FIN or RST in the middle of a stream,
the recipient application should ignore the current message instead of
considering it complete.
This is important for protocols such as HTTP/0.9 that rely on the TCP FIN
instead of using prefixed lengths.
Given that DSO messages contain a prefixed length, making sure
implementations do not act on partial messages is sufficient to prevent
truncation attacks.
If my understanding is correct, DNS-PUSH does not get ANY security benefits
from close_notify.

When a critical error occurs, a TCP RST should cause the other side to tear
down state more aggressively than a close_notify, which is what we want.

I'd recommend using close_notify for gracefully closes, and TCP RST for any
critical failures.

I agree with Ted that if an implementation is broken it's best to have it
fail very visibly.

David

On Thu, Jul 11, 2019 at 11:20 AM Tom Pusateri <[email-masked]> wrote:

> But the server sent the retry delay and knows the timeout value and so the
> server can filter this client for that period of time regardless of whether
> the client honors it or not. In fact, a server SHOULD do the filtering
> because the RETRY DELAY is really saying, I’m not going to listen to you
> until after this timeout.
>
> Also, even if the client closes because of an error, that doesn’t preclude
> it from using TLS session resumption for the next subscription.
>
> But I think it would be helpful to outline the actual errors that could
> occur on either end and verify this works in every case. Sending as much
> information to the other side as possible is helpful for determining bugs.
> TCP RST signaling doesn’t convey much information.
>
> Tom
>
> On Jul 9, 2019, at 10:22 PM, Stuart Cheshire <
> cheshire=[email-masked]> wrote:
>
> To add some further nuance from a discussion that Stuart and I had today
> on this, there are actually several different cases where connection closes
> are done, and how they should be done is something we should talk about.
>
> I think in all cases where the client is closing the connection, there’s a
> case to be made that we *don’t* want to use close_notify.   It’s true
> that an attacker can kill our DNS Push connection in this case by forging
> an RST to the server.   We should discuss whether this is a serious concern
> that we need to take into account.   If it is, then using close_notify
> would protect against this iff the server ignores TCP RSTs for active TLS
> sessions.
>
> But the main argument for using close_notify in this case is that we want
> to be able to resume.   This will not be the case if the client closed the
> connection because of a protocol error.   It will be the case when the
> client is closing the connection due to inactivity.
>
> There is a case where the server closes the connection when the client
> sends a duplicate subscribe.   That’s because this is a protocol error: the
> client is broken, and cannot be expected to take corrective action.   Then
> the question is, do we close the connection down with a retry-delay to make
> the client go away, or do we just send an RST?
>
>    - if the client implements it, it will shut up for a while.
>
>
> Arguments against:
>
>    - If the client doesn’t implement it, it won’t shut up, so we haven’t
>    gained anything
>    - Making things “sort of work” when the client is broken isn’t all
>    that helpful—we actually want the behavior in this case to be
>    dysfunctional, so that it is noticed and fixed.
>
>
> My own personal opinion is that we should always do close_notify, because
> if we can assume this, then an attacker can’t kill the connection by
> sending an RST, if that behavior is implemented in the TLS/TCP stack.   My
> one doubt about this is that if we are going through a NAT, will the NAT
> drop its mapping when it sees the RST?   If so, then close_notify doesn’t
> protect against this attack for a majority of current users.   It still
> might be worth doing for IPv6, of course.
>
> As to whether we should use retry-delay, I have really mixed feelings
> about this.   I want implementations to be visibly broken when they are
> broken, but I don’t want to have to operate a server that has to deal with
> broken clients.   The question is whether forcibly disconnecting will
> actually cause implementors to take action, or whether it will not be
> noticed and contribute to dysfunction.
>
> My personal experience is that breaking badly is actually conducive to
> improvement, so that’s the direction I’m leaning at the moment.
>

---

From: Tom Pusateri <[email-masked]>
Date: Thu, 11 Jul 2019 17:46:42 -0400

> On Jul 11, 2019, at 2:20 PM, Tom Pusateri <[email-masked]> wrote:
> 

Here are all the places a TCP RST was suggested in -20. These are all likely BUGS.

1. CLIENT receives SUBSCRIBE from server
2. SERVER receives duplicate SUBSCRIBE
3. CLIENT receives PUSH with no change notifications
4. CLIENT receives PUSH notification with ‘collective remove’ TTL and non-zero RDLEN
5. CLIENT receives PUSH notification with DNS message length larger than 16k
6. CLIENT receives UNSUBSCRIBE from SERVER

In cases 1, 3, 4, 5, and 6, the CLIENT will likely mark the SERVER as buggy, close the connection and not try to reconnect for a period of time or use another SERVER if available. A RETRY DELAY isn’t applicable and the CLIENT can close the connection either with a TCP RST or a TLS close_notify. The consensus seems to be to send a TCP RST in this case.

In case 2, the SERVER should send a RETRY DELAY TLV and then send a TLS close_notify or it could wait for the CLIENT to close_notify or RST but the SERVER shouldn’t RST because it wants the CLIENT to get the RETRY DELAY.

There are also cases where the server sends a RETRY DELAY TLV because of a SUBSCRIBE error and the CLIENT is supposed to close the connection. However, if the CLIENT does not close the connection, the SERVER may choose to close it in which case it should close with TLS close_notify to make sure the RETRY DELAY is received.

I think it would be better for the CLIENT to try to send a TLS close_notify after receiving a RETRY DELAY so it can resume later especially in the cases when a SUBSCRIBE fails.

Tom

---

From: Ted Lemon <[email-masked]>
Date: Thu, 11 Jul 2019 18:58:16 -0400

> I think it would be better for the CLIENT to try to send a TLS close_notify after receiving a RETRY DELAY so it can resume later especially in the cases when a SUBSCRIBE fails.

Given David’s points, I don’t think there’s any reason to send a RETRY DELAY in case 2.   It should be handled the same as the other “protocol error” cases.  Can you think of a reason why case 2 is different, other than that it’s the one case where it’s possible to send a RETRY DELAY?  Bear in mind that it is a protocol error for the client to send a duplicate subscribe: it means that the client’s internal state is not being tracked correctly.   So it’s not a recoverable error.  We don’t want the client to reconnect.

Additionally, as you’ve pointed out, although the server could send the client a RETRY DELAY, if the client isn’t following the spec in one context there’s no particular reason to expect it to follow the spec in another.   Requiring the server to have special heuristics to follow in the case of broken clients seems to be creating an unreasonable burden.

---

From: Ted Lemon <[email-masked]>
Date: Thu, 11 Jul 2019 18:58:42 -0400

On Jul 11, 2019, at 5:44 PM, David Schinazi <[email-masked]> wrote:
> I'd recommend using close_notify for gracefully closes, and TCP RST for any critical failures.
> I agree with Ted that if an implementation is broken it's best to have it fail very visibly.

SGTM.  Thanks for clarifying!

---

From: Tom Pusateri <[email-masked]>
Date: Thu, 11 Jul 2019 19:03:17 -0400

> On Jul 11, 2019, at 6:58 PM, Ted Lemon <[email-masked]> wrote:
> 
> On Jul 11, 2019, at 5:46 PM, Tom Pusateri <[email-masked] <mailto:[email-masked]>> wrote:

The SERVER may want to conserve resources from broken clients. The RETRY DELAY is for the SERVER’s protection and it’s thoughtfully informing the CLIENT how long it has to wait before the SERVER will accept a connection from it. It doesn’t matter if the CLIENT agrees or not or whether the CLIENT follows the spec or not.

Tom

---

From: Ted Lemon <[email-masked]>
Date: Thu, 11 Jul 2019 19:31:49 -0400

> The SERVER may want to conserve resources from broken clients. The RETRY DELAY is for the SERVER’s protection and it’s thoughtfully informing the CLIENT how long it has to wait before the SERVER wi…

Okay, so we have the following scenarios:

1. Server sends RETRY-DELAY, client honors it.   Result: client goes away for a long time and does polling instead of push.  Behavior on client is not great, but probably won’t be noticed by a typical end-user other than as an inexplicable inconvenience.   Vendor will have no strong motivation to fix.
2. Server sends RETRY-DELAY, client ignores it.  Result: whatever broken behavior the client exhibits, up to and including back-to-back reconnects.  UX may degrade substantially: in particular, this will affect battery life for mobile devices.  Practically speaking, in this case the vendor will be motivated to fix.
3. Server sends RST.   Result: same as 2.

So the question is, do we want outcome (1) or outcome (2)?

---

From: Tom Pusateri <[email-masked]>
Date: Thu, 11 Jul 2019 19:39:21 -0400

> On Jul 11, 2019, at 7:31 PM, Ted Lemon <[email-masked]> wrote:
> 
> On Jul 11, 2019, at 7:03 PM, Tom Pusateri <[email-masked] <mailto:[email-masked]>> wrote:

There is no back to back reconnect in the #2 case. The server shouldn’t accept the connection after sending a RETRY DELAY to this client. The client may ignore the RETRY DELAY but this doesn’t matter. It’s only a courtesy that the SERVER tells the client how long it has to wait.

So #1 and #2 are the same.

The server gets abused if it does #3. It should always send the RETRY DELAY to protect itself.

Tom

---

From: Stuart Cheshire <[email-masked]>
Date: Thu, 11 Jul 2019 16:39:54 -0700

> 6. CLIENT receives UNSUBSCRIBE from SERVER

Reviewing the list above, I realize that we state explicitly that SUBSCRIBE and UNSUBSCRIBE sent from server are both invalid. But we don’t enumerate the other bogus message directions.

I have added some clarifying text around this, and will submit an updated draft once they open for submissions again. Not that it was every really unclear, but it doesn’t hurt to be abundantly unambiguous. I have added:

A server MUST NOT send a RECONFIRM message.
A client MUST NOT send a SUBSCRIBE response.
A client MUST NOT send a PUSH message.

All three are fatal errors of the “this should never ever happen” variety.

Stuart Cheshire

---

From: Ted Lemon <[email-masked]>
Date: Thu, 11 Jul 2019 20:32:26 -0400

> There is no back to back reconnect in the #2 case. The server shouldn’t accept the connection after sending a RETRY DELAY to this client. The client may ignore the RETRY DELAY but this doesn’t matt…

I don’t know how the server refuses connections on a per-client basis.   We certainly don’t describe how to do that in the DSO RFC.  Practically speaking, unless the server is maintaining a list of all the clients that connect to it, it has no idea that client X that it just dropped is client Y that just connected.   Even if it does maintain such a list, it would be unusual for that to allow it to actually drop the connection immediately.   And supposing it does, that might just make the client reconnect a bit faster.   That is, if the SYN gets an RST in response, the client will just get -1 from connect() and call connect() again unless it has some kind of backoff strategy.

---

From: David Schinazi <[email-masked]>
Date: Thu, 11 Jul 2019 17:32:21 -0700

Hi Tom,

You wrote:

> There is no back to back reconnect in the #2 case. The server shouldn’t
> accept the connection after sending a RETRY DELAY to this client. The
> client may ignore the RETRY DELAY but this doesn’t matter. It’s only a
> courtesy that the SERVER tells the client how long it has to wait.

When you say that the server shouldn't accept the second connection, how
does the server know that the new connection comes from the same client?
Unless you're using TLS client certificates, you have no way of
differentiating clients because of NAT. Or am I missing something?

Thanks,
David

On Thu, Jul 11, 2019 at 4:40 PM Stuart Cheshire <[email-masked]> wrote:

> Reviewing the list above, I realize that we state explicitly that
> SUBSCRIBE and UNSUBSCRIBE sent from server are both invalid. But we don’t
> enumerate the other bogus message directions.
>
> I have added some clarifying text around this, and will submit an updated
> draft once they open for submissions again. Not that it was every really
> unclear, but it doesn’t hurt to be abundantly unambiguous. I have added:
>

---

From: Tom Pusateri <[email-masked]>
Date: Thu, 11 Jul 2019 20:56:25 -0400

> On Jul 11, 2019, at 8:32 PM, Ted Lemon <[email-masked]> wrote:
> 
> On Jul 11, 2019, at 7:39 PM, Tom Pusateri <[email-masked] <mailto:[email-masked]>> wrote:

What is the point of the server sending a RETRY DELAY TLV if it isn’t going to enforce it?

The purpose of the RETRY DELAY TLV isn’t to say please be nice and wait.

It’s to say, I’ve put this filter in place that I am enforcing for this period of time. You might as well wait until you try to connect again because it won’t do any good to try before the timer expires.

So, yes, you have to keep state when you send a RETRY DELAY. When the timer fires, you remove the filter and allow the connection to proceed. You have state for the current connections and you have state for the RETRY DELAY timers you have running. SYN attacks happen and those are dealt with like any other SYN attack.

As for NAT, you’re going to block everyone coming from the same address. You should use IPv6 if you’re remote and don’t want to get blocked because of a bad actor.

If you don’t filter, you might was well not send a RETRY DELAY because the clients you’re trying to throttle are the ones that are going to ignore it.

Tom

---

From: Eric Rescorla <[email-masked]>
Date: Thu, 11 Jul 2019 21:42:42 -0700

On Thu, Jul 11, 2019 at 2:44 PM David Schinazi <[email-masked]>
wrote:

> To reiterate my earlier point, TLS close_notify DOES NOT protect against
> an attacker closing the connection by sending a FIN or RST.
>

Just to clarify: it allows the receiver to distinguish between closes which
were initiated by the peer and those which were not

> If my understanding is correct, DNS-PUSH does not get ANY security
> benefits from close_notify.
>

Well, maybe. You could, for instance, log an error if a finished was
received without a close_notify.

-Ekr

> When a critical error occurs, a TCP RST should cause the other side to
> tear down state more aggressively than a close_notify, which is what we
> want.
>

>> But the server sent the retry delay and knows the timeout value and so
>> the server can filter this client for that period of time regardless of
>> whether the client honors it or not. In fact, a server SHOULD do the
>> filtering because the RETRY DELAY is really saying, I’m not going to listen
>> to you until after this timeout.
>>
>> Also, even if the client closes because of an error, that doesn’t
>> preclude it from using TLS session resumption for the next subscription.
>>
>> You then suggested changing TCP RST to TLS close_notify, not realizing
>> (a) this is only for fatal errors, and (b) the precedent already set by RFC
>> 8490.
>>
>> If not, we at least need to have a thorough DNSSD Working Group
>> discussion about this before making a last-minute change to the protocol.
>>
>>
>> I think in all cases where the client is closing the connection, there’s
>> a case to be made that we *don’t* want to use close_notify.   It’s true

---

From: "Jan Komissar (jkomissa)" <[email-masked]>
Date: Fri, 12 Jul 2019 16:16:10 +0000

Hi Stuart,

You may also want to make sure section 6.6 (DNS Stateful Operations TLV Context Summary) is up to date.

Jan.

On 7/11/19, 7:41 PM, "dnssd on behalf of Stuart Cheshire" <[email-masked] on behalf of cheshire=[email-masked]> wrote:

    

    
    
    Stuart Cheshire

---

From: Michael Richardson <[email-masked]>
Date: Sun, 14 Jul 2019 17:05:26 -0400

Ted Lemon <[email-masked]> wrote:
    > forging an RST to the server. We should discuss whether this is a
    > serious concern that we need to take into account. If it is, then using
    > close_notify would protect against this iff the server ignores TCP RSTs
    > for active TLS sessions.

I'm unaware of TCP stacks that can turn off RSTs on a per-socket basis.
Is this commonly available now and I'm just ignorant of it?

I imagine that is the only way the application could inform the kernel that
TLS is active.

I guess a filter could drop RSTs to port 853, and assume it's always TLS.

Securing TCP against fake RSTs is something the TCP-AO was designed to do.
AFAIK, we never defined a way to key TCP-AO from TLS.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Ted Lemon <[email-masked]>
Date: Sun, 14 Jul 2019 17:08:41 -0400

I don’t know if a stack that does this either. It could be done in principle. But it sounds like that wasn’t the goal. 

Sent from my iPhone

> On Jul 14, 2019, at 5:05 PM, Michael Richardson <[email-masked]> wrote:
> 
>

---

From: Eric Rescorla <[email-masked]>
Date: Sun, 14 Jul 2019 14:27:54 -0700

> I guess a filter could drop RSTs to port 853, and assume it's always TLS.

I'm not sure how much this would help: if the attacker can inject an invalid
data segment full of random data, then this will likely lead to TLS
connection
failure (i.e., you'll be able to detect the problem but it's not
recoverable).
I'm not sufficiently up to date on the state of TCP injection attacks to
know
whether off-path RST attacks are significantly harder than off-path data
injection
attacks (obviously, on-path attacks are straightforward in both cases).

> AFAIK, we never defined a way to key TCP-AO from TLS.

No, we did not. It's theoretically straightforward, but you know what they
say
about the difference between theory and practice. In this particular case,
given the low level of deployment, it seems like it might be easier to just
move to QUIC.

-Ekr