Subject: [Dime] Question regarding RFC 6733 on transport end disconnections

From: Miguel Rodriguez Caudevilla <[email-masked]>
Date: Thu, 18 Oct 2018 15:56:52 +0000

Hello,

We are working in a product which uses an implementation of a diameter stack based on RFC 6733.
We recently had a discussions with the developers of the stack and after some thoughts and post analysis of the discussion, we personally think this could be due to an open interpretation of the RFC regarding DPR messages and how a stack implementation should deal with transport end disconnections. Let me try to explain the reason of the discussion.

We have a cluster wide distributed stack. The diameter instances in the cluster can be administratively be locked, which in consequence ends a transport end connection with a peer.

From https://tools.ietf.org/html/rfc6733#section-5.4 we interpret that the DPR/DPA flow before transport end disconnection is as follows (Simplified flow):

When a diameter instance disconnects one of its transport connections it must be notified to the peer with a DPR.
A DPA will be sent by the peer and the diameter instance initiates the transport disconnect.
The peer will receive the transport closure request and close its transport end.

What we argue is that what the peer does after this flow depends completely in the Disconnection-Reason AVP received in the DPR.

In our case, when the lock is performed we expected that the diameter stack would send a DPR with Disconnect-Cause REBOOTING, to inform the peer that the transport end connection is going to be closed and that the peer can immediately retry to connect as there are other diameter instances available to provide service in the same cluster.

The stack implementers have a different argument. The administrative lock of a single diameter instance, they consider that it is performed in a way that all the services on cluster level are available. So, it would be quite strange to indicate the peer disconnect-case REBOOTING for the loss of a single node considering we can still provide diameter service through all the other nodes in the cluster. So, the loss of a node (and connections handled by it) is a transient fault that is easily handled if the peer reconnects. They consider that this way the peer reconnects as quickly as possible in such cases. That the stack can achieve this according to the standard as well if it does not send any DPR with connection loss reasons.

They point to https://tools.ietf.org/html/rfc6733#section-2.1

A given Diameter instance of the peer state machine MUST NOT use more than one transport connection to communicate with a given peer, unless multiple instances exist on the peer, in which, case a separate connection per process is allowed.

And consider that a diameter disconnect cause (eg: REBOOTING or whatever else) sent by a diameter node to a peer actually reflects the overall state of the sending node. Since we have a cluster wide distributed implementation it would be not correct to send a disconnect case REBOOTING as that indicates the whole node is going to be rebooted. We just perform some node level operations while still providing full service to diameter peers.

We would like to have your point of view regarding this discussion.
Is this intentionally left open for the stack implementation?
Did we interpret something wrong?

Thanks in advance.

Best Regards,
Miguel Rodriguez Caudevilla

---

From: Ivan Skytte Jørgensen <[email-masked]>
Date: Mon, 22 Oct 2018 16:01:17 +0200

> We have a cluster wide distributed stack. The diameter instances in the cluster can be administratively be locked, which in consequence ends a transport end connection with a peer.
[...]
> Did we interpret something wrong?

The RFCs can be confusing/convoluted in this area. But re-read section 1.2:

"Diameter Node
      A Diameter node is a host process that implements the Diameter
      protocol and acts as either a client, an agent, or a server."

and combine with section 4.3.1:

"If multiple Diameter nodes run on
the same host, each Diameter node MUST be assigned a unique
DiameterIdentity."

combine with section 5.6.1:

"Upon receipt of a CER, the
   identity of the connecting peer can be uniquely determined from the
   Origin-Host."

Don't be fooled by the "host" part of "origin-host". The payload ia a diameter identity. The AVP was named before anyone thought of these problems.

Then you can deduce that a diameter node is an entity (that being a process, thread, task, fiber, machine, vm, ...) that:
 - has an identity, which is unique and used as the host identity (which is actually a node identity)
 - has a state machine

and therefore:
 - has at most one transport connection per peer.

For simple setups those distinctions aren't that important but when you have a cluster of cooperating nodes with shared state they are.

If you have such a cluster of nodes then you cannot have them all have the same host-identity because identities must be unique and if you somehow manage to get two transport connections to a peer using the same host-identity then the peer must close one of them (as per election process).

What you can do is that each node (vm/process/thread/whatever) has a unique identity:
  * node1.example.com
  * node2.example.com
  * node3.example.com
  * node4.example.com
  ...
and the messages (Eg. CCR/EAP/UD... but not CER/DPR) use a common identity in origin-host.
  * system.example.com

So if node1 sends a eg. CCR to a peer it will have:
  * origin-host: system.example.com
  * route-record: node1.example.com

The peer will simply treat node1 as an intermediate proxy. Nothing special. And your problem with node lock / reboots becomes trivial - just send a DPR with whatever-reason and the peer will only treat it as a message from node1 and only concerning node1. 

The only drawback I can think of is that there might be some non-compliant peers that don't support proxies. But back in 2013/2014 when I tested such cluster setups I didn't encounter any peers that were non-compliant in this area.

Regards,
  Ivan