Subject: Re: [sipcore] sip/tcp with primary & secondary outbound proxies (behavior upon receipt of tcp rst)

From: [email-masked] (Dale R. Worley)
Date: Mon, 25 Jan 2016 15:23:12 -0500

"Penar, Russell" <[email-masked]> writes:
> I'm seeing non-ideal behavior from cpe and hoping this group can
> provide guidelines specific to the scenario below, and ultimately
> standardize ideal cpe behavior for the given context.  I haven't found
> any specifications defining the ideal behavior, and it was suggested
> during sipnoc to contact sipcore as an appropriate next step.

Unfortunately, Sipcore doesn't so much define ideal behavior as define
the behaviors that a UA (CPE) is permitted to have.  Within that
context, a UA can drop a call whenever it chooses.  In the use cases we
see, the general advice is "If the UA is behaving in a way that provides
a poor user experience, buy a different type of UA."  That usually gives
the UA manufacturers incentive to improve their UAs.

> Where a standing call is setup against the primary sbc; after a
> primary sbc mated pair failover, calls are at risk of being dropped if
> a tcp connection is not established back to the primary sbc.  This can
> be due to session audits from the platform, or more specific to my ask
> is a cpe re-Invite scenario.  For example, when cpe puts standing call
> on hold (after sbc failover), the sbc target for re-establishing the
> signaling tcp session is critical.
>
> I've seen cpe exhibit ideal behavior, where upon receipt of tcp rst
> the cpe connects back to the primary sbc.  In that ideal scenario, the
> call stays up, mid-call control doesn't fail, etc.  And I've seen
> non-ideal behavior where once the cpe attempts to put the call on
> hold, it reacts to the tcp rst by failing over to the secondary sbc
> and dropping the standing call.  Fwiw, I'm told the vendor exhibiting
> the ideal behavior only recently began to do so (reportedly from
> customer pressure to properly support geo-redundant setups as
> described above).

As a general rule, the failure of a TCP connection does not change the
state of any SIP call.  But if the UA wants to issue a re-INVITE for an
existing call, it will send an INVITE request using the "route set" of
the call.  The UA learns the route set during call set-up; the devices
on the path that the INVITE request takes will add Record-Route or
Contact headers containing addresses; the collection of these headers is
the route set and specifies the path which all further requests for that
call will take.

Since the SBC wants to see further requests for the call, it adds an
address for itself in either a Record-Route header or a Contact header
(depending on the technical details of the SBC).  If you want the calls
to fail over, it's critical that the address is one that routes to both
the primary and secondary SBCs -- it is a host name which DNS maps to
the addresses of both SBCs.

If the address is only for the SBC involved in the call setup, if that
SBC goes down and the UA attempts a re-INVITE, the UA can't send the
re-INVITE.  The UA may decide to tear down the call, but some UAs
don't -- instead, they assume that if the call truly fails, the user
will hang up and so they maintain the call until the user hangs up.

It's understood that if a UA attempts to send a request on a TCP
connection and the TCP connection produces some sort of error that the
UA should either attempt to reestablish the TCP connection or send the
request to an alternative destination (e.g., the other SBC).  This is
specified in RFC 3263 section 4.3:

   For SIP requests, failure occurs if the transaction layer reports a
   503 error response or a transport failure of some sort (generally,
   due to fatal ICMP errors in UDP or connection failures in TCP).
   Failure also occurs if the transaction layer times out without ever
   having received any response, provisional or final (i.e., timer B or
   timer F in RFC 3261 [1] fires).  If a failure occurs, the client
   SHOULD create a new request, which is identical to the previous, but
   has a different value of the Via branch ID than the previous (and
   therefore constitutes a new SIP transaction).  That request is sent
   to the next element in the list as specified by RFC 2782.

Dale

---

From: "Penar, Russell" <[email-masked]>
Date: Mon, 25 Jan 2016 23:23:51 +0000

Thanks for the reply Dale, comments in line.
Regards,
Russ

-----Original Message-----
From: Dale R. Worley [mailto:[email-masked]]
Sent: Monday, January 25, 2016 1:23 PM
To: Penar, Russell
Cc: [email-masked]
Subject: Re: [sipcore] sip/tcp with primary & secondary outbound proxies (behavior upon receipt of tcp rst)

> during sipnoc to contact sipcore as an appropriate next step.

Unfortunately, Sipcore doesn't so much define ideal behavior as define the behaviors that a UA (CPE) is permitted to have.  Within that context, a UA can drop a call whenever it chooses.  In the use cases we see, the general advice is "If the UA is behaving in a way that provides a poor user experience, buy a different type of UA."  That usually gives the UA manufacturers incentive to improve their UAs.

RP - That's a fair stance, we have seen at least one vendor move to the "ideal" behavior presumably from customer pressure.  Would it also be fair to state Sipcore defines behavior that a UA is not permitted to have?  Perhaps what I'm really trying to get here is a MUST NOT statement against connecting to secondary targets upon receipt of a TCP RST.

> customer pressure to properly support geo-redundant setups as

As a general rule, the failure of a TCP connection does not change the state of any SIP call.  But if the UA wants to issue a re-INVITE for an existing call, it will send an INVITE request using the "route set" of the call.  The UA learns the route set during call set-up; the devices on the path that the INVITE request takes will add Record-Route or Contact headers containing addresses; the collection of these headers is the route set and specifies the path which all further requests for that call will take.

Since the SBC wants to see further requests for the call, it adds an address for itself in either a Record-Route header or a Contact header (depending on the technical details of the SBC).  If you want the calls to fail over, it's critical that the address is one that routes to both the primary and secondary SBCs -- it is a host name which DNS maps to the addresses of both SBCs.

RP - We're using DNS SRV to support geo-redundant access SBCs.  The "route set" doesn't appear to be an issue, as the re-Invite does go to the correct location (primary), but upon receipt of the TCP RST the UA immediately drops the call, sets up a new TCP connection to the secondary and Registers.  The UA gathers primary and secondary targets (for outbound proxy FQDN) upon boot up and then accordingly based on TTL.

If the address is only for the SBC involved in the call setup, if that SBC goes down and the UA attempts a re-INVITE, the UA can't send the re-INVITE.  The UA may decide to tear down the call, but some UAs don't -- instead, they assume that if the call truly fails, the user will hang up and so they maintain the call until the user hangs up.

RP - I agree in principle when using SRV the SBC's Contact/R-R should have the SBC FQDN and not a specific IP address.  With use of outbound proxy, I haven't seen issues with UA's not failing over when Contact (for example) is returned with only the SBC (HA pair IP address) involved in the call setup.

It's understood that if a UA attempts to send a request on a TCP connection and the TCP connection produces some sort of error that the UA should either attempt to reestablish the TCP connection or send the request to an alternative destination (e.g., the other SBC).  This is specified in RFC 3263 section 4.3:

RP - Could we amend this section of 3263, to reflect a UA SHOULD (specific to context of TCP) try the current element where a SIP dialog is established before trying the next element per 2782?

Dale

This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.

---

From: Paul Kyzivat <[email-masked]>
Date: Tue, 26 Jan 2016 11:30:52 -0500

Russ,

Are you aware of RFC5626? It provides an explicit mechanism for 
redundant registration paths.

	Thanks,
	Paul

On 1/25/16 6:23 PM, Penar, Russell wrote:
> _______________________________________________
> sipcore mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sipcore
>

---

From: [email-masked] (Dale R. Worley)
Date: Tue, 26 Jan 2016 21:11:01 -0500

"Penar, Russell" <[email-masked]> writes:
> RP - That's a fair stance, we have seen at least one vendor move to
> the "ideal" behavior presumably from customer pressure.  Would it also
> be fair to state Sipcore defines behavior that a UA is not permitted
> to have?

It's true that Sipcore defines behavior that a UA is not permitted to
have, because that is the logical negation of the behavior that it is
required to have.

> Perhaps what I'm really trying to get here is a MUST NOT statement
> against connecting to secondary targets upon receipt of a TCP RST.

I don't think that there is any intention to make such a requirement.
In principle, a UA has to contact targets in the order specified by RFC
3263, e.g., the SRV records for the URI, but in practice -- given
transient network failures -- there can be no guarantee.

It's not clear to me what that matters to you.  You wrote:

> Most, if not all, vendor sbc mated pairs maintain signaling/media state,
> but not tcp state (reportedly doing so is significantly hard/expensive).

If the SBCs share call (signaling/media) state, then the re-INVITE
doesn't have to go to the same SBC that the original INVITE did.

On the other hand, if updates to a call must be made at the same SBC
that the call was established (that is, if they don't share signaling
state), then an SBC should Record-Route itself with a URI which routes
to itself only.  And in that case, if the UA's TCP connection to the SBC
fails, it will be forced to reestablish a TCP connection to that SBC,
because the URI in the route set only routes to that address.

> RP - We're using DNS SRV to support geo-redundant access SBCs.  The
> "route set" doesn't appear to be an issue, as the re-Invite does go to
> the correct location (primary), but upon receipt of the TCP RST the UA
> immediately drops the call, sets up a new TCP connection to the
> secondary and Registers.  The UA gathers primary and secondary targets
> (for outbound proxy FQDN) upon boot up and then accordingly based on
> TTL.

That's a strange UA; it seems to consider the secondary SBC to be a
different destination from the primary SBC, rather than being two
destination addresses for one destination URI.

> RP - Could we amend this section of 3263, to reflect a UA SHOULD
> (specific to context of TCP) try the current element where a SIP
> dialog is established before trying the next element per 2782?

I don't think there would be much support for that, at least not without
considerably more justification.

What I don't seem to be understanding is how the two SBCs are presented
to the UA.  You write as if they are redudant deeply in SIP, so that
they share dialog (signaling) state.  In that case, there is some URI
which maps to *both* of them, and the UA is told to register at that one
URI.  Of course, if the UA needs to update a call, it can send the
re-INVITE to either SBC, because both SBCs can manage the state of the
call.

But your failure mode sounds like the UA is told of the two SBCs
separately, and doesn't treat them as alternative addresses for the
"same" destination.  Of course, in that case, the UA doesn't know that
it can send re-INVITEs to the secondary SBC to update calls that were
placed through the primary SBC.

Dale

---

From: "Penar, Russell" <[email-masked]>
Date: Wed, 27 Jan 2016 20:52:32 +0000

Thank you Dale,
Apologies for any confusion, I worked to clear it up in line below.
Regards,
Russ

-----Original Message-----
From: Dale R. Worley [mailto:[email-masked]]
Sent: Tuesday, January 26, 2016 7:11 PM
To: Penar, Russell
Cc: [email-masked]
Subject: Re: [sipcore] sip/tcp with primary & secondary outbound proxies (behavior upon receipt of tcp rst)

> be fair to state Sipcore defines behavior that a UA is not permitted

It's true that Sipcore defines behavior that a UA is not permitted to have, because that is the logical negation of the behavior that it is required to have.

I don't think that there is any intention to make such a requirement.
In principle, a UA has to contact targets in the order specified by RFC 3263, e.g., the SRV records for the URI, but in practice -- given transient network failures -- there can be no guarantee.

RP - It's a very specific case, where the single primary target in SRV consists of 2 SBCs in a Highly Available active/hot-standby configuration.  We want to steer UA makers away from taking a simple TCP RST as a reason to move to the next target.  In doing so, they will force active calls to be dropped.  I see value in trying a TCP connection with the primary before moving to the next and dropping calls.

It's not clear to me what that matters to you.  You wrote:

> Most, if not all, vendor sbc mated pairs maintain signaling/media
> state, but not tcp state (reportedly doing so is significantly hard/expensive).

If the SBCs share call (signaling/media) state, then the re-INVITE doesn't have to go to the same SBC that the original INVITE did.

On the other hand, if updates to a call must be made at the same SBC that the call was established (that is, if they don't share signaling state), then an SBC should Record-Route itself with a URI which routes to itself only.  And in that case, if the UA's TCP connection to the SBC fails, it will be forced to reestablish a TCP connection to that SBC, because the URI in the route set only routes to that address.

> (for outbound proxy FQDN) upon boot up and then accordingly based on

That's a strange UA; it seems to consider the secondary SBC to be a different destination from the primary SBC, rather than being two destination addresses for one destination URI.
RP - In the context of how it gathers the targets it is a very typical UA, market leader type.

> dialog is established before trying the next element per 2782?

I don't think there would be much support for that, at least not without considerably more justification.

RP - Yeah, I think I wasn't clear enough in my problem statement (rely too much on pictures in other forums).  This is a very common deployment model for medium to large VoIP providers, at least with respect to how the SBCs are deployed.

What I don't seem to be understanding is how the two SBCs are presented to the UA.  You write as if they are redudant deeply in SIP, so that they share dialog (signaling) state.  In that case, there is some URI which maps to *both* of them, and the UA is told to register at that one URI.  Of course, if the UA needs to update a call, it can send the re-INVITE to either SBC, because both SBCs can manage the state of the call.

RP - Each target in DNS (e.g. SBC1- weighted primary target and SBC2- weighted secondary target) consists of a mated pair of SBCs.  The mated pair is deeply redundant in SIP, but not in TCP.  The lack of TCP redundancy is what leads us to the TCP RST scenario.  One of the mated pair has faulted or been taken down for maintenance; RTP is still flowing over UDP via the now active second mate in the pair, and given a new TCP connection the SIP channel would be healthy as well.  So the UA is causing an unnecessary call drop by considering the RST a complete primary target failure and moving on.

But your failure mode sounds like the UA is told of the two SBCs separately, and doesn't treat them as alternative addresses for the "same" destination.  Of course, in that case, the UA doesn't know that it can send re-INVITEs to the secondary SBC to update calls that were placed through the primary SBC.

RP - The destination is a single FQDN - e.g. sbc.carrier.com.  DNS resolves that to two weighted targets, essentially a primary and failover scheme.  The UA is correct in assuming once it connects to the second target all bets are off with any existing dialogs.

Dale

---

From: [email-masked] (Dale R. Worley)
Date: Wed, 27 Jan 2016 20:42:21 -0500

"Penar, Russell" <[email-masked]> writes:
> RP - It's a very specific case, where the single primary target in SRV
> consists of 2 SBCs in a Highly Available active/hot-standby
> configuration.  We want to steer UA makers away from taking a simple
> TCP RST as a reason to move to the next target.  In doing so, they
> will force active calls to be dropped.  I see value in trying a TCP
> connection with the primary before moving to the next and dropping
> calls.

If I understand you correctly, the UA is configured with a single FQDN,
which maps through SRV records to two SBCs.  The SBCs share signaling
and media state.

Initially, the UA creates a call by contacting one SBC -- from the SIP
point of view, it establishes a TCP connection to one of the addresses
associated with the FQDN for the proxy to which it is to send calls.

In the failure case, the first SBC goes down.  Since the SBCs share
media state, the media continues to flow and the call state of the
dialog is maintained at the UA, at the SBCs collectively, and at the
far-end UA.

Now, the UA attempts a re-INVITE.  It sends the request to the first
address for the Record-Route URI, which is the common FQDN of the two
SBCs.  Since the UA has a TCP connection to this address, it uses this
TCP connection.  This TCP send fails immediately with an RST.

At this point, the UA should attempt to resend the request, and it
should attempt to contact both addresses for the SBCs -- a new TCP
connection to the first SBC tried as the TCP connection may have been
lost due to a transient error, and the second SBC should be contacted as
it is the other address for the FQDN.  Since the SBCs share
signaling/media state, the dialog can be updates via either of them, so
it doesn't matter which SBC the UA sends the re-INVITE to.

It seems that what happens is that the UA drops the call and then tries
to contact the second SBC.

I see that there are two possible situations, which require different
solutions:

1. If the Record-Route is the FQDN for both SBCs, the UA must not
conclude that the dialog is broken until it has attempted to contact
every address for the Record-Route FQDN and has failed to reach all of
the addresses.

2. If the Record-Route is a FQDN or address for the primary SBC, then
the UA is behaving correctly, since the dialog can only be updated
through that SBC.  In that case, the fix is to change the URI that is
put in the Record-Route by the SBC.

Dale

---

From: "Penar, Russell" <[email-masked]>
Date: Thu, 28 Jan 2016 03:52:52 +0000

That is very close to 100% correct.  The minor but significant delta is DNS target 1 is two SBCs acting as one, in a primary/hot-standby configuration. DNS target 2, is a separate mated-pair of SBCs. So as the UA sees things (and in reality), the two SBCs returned in DNS do not share state for signaling and/or media.

I'm in complete agreement with your quote just below, as I believe it starts to get at the heart of the issue I'm raising.

"Now, the UA attempts a re-INVITE.  It sends the request to the first address for the Record-Route URI, which is the common FQDN of the two SBCs.  Since the UA has a TCP connection to this address, it uses this TCP connection.  This TCP send fails immediately with an RST.

At this point, the UA should attempt to resend the request, ... a new TCP connection to the first SBC tried as the TCP connection may have been lost due to a transient error.." // This last sentence is what I'm looking to enforce, or at least suggest officially.  If existing RFCs already do so, I'd like to review with you and perhaps request to make the language clearer for UA manufacturers and consumer (for reference to said vendors).

Regards,
Russ

-----Original Message-----
From: Dale R. Worley [mailto:[email-masked]]
Sent: Wednesday, January 27, 2016 6:42 PM
To: Penar, Russell
Cc: [email-masked]
Subject: Re: [sipcore] sip/tcp with primary & secondary outbound proxies (behavior upon receipt of tcp rst)

> connection with the primary before moving to the next and dropping

If I understand you correctly, the UA is configured with a single FQDN, which maps through SRV records to two SBCs.  The SBCs share signaling and media state.

Initially, the UA creates a call by contacting one SBC -- from the SIP point of view, it establishes a TCP connection to one of the addresses associated with the FQDN for the proxy to which it is to send calls.

In the failure case, the first SBC goes down.  Since the SBCs share media state, the media continues to flow and the call state of the dialog is maintained at the UA, at the SBCs collectively, and at the far-end UA.

Now, the UA attempts a re-INVITE.  It sends the request to the first address for the Record-Route URI, which is the common FQDN of the two SBCs.  Since the UA has a TCP connection to this address, it uses this TCP connection.  This TCP send fails immediately with an RST.

At this point, the UA should attempt to resend the request, and it should attempt to contact both addresses for the SBCs -- a new TCP connection to the first SBC tried as the TCP connection may have been lost due to a transient error, and the second SBC should be contacted as it is the other address for the FQDN.  Since the SBCs share signaling/media state, the dialog can be updates via either of them, so it doesn't matter which SBC the UA sends the re-INVITE to.

It seems that what happens is that the UA drops the call and then tries to contact the second SBC.

1. If the Record-Route is the FQDN for both SBCs, the UA must not conclude that the dialog is broken until it has attempted to contact every address for the Record-Route FQDN and has failed to reach all of the addresses.

2. If the Record-Route is a FQDN or address for the primary SBC, then the UA is behaving correctly, since the dialog can only be updated through that SBC.  In that case, the fix is to change the URI that is put in the Record-Route by the SBC.

Dale

---

From: [email-masked] (Dale R. Worley)
Date: Thu, 28 Jan 2016 20:44:53 -0500

"Penar, Russell" <Russell.Penar at centurylink.com> writes:
> That is very close to 100% correct.  The minor but significant delta
> is DNS target 1 is two SBCs acting as one, in a primary/hot-standby
> configuration. DNS target 2, is a separate mated-pair of SBCs. So as
> the UA sees things (and in reality), the two SBCs returned in DNS do
> not share state for signaling and/or media.  [trimmed]

OK, now I understand the configuration:  There are two *pairs* of SBCs,
each pair of which shares state, but the two pairs do not.

I think the situation is covered by four rules which are
well-established:

1.  If a device attempts to send a request to a TCP address, and the
device has a previously-existing TCP connection to the address, and the
attempt to send the request on that connection fails, the device must
not assume that the target is unreachable, because the TCP connection
might have failed due to a transient problem in the past that is now
resolved.  The device may only conclude that the address is unreachable
(at this time) if the attempt to create and use a new TCP connection
fails.

2.  A device may not conclude that a URI is unreachable unless attempts
to contact all of its addresses fail.

3.  A device may not conclude that the signaling path of an existing
call has failed unless the Record-Route/Contact URI is unreachable.

4.  An intermediate device should add to an INVITE a Record-Route URI
that routes to itself and all other intermediate devices that can be
used to update the state of the call.

Rule 4 is not a standard, but is known to be the best practice.  In your
case, when a call goes through an SBC, the Record-Route URI should be
one that routes to both SBCs in that pair, but not to the SBCs in the
other pair.

Rule 3 is not a standard, and a UA may tear down a call any time it
desires to.  But clearly, unless the signaling path becomes unusable,
the UA shouldn't tear down the call.  And in some usages, it might be a
better user experience to keep the call up even if the signaling path
fails -- the user may prefer to have his on-hold request fail and just
keep on talking.

Rule 2 is what RFC 3263 specifies.

Rule 1 is the core of your complaint, that the UA concludes that a TCP
target is unreachable just because an old TCP connection to it isn't
usable.  I don't know if rule 1 is specified anywhere, but this sort of
thing is a known problem in any situation when using a persistent
connection, and the proper recovery from it (attempting to reestablish
the connection) is well-known as the way to recover from it.

Of course, the UA my decide to attempt first to contact the other
address for the Record-Route URI, that is, the second SBC of the pair,
on the grounds that the first address seems to be less reliable.  (The
UA has latitude in what order to contact the addresses for the URI.)  If
that is successful, by rules 2 and 3 it should never tear down the call,
since the UA has successfully contacted the Record-Route URI.

Dale

---

From: [email-masked] (Dale R. Worley)
Date: Fri, 29 Jan 2016 11:06:47 -0500

There seems to be another aspect of this:  Why does the UA send a request
along a TCP connection that should have been torn down already?

When the SBC shuts down, the best practice would be for it to close (TCP
FIN) all TCP connections it has open.  In worse practice, it would
force-close (TCP RST) all TCP connections it had open.

In either case, the FIN or RST reaches the UA, which causes its TCP
stack to mark the TCP connection closed.

When the UA next wants to send a request to the SBC, it should notice
that the TCP connection is already torn down and not attempt to send the
request on it.

Dale

---

From: "Penar, Russell" <[email-masked]>
Date: Fri, 29 Jan 2016 22:36:47 +0000

Agreed in principle.  In practice, the SBC (multiple vendors) dies quietly and only when the mate is contacted is the RST sent to the UA.

While we agree on these informal best practices (including retrying primary target upon receipt of RST), it seems vendors aren't getting the memo and ideally these practices/behaviors would be included in an RFC for easy reference and less back and forth with vendors who chose wrong up front.

Regards,
Russ

-----Original Message-----
From: Dale R. Worley [mailto:[email-masked]]
Sent: Friday, January 29, 2016 9:07 AM
To: Penar, Russell
Cc: [email-masked]
Subject: Re: [sipcore] sip/tcp with primary & secondary outbound proxies (behavior upon receipt of tcp rst)

There seems to be another aspect of this:  Why does the UA send a request along a TCP connection that should have been torn down already?

When the SBC shuts down, the best practice would be for it to close (TCP
FIN) all TCP connections it has open.  In worse practice, it would force-close (TCP RST) all TCP connections it had open.

In either case, the FIN or RST reaches the UA, which causes its TCP stack to mark the TCP connection closed.

When the UA next wants to send a request to the SBC, it should notice that the TCP connection is already torn down and not attempt to send the request on it.

Dale

---

From: [email-masked] (Dale R. Worley)
Date: Sun, 31 Jan 2016 22:11:14 -0500

"Penar, Russell" <[email-masked]> writes:
> While we agree on these informal best practices (including retrying
> primary target upon receipt of RST), it seems vendors aren't getting
> the memo and ideally these practices/behaviors would be included in an
> RFC for easy reference and less back and forth with vendors who chose
> wrong up front.

You could write an RFC covering what you want to see:
http://www.ietf.org/edu/documents/81IETF-New-Work.pdf

Dale