Subject: [TLS] draft-green-tls-static-dh-in-tls13-01

From: Steve Fenter <[email-masked]>
Date: Tue, 11 Jul 2017 13:39:08 -0500

Proxies in the Data Center

There are a number of reasons that inline proxies are not a scalable solution for monitoring communications in enterprise environments.

--  cost
--  production risk
--  latency

Here are some specific examples of where the use of proxies for monitoring communications is not viable:

Network Security Monitoring

Network security monitoring is not just monitoring traffic that results from communications with customers and partners.  All it takes is for one user to click on a phishing email and there is malware inside the enterprise.  Once this happens, TLS becomes the enemy, because 30% of malware is TLS encrypted, and any TLS features intended to thwart payload inspection work against the enterprise.

Malware does not always phone home out to the Internet on day 1 of infection.  Sometimes it does reconnaissance and lateral movement for a long time.  The longer this goes on undetected, the worse the situation becomes for the enterprise.

Microsegmentation is an attempt to contain this movement, but if the malware is TLS encrypted it will go right through the firewalls.  The answer to this problem is ubiquitous plain-text traffic inspection.  The scale at which this is needed can't be accomplished by inline solutions (proxies).  Traffic inspection is needed at the Internet, the Extranet, the mainframe, the WAN, DNS servers, email servers, wireless controllers, and a host of other locations.  Traffic inspection is also needed in the virtual environment, where all the virtual tools necessary don't exist yet.  My Seoul presentation documented 150 physical tap points feeding network security monitoring devices - and this footprint is growing.  There are too many inspection points to replace with inline solutions.  You simply cannot place a proxy between every system in the enterprise.

One other crucial point to this discussion is that endpoint monitoring doesn't catch everything.  The strategy of "Defense in Depth" says that the more layers of security infrastructure you have, the better your chances of blocking or identifying malware.  Network security monitoring is catching lots of malware missed by endpoint inspection.

Threat Detection and Incident Analysis

Threat detection inside the enterprise is not just receiving IDS or endpoint monitoring alerts.  Threat detection analysts need to track down the source of the infection, which could be anywhere within the enterprise, and the primary tool for doing this is packet capture with a decrypted payload.  Because the source of the infection could be anywhere, we need ubiquitous packet capture and ubiquitous decryption capability.  Again, inline solutions don't scale to the number of monitoring points needed for this level of visibility.

No only does an out-of-band solution like static DH allow many more permanent monitoring points with simple network taps, it also allows enterprises to deal with gaps in visibility.  If a needed flow is missing, we can set up span or mirror sessions, capture the packets, and decrypt them after the fact.  This can't be done when relying only on inline proxies.  Adding a new inline proxy is a multi-month or more project.  In the midst of an active attack/infiltration every minute is crucial and network re-architecture is not an option.

Troubleshooting

Enterprises provide services through complex, multi-layer applications and backend infrastructure.  Inevitably, there are bugs in these applications and the source of the problem must be found through troubleshooting.  Network troubleshooters need to inspect the packet payload in order to find transactions on the wire in environments where there is extensive NAT and extensive encryption.  Both of these are very common in enterprise networks, and encryption inside the data center is becoming  more ubiquitous over time.  NAT, by the way, is being used in large part not for IP address exhaustion, but to allow for multiple redundant paths.

Complex applications can have hundreds of application and infrastructure nodes as well as many tiers, any of which could be the cause of a slowdown or failure.  Often there are no log messages on any device giving a clue as to the source of a problem.  Troubleshooters need to be able to take a packet trace at any layer of the infrastructure and decrypt the trace in order to find the transaction of interest.  This tracing could be between physical devices, like between a firewall and load balancer, or it could be between VMs.  Tracing in the virtual space could involve tracing between VMs on different blades of the same blade chassis or between VMs on the same ESX host.  Since the root cause of a problem can be anywhere in an enterprise, this translates into thousands of locations in a large enterprise where troubleshooters may need to take a trace, decrypt it, and locate a transaction.

Inline proxy solutions add cost, production risk, and latency at every layer in which they are implemented.  It doesn't take very many inline locations before the cost is in the millions of dollars.  Because the solution is inline, lots of redundancy has to be built in, including multiple columns (doubling the cost), bypass taps for failure scenarios, and extensive procedures for troubleshooting and failover.  Bypass taps and TLS decryption appliances can and do fail, and when they are inline a production outage is caused, usually in a critical part of the network and affecting multiple critical applications.  Proxy solutions also add 1-3 ms of latency to every packet.  This is not a big problem at one layer, but when many tiers are implemented with inlien proxy solutions the latency adds up.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 11 Jul 2017 14:51:57 -0400

> Network security monitoring is not just monitoring traffic that results from communications with customers and partners.  All it takes is for one user to click on a phishing email and there is malw…

I realize that you are making a substantive comment here and might not welcome a sales pitch, but FWIW there are ways of addressing this that don't involve examining each stream, and are likely as effective.   E.g., malicious domain redirect based on the domain name in the URL.   This is a lot cheaper than doing decryption and inspection of the contents of every TLS stream.   It also allows for other methods than payload analysis for detecting probably malware—e.g., pattern analysis.   I'm pretty sure that Stephen will tell you that this is an attack as well, and he's not wrong, but it has the virtue of not requiring that TLS be broken.

Also, please note that the proposal you are referring to does not solve this problem at all, unless the malware is already on a server on your network (which is the scenario you are describing).   If that's the case, why is TLS intercept the most cost effective way to detect this malware?   It seems to me that it would be a lot easier to just scan the file on the server before making it available for download.

Can you explain why that's not the right solution for this particular use case?

Also, although there was a lot of talk about how you need to do decryption-and-inspection rather than using proxies for performance, that actually just doesn't make sense.   Is it because the proxy is too close to the edge, resulting in trombone routes?   Otherwise I don't see how it could possibly be more efficient to decrypt and eavesdrop than to proxy.   Can you explain?

Sorry to hold your feet to the fire, but what you are saying here just doesn't make sense to me.

As for threat detection, again I'm not seeing how decryption and packet analysis is a win here.   Can you explain?

I see why it's helpful for troubleshooting, although again a proxy would take care of that.   But in any case, for the other two examples, it doesn't make sense even from a performance perspective.

---

From: Stephen Farrell <[email-masked]>
Date: Tue, 11 Jul 2017 20:15:40 +0100

To add to Ted's clarification requests:

On 11/07/17 19:39, Steve Fenter wrote:
> Network security monitoring is not just monitoring traffic that
> results from communications with customers and partners.  All it
> takes is for one user to click on a phishing email and there is
> malware inside the enterprise.  Once this happens, TLS becomes the
> enemy, because 30% of malware is TLS encrypted, and any TLS features
> intended to thwart payload inspection work against the enterprise.

I'd appreciate a citation for that 30% figure.

And if you had one an estimate for how much malware does it's own
obfuscation or home-grown crypto in addition or instead of using TLS.
The reason to ask is that as soon as malware does that then you
are back to analysis based on ciphertext only. From descriptions
of advanced attack schemes, they do seem to do both when calling
home or exfiltrating data. In which case I think your argument
falls.

> Malware does not always phone home out to the Internet on day 1 of
> infection.  

In what circumstance will malware phone home to a TLS server that
is playing your wiretap game? That seems utterly illogical but
maybe I'm missing a reason why someone's malware will use TLS to
talk to a server that is controlled by the victim network as part
of phoning home. Please clarify.

S.

---

From: Steve Fenter <[email-masked]>
Date: Tue, 11 Jul 2017 18:59:55 -0500

> On Jul 11, 2017, at 2:15 PM, Stephen Farrell <[email-masked]> wrote:
> 
> 

30% came from Cisco Systems at a recent Cisco Live conference.  Their numbers indicated 10% in 2015 and 30% today
> home or exfiltrating data. In which case I think your argument

I don't have any numbers for home-grown crypto.  I would think the odds are better for the enterprise if they can decrypt and inspect whatever portion is TLS.
> of phoning home. Please clarify.

Phone home would have to be caught by an inline solution on the way out the Internet.  I was just suggesting that malware could be caught earlier in the process with multiple inspection points throughout the enterprise.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 11 Jul 2017 21:21:40 -0400

To paraphrase (and forgive me for being a bit brutal here), you have no
basis for what you said other than handwaves and something from a Cisco
marketing presentation?

That is, "the odds are better if..." is a handwave, and not clearly true.
"Malware could be caught ... with multiple inspection points..." is only
true if those inspection points are able to detect the malware.   Phoning
home is actually pretty detectable using DNS snooping--you don't need deep
packet inspection, and it's orders of magnitude cheaper.

On Tue, Jul 11, 2017 at 7:59 PM, Steve Fenter <[email-masked]>
wrote:

> 30% came from Cisco Systems at a recent Cisco Live conference.  Their
> numbers indicated 10% in 2015 and 30% today
> >
> I don't have any numbers for home-grown crypto.  I would think the odds
> are better for the enterprise if they can decrypt and inspect whatever
> portion is TLS.
> >
> Phone home would have to be caught by an inline solution on the way out
> the Internet.  I was just suggesting that malware could be caught earlier
> in the process with multiple inspection points throughout the enterprise.
> >
> > S.
> >
>
> _______________________________________________
> TLS mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tls
>

---

From: Martin Thomson <[email-masked]>
Date: Wed, 12 Jul 2017 11:39:24 +1000

> I don't have any numbers for home-grown crypto.  I would think the odds are better for the enterprise if they can decrypt and inspect whatever portion is TLS.

Wouldn't malware avoid connecting to servers that offer the wrong
credentials?  Implementing elementary key pinning or overriding trust
anchors is pretty trivial - it's a feature that enterprises frequently
rely on after all.

---

From: Kathleen Moriarty <[email-masked]>
Date: Thu, 13 Jul 2017 11:08:21 -0400

Hi Steve,

Thanks for taking the time to detail out your concerns and current use
cases.  This is helpful.

> anchors is pretty trivial - it's a feature that enterprises frequently

It sounds like for malware, we could do something to better document
your security options as well as monitoring.  While the documentation
is there for key pinning and trust anchors, this might not be obvious
to network managers - what RFC to look at and how they fit together.

The points Stephen, Ted, Martin have made are good.  A TLS overview
document with an appendix of use cases might be a good start to help
fill this gap for operators.  Even if it isn't published, we could
figure out wiki space or something like wikipedia to make sure it's
available to operators. If the start of this documentation looks like
it will generate new work developing alternate ways to accomplish the
goals while maintaining the integrity of TLS, a new WG could even be
an option.

For malware, the proposed solution (Matt Green draft) isn't a great
fit as the server side won't be managed within your enterprise to
allow for the decryption described in the proposal.

For the other parts of your original message that were snipped from
this thread, I have a few questions/comments to see how we might be
able to narrow the scope and clarify a few things.

For the Troubleshooting description, could redefining the end point of
the server work as terminating at a load balancer to help with at
least some of your use cases?

For the threat detection and security analytics, I know a number of
current products rely on the ability to TLS.  The primary concern here
would be the remote server not managed by the enterprise.  TLS 1.3
prevents this from being possible and the proposed draft doesn't help,
so I think it would be best to figure out a way forward for this use
case either with the help of MILE (incident responders) or others.
Currently products use proprietary methods to accomplish this task (at
least some do).  For DDoS, the experts say they can work with
fingerprints of encrypted streams, so it's other attack types that may
require some thinking through of options.  I'm happy to chat about
that as I've done a lot of work in incident response and know of
others that may be able to assist as well.

I'm still reading through messages and drafts on this topic, so this
message should not be read as a position on either side, but more to
narrow the scope if possible and think through what is being requested
and why - so it is clarified.

Thanks.

-- 

Best regards,
Kathleen