A recent proxy upgrade, has seen me working many hours – fixing things that weren’t broken before. It was intended to be a drop-in replacement, but somebody couldn’t resist the opportunity to specify ‘a few minor’ new requirements.

• 1 year log retention of all traffic
• SSL interception to enable data leakage protection for all traffic types

The first doesn’t sound like a big issue, however it turned our we underestimated the logging volume for 8000 concurrent users. Additionally the reseller hadn’t flagged the issue either, I’m ‘sure’ they’ll pay more attention next time… As for SSL interception. It broke a host of things. Some lessons learned:

• Bluecoat ProxySG devices come with root CA certificates installed. Many site-admins using SSl fail to install the intermediate certs which slows down session set-up but also means we had to install many intermediates as the proxy does not go looking for them. This means manually finding and installing certs based on users calling the help-desk because they weren’t allowed to access sites with untrusted certificates.
• Commercial sites using self signed certificates. Bad practise, but sadly it’s not always up to engineers/consultants whether or not such a site should be honoured with a business critical status or not.
• Applications tunneling proprietary protocols over TCP:443. Some encrypted, some not so much. The ProxySG was configured to detect the protocol and to deny all unrecognised traffic. This breaks Adobe Creative Cloud for example. Skype is another hot-potato.

Skype in particular proved to be a big time-waster. As you may well know Skype uses proprietary protocols and tries very hard to remain hidden from prying eyes. As Skype was an application that was in use before the migration and the ‘as-is’ rule lingered, there was some pressure to get Skype working. The short is that I got it working without globally turning off SSL Intercept, well – to a degree anyway… Using Sysinternal’s tcpview revealed that Skype is coded to try and access the internet using the following sequence:

1. Try direct internet access using non-standard ports (>1024)
2. Try direct internet access using standard ports (tcp:80 & 443)
3. Try system proxy
4. Attempt to use proxy settings configured in Skype application (only if  nothing else works)

#4 Proved to be the final nail for not getting Skype fully working, more on that later. Obviously #1 and 2, fail on a corporate network without transparent proxies. #3 Required SSL bypass rules for Skype and Microsoft IP addresses (live.com and IP ranges), for a user to be able to sign into Skype. The Status indicator turned green, I could send and receive messages. Success! Until I tried to call someone. It turns out that Skype has changed their network a few times. In part due to the acquisition by Microsoft but according to insiders Skype was already in the process of stepping away from their pure peer-to-peer network due to some large network outages causing global news coverage. The process I observed:

1. Login to live.com (for both Skype and microsoft.live account holders)
2. Chat using Skype and Microsoft supernodes. The initial hosts are on Skype and Microsoft owned IP addresses. Some attempts are made to reach IP addresses owned by large ISP’s.
3. Real time traffic however did not rely on the nodes from #2. This traffic completely relies on a distributed network of ISP’s hosted nodes. I think Skype has adopted a CDN (Content Delivery Network) approach where they have supernodes installed on site at major ISP’s.

Having these super nodes means that Skype can handle traffic locally, reducing latency, and I believe these nodes have replaced the old Skype client peer-to-peer supernodes that used to be the clients with big internet pipes and long up-times. When a number of them went down simultaneously/in short succession the mentioned proverbial hit the media fan. The above is relevant as anyone who has ever tried to sniff (capture) Skype traffic will be able to tell you that there’s nothing in the packets that will indicate it is a Skype packet that you’re looking at. Browsers tend to boast about how recent they are and what they’re made of, no such luck here. Skype encrypts the packet content and the servers they use on port 443 (I think) don’t actually do SSL/TLS. This leaves our ‘white hat’ proxy device with an unknown session which it will happily drop for the sake of corporate security. Other than turning SSL intercept off for all traffic, I found no way to enable any real-time traffic in Skype other than basic chat and managing contacts. No file or screen sharing and no Skype calls. If you think you can convince your operations department that they should monitor and manage a dynamic list of random IP addresses in order to get real-time traffic working, then good on you but this is not something I’m willing to condone let alone propose to security or the operations manager… The final “Web Access Layer” rule looked like this:

• Source: LAN
• Destination: Combined Destination Object && “Destination: :443 (Contains)”.  Object also excludes “mail.live.com:443”, in order to not break inspection of Yahoo webmail.
• Service: HTTP/HTTPS Methods Object: HTTP CONNECT only.
• Action: Disable protocol (SSL) detection for HTTP Connect sessions.

              An additional rule was required to turn off SSL intercept for the same Microsoft and Skype destination object(s):       A last attempt was made to try and force the Skype client to use a SOCKS proxy, as a way to single out Skype traffic. However due to the system proxy settings allowing Skype to access the internet on ports 80 and 443, Skype never tried to use SOCKS. Even using a local client application level firewall proved to be pointless as Skype circumvented it and was able to connect to the proxy server despite attempts to block it. It turns out that Skype’s model of hiding, secrecy and obscurity is hard to block let alone guide. To the point where it’s impossible to permit real-time traffic without impacting business requirements. If you’re using Lync, then a better but longer term solution would be to keep an eye on rumours(?) of some sort of Skype integration into Lync. I’m expecting this to be some sort of SIP to Skype bridge through a system plugin/API, I’m very curious what the user experience of this would be. Corporately speaking this solution would solve many headaches but experience tells me it won’t be perfect.

Categories: Work
Comments Off on SSL Intercept headaches