Browser certificate warnings and its not your firewall? It might be your ISP...
Very recently I have been consulted by clients and colleagues on network performance issues, all with one similar symptom: "I'm getting a 'your connection is not private' error in my web browser when I try to access this one site!". Each of these incidents I've encountered turned out to be a problem with the internet provider. In this article, I'll discuss ISP proxying and why you should not use it.
What is traffic proxying?
Before we dive in, you should understand how proxies work. What is traffic proxying? Why would you proxy traffic? How is traffic typically proxied? I'll start with defining the concept of proxying.
Proxying
Let's first understand the definition of a proxy: a proxy is someone who represents someone else. That's simple.
A simple scenario.
Imagine this scenario: you are sick, and you want some cough drops. You don't want to drive, so you phone a friend. You ask your friend to go to the store for you and buy you some cough drops. Your friend is the proxy.
You, the client, ask your friend over the phone, for a specific flavour of cough drops. Your friend then picks out the flavour you asked for. You ask your friend to ask the cashier, "how much do the cough drops cost?"
The cashier tells your friend, "$3.99". Your friend then tells you, over the phone, "it costs $3.99". The cashier then asks your friend, "would you like to purchase the cough drops?" Your friend then asks you the same question, and you respond, "yes".
Your friend was acting as the proxy in this scenario. Why? Because your friend was
handling the transaction of the cough drops on your behalf, while you were at home, in bed, resting.
Proxies in networking terms.
Now let's talk about network proxies. Network proxies, also known as internet proxies, do the same essential thing as what was previously discussed.
When you go out to the internet, your proxy, a hardware device, will handle the internet traffic on your behalf.
When you go to google.com, the request traffic will be sent to your proxy, and then your proxy will send the request to google.com. Your proxy will then handle the reply traffic from google.com, process it, then forward it to you.
Below shows an example of the TCP handshake process between a client and server, with an internet proxy in between.
This is the TCP 3-way handshake process when there is a proxy involved.
Why internet proxies are used
Internet proxies can be useful, depending on your network security requirements. If you need to protect your endpoints from vulnerabilities like cross-site scripting attacks, then you should use an internet proxy.
In most cases, internet proxies modify the contents of a webpage. Proxies will filter out malicious code found on websites. Some may be able to prevent malicious files from being downloaded to the client.
Internet proxies can block endpoint access to websites, restrict unsafe or inappropriate content, and keep endpoints safe as they browse the internet by modifying the contents of HTTP data as it is filtered through the proxy.
How proxies are used
Modern internet proxies typically live on a next-generation firewall (NGFW). Fortinet, Palo Alto, and SonicWALL are some vendors who carry support for proxy-based packet inspection, through some simple profile configurations.
To add a layer of protection, a proxy will have its own SSL certificate installed on itself and when an endpoint reaches out to the internet, it will validate trust to the website against the SSL certificate installed on the proxy, and not that of the web server hosting the website.
Now, your proxy will reach out to the web server, and it will work against the web server's certificate to determine if the website is trusted or not.
Your computer will not know anything about the web server's certificate, and it can add a double layer of protection if your proxy determines the website is not secure and could block the connection before your users access the website in an insecure manner.
In common scenarios, as much as I've dealt with it, there are times when proxy SSL certificates do not get trusted on the endpoints, generating browser messages, as shown below.
This is because the CA for the certificate is not added to the endpoints' certificate manager.
The simple solution to that problem is to either obtain a certificate from a globally trusted CA, such as GoDaddy, and then install it on the proxy. Otherwise, you can manually install the device certificate to each endpoint.
Nevertheless, as your endpoints access the internet, the traffic will get filtered through the proxy, and any content that is deemed malicious, is blocked.
Why proxying can be a problem
Proxying can cause significant network performance issues if it's not implemented correctly, I will now discuss the reasons why.
High latency & network slowness
When you configure your firewall to act as a proxy, your firewall now has to handle twice the amount of traffic it normally does. Why is that?
Well, think about what we just discussed: you have a session between a client-to-proxy and proxy-to-server. That's twice the amount of sessions that now have to be handled by your firewall.
Now, think about your ISP CPE (customer-premise equipment). Do you think those little modems are meant to proxy network traffic? Especially high amounts of it? I don't think so.
CPU/NPU and RAM usage goes up
As a result of the higher session traffic, your firewall is now subject to high CPU or NPU usage and RAM usage.
Proxies typically inspect session traffic in order. The data stream for the packets is stored in RAM as the proxy inspects the packets one by one. The more traffic in a session, the higher the RAM usage. As your firewall inspects the packets, it will work significantly harder to process the traffic as fast as possible, elevating the CPU/NPU usage.
Once the firewall is done inspecting the data stream, it then tears the data stream apart and forwards the packets it decided safe in order, to the endpoint, the endpoint, and the endpoint then reconstructs the data stream on its own.
This repetitive process of proxying traffic, buffering the packets in the order of the data stream, and the individual inspection of these packets, results in higher CPU/RAM usage overall.
Performance goes down
As a result of the higher system usage on your firewall, you will notice a significant deterioration in network performance. Firewalls will slow down if their CPU and RAM are captivated to just proxy inspection. Imagine the rest of the traffic the firewall has to handle on top of it. The longer wait times for packet handling cause higher latency, and that's subject to packet TTL expiration and packet drops if the firewall's memory buffers become full.
Session timeouts & risk of failure
Another thing you need to take into account is that there is a chance of session timeouts when there's a proxy.
Unbalanced client-proxy and proxy-server sessions.
Sessions have timers. If one side of the proxy session is not in time with the other (of course they won't be, but if they are too far apart), that will cause a performance degradation.
If your proxy-server session hangs due to latency, whether caused by the physical distance between your uplink route to the content delivery network of the web server, or there's a problem with your proxy, or the web server itself, that time will reflect on the client-proxy session.
Risk of failure
As discussed before, your proxy has memory buffers. If those memory buffers become full, traffic will time out and drop. Proxies can cause a direct single point of failure in browsing the internet if the device becomes stalled and can no longer handle session traffic.
If you use a standalone proxy device, and all of your DNS is intercepted at that device (i.e. your endpoints are set to resolve DNS through this device), then you better hope your DHCP server hands out alternate internet-facing DNS server addresses, because if they don't, then if your proxy device goes down, your devices will no longer resolve to the internet.
Browser warnings
The most common problem I've run into with proxies is the endless amount of browser warnings due to improperly deployed proxy systems.
Untrusted certificates
As mentioned previously, a computer has its certificate manager, and it contains CAs and some pre-defined certificates. Certificates validate the legitimacy of networked systems, servers, and endpoints.
If a proxy device has a default, untrusted CA certificate installed on itself, then your users will experience browser warnings as they try to browse the internet, if you either do not import the device certificate to your endpoints, or you do not install a trusted CA certificate on the proxy.
If it's not your firewall, it's your ISP.
Alright, now let's shift gears and discuss the next topic. So, you understand how proxies work, you're aware that your firewall may support proxy inspection, but you've gone through and validated that your firewall is not acting as a proxy.
Your users are getting browser warnings and you know its something happening on your network because you can access the website without an issue anywhere else.
What do you do now?
It might be your internet service provider.
You should check with your ISP to see if they use web proxying or have any advanced web security features rolled into your subscription.
Check the SSL certificate in the browser
Take a look at the certificate to the website you're trying to access. Check and see what the CN for the certificate is and who the CA is. If it doesn't add up to what you expect, then there's a proxy sitting between you and the website.
Below shows an example of a user viewing the SSL certificate in their web browser after getting browser warnings on their corporate network.
ISPs known to proxy traffic.
Below is a list of ISPs who I've run into that proxy web traffic:
Comcast
Verizon
AT&T
CenturyLink
I have had to call each of these providers in the past and ask them to turn off the advanced security features on behalf of my customers to stop the proxying from slowing down their networks and causing website access issues.
Research the certificate CN.
If you're unsure of whether or not your ISP is proxying the traffic, you should copy and paste the CN of the certificate to google and see the results. Look around to try and find the certificate CN. If it's untrusted, chances are that it belongs to a proprietary device.
Conclusion
Proxying can cause significant problems to your network, if not done properly. If you are experiencing proxy behaviour on your network, and you cannot trace where it's coming from, then chances are that your ISP is proxying the traffic.
Thanks for reading this article. I hope you found this article helpful. Feel free to leave feedback in the comments.