PIA Kill Switch not working! PIA Client Disconnected, BUT FOUND WINDOWS BEING ONLINE
Sorry for sounding so alarmed, but this is objectively and technically alarming.
Last night I right clicked my PIA tray icon, did Disconnect (kill switch was on), it disconnected, left the computer on until this morning.
This morning, PIA client is running fine, it's still disconnected (kill switch is still on), BUT WINDOWS IN ONLINE.
My internet Network Connection is on and running (WTF), and my TAP-Windows Adapter is disconnected (normal).
I can see all my PIA manager and PIA network manager processes in Process Explorer are running "fine". I then right clicked the PIA Client tray icon and Connected again, then Disconnected, (without restarting the PIA client) and the internet was killed properly again.
I am moving the heck away from the PIA client ASAP until someone can ~guarantee this never happens again. (I don't care how technical and "consumer unfriendly" the solution is)
PS: Another issue I experienced before at least 5 times over a couple months was, coming back to my PC, hovering my mouse over the (either red or green) PIA tray icon (wanting to right click it), only to discover it disappeared because the process had crashed in the background a long time ago and windows was connected outside the VPN.
PPS: Does anyone know of any FULLPROOF way of making a VPN kill switch in ANY operating system (linux, unix, win) that runs on your PC (e.g. laptop while on the go)? Fullproof means your PC cannot under any circumstance and any stage ping the internet if it isn't through a successful VPN connection.
Last night I right clicked my PIA tray icon, did Disconnect (kill switch was on), it disconnected, left the computer on until this morning.
This morning, PIA client is running fine, it's still disconnected (kill switch is still on), BUT WINDOWS IN ONLINE.
My internet Network Connection is on and running (WTF), and my TAP-Windows Adapter is disconnected (normal).
I can see all my PIA manager and PIA network manager processes in Process Explorer are running "fine". I then right clicked the PIA Client tray icon and Connected again, then Disconnected, (without restarting the PIA client) and the internet was killed properly again.
I am moving the heck away from the PIA client ASAP until someone can ~guarantee this never happens again. (I don't care how technical and "consumer unfriendly" the solution is)
PS: Another issue I experienced before at least 5 times over a couple months was, coming back to my PC, hovering my mouse over the (either red or green) PIA tray icon (wanting to right click it), only to discover it disappeared because the process had crashed in the background a long time ago and windows was connected outside the VPN.
PPS: Does anyone know of any FULLPROOF way of making a VPN kill switch in ANY operating system (linux, unix, win) that runs on your PC (e.g. laptop while on the go)? Fullproof means your PC cannot under any circumstance and any stage ping the internet if it isn't through a successful VPN connection.
Comments
If you manually assign your computer's IP address in the network adapter on Windows, I think it should also be leak-proof as currently PIA doesn't restore the default gateway in this situation when exiting, but most importantly there can't be DHCP renews going on to possibly interfere. If it still fails, then something else on your computer has to be "fixing" the network without your knowledge outside of PIA.
I'll start a long-lived test in a few minutes to see if I can reproduce this on my end. This is a serious issue we'll fix ASAP if I can reproduce.
------
What I can confirm is the PIA app on Linux will never leak for sure. On Linux it uses iptables to drop all non-VPN traffic, so even if the app crashes, dies, gets killed, as long as iptables is not being reset (it shouldn't if you don't use a third-party firewall like UFW or other stuff), then you should be guaranteed leak-free. You can check with the "sudo iptables -S" command while the killswitch is enabled.
Another fun thing to do on Linux is to use network namespaces so that the only network adapter ever available to the applications you intend to protect is the tun interface.
After reading your post I also discovered that something at some point had reset my network IP configuration to Automatic. Probably a windows update or something. I'll test once more while having an actual manual ip and dns set. But I'll switch to windows as a virtual machine inside linux on all my PCs asap.
Pardon my ignorance, but can't PIA take a more aggressive approach to the kill switch?
I see its process hierarchy has duplicates: pia_manager.exe, rubyw.exe, pia_manager.exe, rubyw.exe, pia_nw.exe, pia_nw.exe, pia_nw.exe, openvpn.exe.
But that doesn't seem to be providing the redundancy I would expect it wants to achieve with this. E.g. Can't you make a separate elevated service in the background that checks every 0.1s if the PIA agent is up and running, every 1s if the network configs are not changed, every 10s ping the internet to make sure it's through a vpn? This won't be 100% fine as even a 0.1s window of potential leakage is bad, but at least it's not a whole night or weekend... Also the same approach if PIA tried to connect to the internet but it timed out. Maybe it should try ever 10 seconds, forever, instead of just giving up?
Three alternatives:
A. Configure Window Firewall to disable outgoing connections on all Private networks, then make the OpenVPN Adapter Public.
B. Fiddle with the Windows routing table directly. Requires some strong networking awareness.
C. Create a virtual router using VM software like VMWare, Virtualbox, QEMU, Parallels, to install a router-friendly OS like PFSense or Linux + OpenVPN config in a new guest virtual machine. Route your host machine through the guest.
We can't be too happy about it though since sometimes you can't use static IP addresses (like in a cafe/hotel). Also your network adapter can be (re)set back to automatic by a number of external sources when you don't expect it. Like installing a new adapter or remoting software, installing a new network driver, windows updating itself etc.
Configuring Windows Firewall like in "alternative A" mentioned above, does not cover everything, especially not in Windows 10 from Microsoft and from deeply installed 3rd parties.
I suggest the default VPN killswitch behaviour for everyone to be:
Install VirtualBox as a (startup) service ( - an open source Virtual Machine software (and not Microsoft's Hyper-V)), install PFSense on that VM, then castrate all Windows network services and network drivers, and use the Windows Loopback Adapter as the only network gateway, which connects directly to the PFSense VirtualBox - where you have a proper firewall and a proper unkillable VPN connection. And you can also use this VM as an enterprise-grade router for the rest of your house, cafe etc..If you have a better than average laptop or PC, then I suggest installing Linux as your Host OS, then installing Windows as a VM inside Linux (if your hardware drivers are supported in linux, you can do hardware passthrough and there will be practically no overhead for playing videogames or doing intensive work), then installing PFSense as another VM, and routing everything through PFSense. That my dudes is the actual -normal- in privacy and security and reliability. Otherwise you're just pretending and hoping.
I opted to go the route A and disable Private internet connection traffic. It was the easiest to implement, and proved to work. Only downside is that the network will still appear to be "on" in the task tray, so I lose the visual queue of a network without internet, even though I have confirmed no outgoing connections are possible. I have never seen an update mess with user firewall configured rules, so IMO this is the best approach to make sure off is off. Possibly also set manual IP and DNS settings for the network, but the user is assumed to be stupid and those aren't sacred apparently.
I also agree with @tdd that if Microsoft really wants to get through their own firewall, I'm sure they can, so probably not the best solution for the ultra paranoid ... so like 90% of this forum amirite
@Max-P, this might be a good improvement opportunity for the Windows client actually. Verifying and re-setting a manual IP and DNS setting on program launch just to make absolutely sure the kill switch works as intended. Another radio button and 2 added dialogs to the advanced settings, and some behind the scenes magic to never have a leak again.
Ideally this really should be plugged with a network filter driver that does the firewall job properly, but then comes a whole world of antivirus/firewall conflicts and a very non-obvious failure point if the driver gets stuck in blocking mode with the application shut down as there wouldn't be any way for the user to even know the killswitch is active. And in general convincing people to install two drivers to run PIA.
The best way to not have killswitch issues is really to not use Windows, or isolate Windows from the outside with a router of VM setup. The problem is that there's just too many of those "utilities" fighting to be the one responsible for the network to be able to be 100% reliable. About everytime a killswitch issue pops up, it's something external resetting the network. Just now, I have this thread open in another tab that's precisely about yet another bloatware implementing a "feature" that ends up bypassing the VPN in really odd ways despite the routing being set up properly. I have no idea why it happens but I have a feeling they're probably sending bypassing the whole network stack and sending the data directly through whichever link it decided is the best.
But, let's be honest. Windows isn't going anywhere and likely most people that just want a solution that works (for better or worse) use Windows. So, this issue isn't going anywhere either.
I think you may be onto something about isolating Windows though. I had tried to do that about 2 years ago by running PIA via openvpn on my Tomato based Asus router, but quickly found that:
1) Getting my desired VPN configuration on the network (some machines on the VPN, others off) was not as straight forward as it sounds.
2) Altering that configuration required modifying openvpn config files, SSHing to the router, and disrupting other PCs on the VPN while I took it down to switch. (Cumbersome and intrusive)
3. Even with 256MB of RAM for a large NAT tables and a 600 MHz CPU, the router wasn't able to keep up with the demand and frequently would have packet loss and consistency issues under modest loads. (Non-starter)
Realizing this, I either needed a beefier network appliance, or use the VPN solution on the client. I chose the later (free) solution.
but ... running a container as a builtin firewall (docker, VM,etc.) for the client app so Windows effectively is using a container to talk to the outside world seems like it has some legs. Though, any solution that alters the way the PC interfaces with the outside (regardless of OS) will be intrusive because it needs to live outside the VPN app and can muck up a lot if users are not aware what they are using. Kinda a rock and a hard place for a solution provider (PIA) if you ask me.
I realize none of the strong options I suggested are particularly user friendly or usable depending on the situation (you mentioned setting it up on Tomato - while that works it tends to cap around 30-50 Mbps so dealbreaker for some users). The VM solution in particular isn't guaranteed to help either anyway because if Windows decides it grabs an IP on the main adapter despite it being bridged to a VM, we've got leaks again.
I'm sure there's a way to better handle this. If anything, we probably should be able to handle it similarly as we do on Android by basically keeping the TAP adapter active and running but leave packets go nowhere when the VPN is off. There's an option in the TAP properties in Device Manager to set the interface to either always appear connected, or to be only connected when OpenVPN is driving it. If you leave it always on, I'd imagine it will prevent leaks but we still don't know whether Windows will be smart enough to automatically bump back the normal adapter to a higher priority to restore connectivity
On a whim, I tried out your last comment about keeping the TAP adapter always on:
Windows Settings -> Network & Internet -> Ethernet -> Change adapter options -> Ethernet 2 Windows TAP-Windows Adapter V9 -> Properties -> Configure -> Advanced Tab -> Media Status -> Always Connected -> OK
No dice on Windows 10 x64 1709. The TAP adapter indeed stays active when the PIA client is disconnected and/or closed, but Windows proves to be an overachiever and falls back to the regular network adapters connection. I tried with Kill Switch both on and off and got the same result. Unless there is a driver configuration thing that needs to be set or adjusted before installation, for me, this was not a viable solution. Just as a data point for PIA dev.
Note that previously I was using v75 and before that v65,and when PIA was closed with Kill Switch activated, the main adapter would get put back online but with corrupted settings such that the network would stay disconnected. For me, this was perfect because the biggest thing I am looking to avoid is dropping my PIA connection, and reconnecting to the primary network while reconnecting or restarting the client. This is especially important because since there is no autoreconnect (which was recently requested elsewhere on the forum and would be really great), to automate this process, I have a program kill PIA and restart it until a VPN connection is secured. So I basically already do what the request is asking for, but way dirtier with a watchdog process and force close operations, which honestly I would love not to do. I know that others might not like how this worked in the past versions because until you either restarted the client and connected to the VPN or reset the ethernet adapter IPv4 settings, you can't get online, which is probably why you guys fixed it in the newest versions (v77). Good for you guys at PIA and everyone else, bad for me. This client quirk is likely the scenario @Max-P was referring to a couple posts up ... and I miss it
Anyway, onward and upward.
VERY IMPORTANT NOTE: This will Deny all outbound connections UNLESS there is a specific Allow rule set under the Outbound Rules section. This sets the default behavior, not all behavior. It is common for programs (especially Windows itself) to add Allow rules that will bypass this default behavior.
11) In the left pane select: Outbound Rules
12) In the right pane select:
Actions -> Outbound Rules -> New Rule
13) In the New Outbound Rule Wizard make the following selections:
Rule Type: Custom
Program:
All Programs
Services -> Customize: Apply to all programs and services (default)
Protocol and Ports:
Protocol type: Any (default)
Scope:
Which local IP address does this apply to?: Any IP address (default)
Customize the interface types which this rule applies -> Customize: All interface types (default)
Which remote IP address does this apply to?: Any IP address (default)
Action: Block the connection (default)
Profile: Check Private
Name: Block All Private Network Traffic
and click Finish
NOTE: This will go above and beyond the default connection settings made in step 8. Here, a specific block rule is defined for all Private connections. Windows firewall prioritizes Deny rules over Allow rules. So, all the allow rules that Windows and other programs add that pertain to outbound connections on Private networks will be ignored. This step effectively makes step 8 redundant, and is much more important as it makes sure exclusions to the default behavior don't interfere with your security.
14) Close the Windows Defender Firewall with Advanced Security and Windows Defender Security Center windows.
Update 3/18/18: Added the creation of a Outbound Deny rule for all Private networks to further protect against everything that doesn't explicitly have a firewall Allow rule in place.
I've looked at the firewall method in the past, but the problem I ran into was that the Windows Firewall favors Deny rules over Allow rules, so there is no way to block everything except the PIA app or OpenVPN to let it go through. So it ended up being a very lenghty process where you also had to deny specific apps you really didn't want leaking because there was no way to punch a hole for the PIA app.
Soooo, if I understand this right, you're relying on the Killswitch causing Windows to recognize your local adapter as two different things (Public when killswitch is active, Private when not killed) and then use that to kill the network if the killswitch fails or PIA exits. That's really clever!
I'm going to test that out myself a bit later to verify but if I can verify the different states I'm going to be sending lots of people to that post.
The killswitch shouldn't affect your mapped drives however unless you need to route to your router first. Most people's mapped drives are on the same physical network so the default gateway shouldn't be involved at all since you can talk to the machine directly. Is your mapped drives located on a machine on a different subnet, or mounted over the Internet?
Yes, Windows firewall does make the Deny rules a priority over the Allow rules. This is what makes the above procedure work! You almost got it but not quite. Using the above procedure, the Killswitch in PIA actually doesn't do anything for you; on or off should be the same.
We are specifically telling Windows:
So when PIA is not running or disconnected, Windows tries to use the Private network to make the network connection but can't as the outbound packets are all denied. When PIA tries to connect, it will bridge the default Private network adapter to the Public TAP network allowing the connections to pass because Public networks are set (by default) to allow outgoing connections. This works, but has some really notable shortcomings. This may be a bit confusing ... sorry, I tried to make it as clear as I could.
Windows Firewall is not a cascaded filter firewall like iptables. It just doesn't do that. Rules don't have priorities, so there's not a really good way to be very selective about connections without both breaking default Windows functionality and babysitting the allow/deny rules over time as programs are installed and updated. So buttoning up the connection completely using just the built in functionality is probably not possible. I think the solution I gave in the previous post is relevant, but only for a very very specific use case. Because of this and the concerns I mentioned here, I'm going to try some other firewall software for myself.
I think issues regarding proper setup and use of DNS leak protection and Killswitch are a pretty big concerns that warrant prescribed support, or at the very least setup tutorials from PIA to make sure customers are using the software in conjunction with the supported OSs correctly. The forums are great for sharing ideas, but it's pretty hit and miss and lacks a level of authenticity that a PIA suggested solution would provide.
Yes, this is why I'm a bit confused. If when on the Private network, no connections are allowed, how does PIA connect to its servers?
It can't be the TAP adapter, because it's not quite how it works. All the TAP does is bridge raw packets between the TAP adapter itself, and the OpenVPN process. So public internet access through the TAP using your ISP is impossible, because for the TAP to report "connected" to Windows, it needs to have an OpenVPN process attached to it. As soon as OpenVPN goes down, the TAP goes down with it.
Now, for packets sent through the TAP to reach PIA's servers, they first go through the TAP which sends them to OpenVPN, which encrypts them and then send them to PIA's servers. The connection between OpenVPN itself and PIA's servers is done over your regular non-VPN adapter. If it tried to talk to PIA's servers over the TAP, it would just loop into itself infinitely as that would be OpenVPN->TAP->OpenVPN->TAP->OpenVPN.
This is why I think you are relying on the Killswitch's behavior to actually punch a hole through your firewall. When PIA enables the killswitch, it changes your network settings and Windows sees the adapter as a new network profile. When PIA exits or dies, the original network profile returns. So you have three states: Private (no VPN), Public (no VPN, killswitch enabled), and Publc (on VPN, TAP), and the magic happens when it switches between Private and Public.
I've recorded a quick video of my test, pay close attention to the network adapters: https://d.max-p.me/pia/forum/30340/killswitch.mp4
I wound up scrapping the use of Windows Firewall for a slightly more robust solution with Comodo Firewall 10, so I can not easily revert to test the above. I'll take your word for it and hope others do as well! I first tried ZoneAlarm Pro ... and while not free, it still couldn't do what I wanted (and I think Windows Defender is actually a better piece of software). In Comodo (which is free) I implemented KillSwitch via a set of global firewall rules that get activated for connections using the TAP adapter (connected to PIA via openvpn). This firewall also lets you add priority to rules so Allow and Deny rules can be mixed unlike Windows Firewall. This way I set a higher priority for LAN traffic (so it uses the LAN Ethernet adapter and not the TAP connection), and lower priority for the VPN connection (tied to the TAP adapter), and conclude with a global block for all outgoing connections. I currently use it without the PIA KillSwitch engaged and did lots of tests to make sure it was working under all PIA client states (closed, started and disconnected, started and connected). No issues keeping all non-VPN traffic stifled regardless of what software is sending the connection attempts.
I will continue to use the PIA client KillSwitch on other computers using the PIA client where the VPN is used occasionally. For this use case, I don't necessarily care if I leak when the program is shut down because while I am connected to the internet and want the VPN, I'd have it active (e.g. laptop at the coffee shop on wifi). But the computer I was most concerned about, which triggered this whole discussion, uses the VPN for all connections 99% of the time ... and to me, a working KillSwitch is critical as I have noted the VPN connection drops on occasion. For this really niche use case, I will likely stick to my Comodo Firewall solution so I can set it and forget it as I update the PIA client over time. No further worries that a client update could potentially gum up the works.
Thanks for the great discussion on this thread.