Private Internet Access and pfSense

I’ve long been interested in using a VPN to access out-of-country content as well as to secure whatever nefarious activities that I may be up to. Not that I have anything to hide but it’s no one’s business what I do.

I’d toyed with Tor back when I installed Google Voice. It works but is dog slow. My interest in this subject was recently rekindled when I read this article. I found myself drawn to Private Internet Access (PIA) and it’s promises of low cost and unlimited bandwidth coupled with no throttling. I signed up for a monthly plan with the intention of either canceling it if I didn’t like it or switching to a full-year subscription if I did.

Signing up with PIA was easy enough. You can download client software that runs on your individual machines if you wish, but since I have a pfSense firewall, I knew I should be able to channel all traffic through PIA over VPN so I wouldn’t have to install any additional software. Here are the config steps for pfSense 2.0.1.

To start with though, make note of your current IP address as evidenced by http://ipchicken.com. We want to confirm that your IP actually changes when we’re all done.

pfSense Config Files

SSH to your pfSense server and cd to /etc. Create a file “openvpn-password.txt” with two lines, one for your PIA userid, the other for your password.

You also need to download this file from PIA and extract its ca.crt to /etc.

Set 0600 permissions on both of these files e.g. “chmod 0600 /etc/ca.crt”. You can exit SSH at this point.

Certificates

In pfSense’s webConfigurator, go to System and select Cert Manager. Add a new CA, call it something like “Internal CA” using method “Create an internal Certificate Authority”. Fill in the Distinguished Name pieces below as you see fit.

Now click on Certificates and add a new certificate using “Create an internal certificate”. Call it something like “OpenVPN” and select type “Certificate Authority”.

OpenVPN Service

Go to VPN, select OpenVPN and click the Client tab. Add a new client. Leave all defaults except the following:

  • Server host or address: enter your desired PIA host e.g. us-texas.privateinternetaccess.com
  • Check “Infinitely resolve server”
  • Give it a meaningful name e.g. “Private Internet Access OpenVPN”
  • Clear “TLS Authentication” check box
  • Make sure the CA and Cert you created are selected
  • Select “BF-CBC (128-bit)” for the encryption algorithm
  • Check “Compress tunnel packets using the LZO algorithm”
  • Enter the following for Advanced at the bottom:
auth-user-pass /etc/openvpn-password.txt
ca /etc/ca.crt

Click Save to write your config and the OpenVPN service should start. You can click the blue “S” just under the Help menu to confirm that its status is “up”. Also check the log (blue “L”) to make sure there aren’t any errors.

Enable Interface

Go to Interfaces and select (assign). Click the add button. A new entry called OPTn should appear with “ovpnc1” as the port. Click Save. Now you can enable your new interface. Go to Interfaces and select OPTn. Simply click Enable and Save. Note that you can rename the interface if you want to something like “VPN” but it’s not necessary.

Restart the OpenVPN service so everything is in sync. Go to Status and select Services, then click the restart button beside the OpenVPN service. Ensure that the OPTn gateway has an IP. Go to System: Routing and make sure the Gateway has an IP address.

Firewall Config

At this point the OpenVPN service is running but you aren’t using it. You may not even be able to access the Internet in this state. While there’s a lot you can do to tailor your firewall access, here’s a quick way to route all your outgoing traffic through your new VPN connection.

Go to Firewall and select NAT, then click the Outbound tab. Select any existing rules and delete them. Select the “Automatic” option at the top and click Save, then select “Manual” and click Save. You should see a new set of rules which you can activate by clicking Apply Changes.

There’s lots more that could be done to pfSense to tighten up your security but this is a starting point.

Defining Exceptions

One client of mine requires me to log in to their Cisco VPN. Unfortunately this does not work through the VPN connection I just set up. It’s easy to force connections to their VPN server over the WAN interface, bypassing our VPN, by defining a new route as follows:

  • Go to System and select Routing. Click Routes and create a new route.
  • Enter the IP address of the remote host, in this case my client’s VPN IP address. Make sure the WAN gateway is selected and enter an appropriate name.
  • Click Save then Apply Changes.

Attempts to connect to this IP address from any device on your network will bypass the VPN and go directly to that IP address. Too easy!

Test

At this point you should have all your traffic going through the PIA VPN. You can confirm this by refreshing your ipchicken screen which should now show a different IP address.

Final Thoughts

The biggest concern I had with using a VPN like this was the performance penalty. I used http://speedtest.net before and after and there’s certainly a considerable penalty to be paid, but it’s not as bad as I had feared. I have 100-megabit service and without the VPN connection realize a max throughput of about 93 Mbps. Running the same test WITH the VPN enabled cut that in half. Fortunately, 40+ Mbps is still plenty fast for most of my needs. I have to wonder if the fact that my pfSense is running as a virtual machine plays much of a role here since there’s a whole lot of encryption going on. Perhaps I’d be better off using dedicated hardware, but that’s an experiment for another day.

I may give a few other services a try to see if they offer improved throughput. So far though I am impressed with Private Internet Access.

NOTE: These instructions are for pfSense 1.2.3. For 2.X versions, please see the additional requirements in this article, specifically relating to a bug in older versions of pfSense. Please see the first quote there for more information.

VMware ESXi Corrupted iSCSI Datastore

Terrible scare yesterday. I rebooted my ESXi server, which hosts several critical systems, and ESXi didn’t recognize the iSCSI datastore. It could see the iSCSI LUN, but attempts to add it as a new datastore warned that all data would be destroyed. Backups were available but I’d rather have (a) found out what the problem was/why it occurred and (b) fixed the problem. So started my 12-hour learning process.

This all came about because I rebooted the host in an attempt to get new vNICs working under a VLAN. Yes, holidays are a great time to play and learn, it would seem. As a consultant, I should spend even more time doing this sort of thing.

So why couldn’t VMware mount the datastore? Did something happen to it? I tried all manner of fixes, including ultimately reconfiguring my host from scratch to wipe out any traces of the old datastore in the hope that some config was corrupted, but no go. The last resort, it would seem, would be to repartition the iSCSI LUN, which to me seemed a last-gasp effort. Since I was at that stage, I followed the following instructions:

esxcfg-scsidevs -c (take note of the disk device)
fdisk -l /dev/disks/t10.F405E46494C4540096D427739387D25525F4A5D245638787

Hmmm, this didn’t show “fb VMFS” like it should, but rather “SFS”. A quick search told me that this indicated a Windows dynamic disk. Uh oh… Rewinding a bit, a couple of weeks ago a Hyper-V Windows Server of mine had lost its iSCSI connection. The disk was there but it couldn’t access it. I saw that it was marked as, you guessed it, Dynamic! Does that mean everything is toast? I can only guess that ESXi saw the disk as VMFS since it was first created, and continued to access it as such even once Windows had marked it as dynamic. Since the Windows server didn’t really use it, odds are that the two didn’t interfere with each other except for the partition table.

Only one way to find out. I continued with the terrifying process of repartitioning the datastore:

fdisk /dev/disks/t10.F405E46494C4540096D427739387D25525F4A5D245638787
d (deletes the partition: gulp!)
n (create new partition)
p (make primary)
enter (accept default)
enter (accept default again)
t (change partition type)
fb (VMFS)
X (expert mode)
b (change beginning of partition)
1 (first partition)
128 (select secdtor)
W (write changes and exit: double gulp!)
vmkfstools -V (discover the VMFS)

At this point, in vShpere I did a Rescan on the Storage Adapters, and after clicking on Storage, to my amazement, my iSCSI datastore was there! I added my VMs to the inventory and started them up, and all was fine. Very cool.

To finish things off, I disconnected the LUN from that rogue Windows server and removed the LUN from OpenFiler so this can’t happen again. While it’s fine for different ESXi hsots to share a LUN, it’s clearly a bad idea for Windows and ESXi to try and play together…

VMware ESXi: at home!

A co-worker was making me jealous the other day about how he built an ESXi whitebox, and I got to thinking that I needed something like this myself to host my company’s servers. I have been an avid virtualization junkie ever since the original Virtual PC was made available to me in an old MSDN subscription, and this obsession continues to this day. Currently I use VMware Workstation 7 on my high-powered (Core i7, 9GB RAM) but wholly underutilized HTPC, and while it works well enough, it’s really not very “enterprisey”. Neither is a homebuilt ESXi server, but I can certainly make it pretty close, and it would be far superior to the HTPC which to my horror people often shut down when they are done watching something.

I have a Dell Inspiron 845 that was used by an employee for a past project. It’s a reasonably powerful machine with a quad-core, VT-enabled Intel processor and 8GB RAM, so I figured it would do the trick. According to http://vm-help.com, by simply adding an Intel 1000 GT or CT NIC, ESXi 4.0 will install without any modifications or funky drivers. I picked up a couple of these NICs for $45 each, installed one (the PCI-e CT version) in the 845, and within minutes I had my own ESXi server. Sweet! The only gotcha is that my Windows 7 host can’t run the vSphere management tool, so I need to run it under an XP VM. Oh, the irony!

Next up was to use some enterprisey storage. I have an OpenFiler server sitting in my wiring closet with a 400GB iSCSI volume that’s sitting idle, so after some frustration getting ESXi to see the iSCSI target, I now have what should be a very robust data store for my VMs. I’ll make a post later about exactly what you need to do to get this configured.

I installed the VMware standalone converter utility and migrated my VMware Workstation VMs, initially a development Oracle server and Redmine plus an (*ahem*) bittorrent server, to the new server in its iSCSI data store. Everything went exactly as I’d hoped it would, very smooth. I only needed to reset some static DHCP mappings due to MAC address changes.

I still needed a backup solution though, and last night I got one working. Briefly, it’s the ghettoVCB script which is highly regarded, and I can see why. There are some nice guides on how to get it set up, and I’ll post more on this later. Here’s hoping that tonight’s daily backup works!

Enable SSH Access to VMware ESX Hosts

Completely unsupported and an easy way to pooch your server and its VMs without even trying, possibly getting yourself fired in the process, it is nevertheless desirable to be able to SSH to your VMware ESX/ESXi hosts. This dangerous feature is disabled by default, but her’s how you can enable it:

  1. Go to the ESXi console and press Alt+F1
  2. Type: “unsupported” (Note: there is no prompt for this, just type and hit ENTER)
  3. Enter the root password and hit ENTER
  4. At the prompt type “vi /etc/inetd.conf”
  5. Find the line that starts with “#ssh” and delete the leading “#” (use “x”)
  6. Save by typing “ZZ”
  7. Do a “ps | grep inetd” and make note of the inetd process id (first number)
  8. Issue “kill -HUP <pid>” where “<pid>” is the inetd process id from above to restart the management services (or reboot if that’s an option)
  9. Enjoy your new SSH capabilities

These instructions were shamelessly stolen from this famous article, but updated because their restart command didn’t work for me.

VMware ESXi 4

Virtualization is something I am obsessed with, and we started using VMware ESXi 4 at work today. I was reading some stuff on ESXi 4, specifically how to create linked clones, when I came across some idiot who said they were running ESXi in a virtual machine under VMware Workstation; as if! This caught my attention, and minutes later, after having spent countless hours over the last year or so trying to get ESXi 3.5 and 4.0 working on some unsupported hardware at home, I have ESXi 4.0 running!

I tried to get it to talk to my iSCSI target of my local OpenFiler NAS but it didn’t work for some reason, so I used NFS instead. I am installing Ubuntu 8.0.10 but it’s awful slow, and this on an Intel i920 processor  with 9GB RAM, 4GB of which is allocated to the ESXi VM. I didn’t expect much but I didn’t expect this. (Note: this was easily fixed, see my comment below)

The only caveat in getting it running was making sure that I selected “Other Linux 64-bit” for the ESXi VM, and editing its .vmx file to add the lines:

monitor.virtual_exec = "hardware"
monitor_control.restrict_backdoor = "true"

Without these, ESXi will not allow you to start any VMs. It’s also important that the NIC be an e1000, likely because that’s one of the few NICs supported by ESX, but that was what I got by default using the above setting.

My intent as mentioned earlier is to play with linked clones. We have a project to create a set of semi-public workstations and I want them to be thin clients (e.g. Thinstation) that connect to a set of VMs running XP such that logging off or disconnecting will revert the VMs to a snapshot. Straightforward enough, but a PITA for maintenance where each VM would require its own large, redundant disk image. Linking each VM to a single image will use minimal disk space, plus when we update that common disk image with new anti-virus defs, Windows updates, etc, rolling out the changes will be as easy as replacing the linked disk. A discussion of how this will work is here, at the bottom of the discussion.