VMware Workstation VMs Shutting Down Slowly?

I recently noticed that several of my VMware Workstation 10 VMs would take several minutes to power off after their OSes had shut down. The screen would go black until power was finally turned off to the VM.

I finally decided to search for suggestions and found this article which suggested adding the following entries to either individual .vmx files or (for Windows 7) the global config file located at C:ProgramDataVMwareVMware Workstationconfig.ini

prefvmx.minVmMemPct = "100"
mainMem.useNamedFile = "FALSE"
mainMem.partialLazySave = "FALSE"
mainMem.partialLazyRestore = "FALSE"

After restarting Workstation I can confirm that this change works! Thanks to the original author for posting this.

Enable Extra Mouse Buttons in Linux Guest VM (VMware)

It’s long bugged me that I was not able to use the extra mouse buttons (e.g. back) in my Linux VMs under VMware Workstation. Apparently it’s bugged a lot of others too because today after finally reaching my breaking point I did a quick search and found that manually adding a single line to your VM’s .vmx file will enable this missing functionality. Here’s the line, simply append it to your vmx file when your VM is powered down and restart:

mouse.vusb.enable = "TRUE"

I wish I’d hit my breaking point long ago and fixed this… Now my Kensington Expert Mouse is a first class Linux citizen in VMware.

Source of solution: http://goo.gl/zCYj6

Private Internet Access and pfSense

I’ve long been interested in using a VPN to access out-of-country content as well as to secure whatever nefarious activities that I may be up to. Not that I have anything to hide but it’s no one’s business what I do.

I’d toyed with Tor back when I installed Google Voice. It works but is dog slow. My interest in this subject was recently rekindled when I read this article. I found myself drawn to Private Internet Access (PIA) and it’s promises of low cost and unlimited bandwidth coupled with no throttling. I signed up for a monthly plan with the intention of either canceling it if I didn’t like it or switching to a full-year subscription if I did.

Signing up with PIA was easy enough. You can download client software that runs on your individual machines if you wish, but since I have a pfSense firewall, I knew I should be able to channel all traffic through PIA over VPN so I wouldn’t have to install any additional software. Here are the config steps for pfSense 2.0.1.

To start with though, make note of your current IP address as evidenced by http://ipchicken.com. We want to confirm that your IP actually changes when we’re all done.

pfSense Config Files

SSH to your pfSense server and cd to /etc. Create a file “openvpn-password.txt” with two lines, one for your PIA userid, the other for your password.

You also need to download this file from PIA and extract its ca.crt to /etc.

Set 0600 permissions on both of these files e.g. “chmod 0600 /etc/ca.crt”. You can exit SSH at this point.

Certificates

In pfSense’s webConfigurator, go to System and select Cert Manager. Add a new CA, call it something like “Internal CA” using method “Create an internal Certificate Authority”. Fill in the Distinguished Name pieces below as you see fit.

Now click on Certificates and add a new certificate using “Create an internal certificate”. Call it something like “OpenVPN” and select type “Certificate Authority”.

OpenVPN Service

Go to VPN, select OpenVPN and click the Client tab. Add a new client. Leave all defaults except the following:

  • Server host or address: enter your desired PIA host e.g. us-texas.privateinternetaccess.com
  • Check “Infinitely resolve server”
  • Give it a meaningful name e.g. “Private Internet Access OpenVPN”
  • Clear “TLS Authentication” check box
  • Make sure the CA and Cert you created are selected
  • Select “BF-CBC (128-bit)” for the encryption algorithm
  • Check “Compress tunnel packets using the LZO algorithm”
  • Enter the following for Advanced at the bottom:
auth-user-pass /etc/openvpn-password.txt
ca /etc/ca.crt

Click Save to write your config and the OpenVPN service should start. You can click the blue “S” just under the Help menu to confirm that its status is “up”. Also check the log (blue “L”) to make sure there aren’t any errors.

Enable Interface

Go to Interfaces and select (assign). Click the add button. A new entry called OPTn should appear with “ovpnc1” as the port. Click Save. Now you can enable your new interface. Go to Interfaces and select OPTn. Simply click Enable and Save. Note that you can rename the interface if you want to something like “VPN” but it’s not necessary.

Restart the OpenVPN service so everything is in sync. Go to Status and select Services, then click the restart button beside the OpenVPN service. Ensure that the OPTn gateway has an IP. Go to System: Routing and make sure the Gateway has an IP address.

Firewall Config

At this point the OpenVPN service is running but you aren’t using it. You may not even be able to access the Internet in this state. While there’s a lot you can do to tailor your firewall access, here’s a quick way to route all your outgoing traffic through your new VPN connection.

Go to Firewall and select NAT, then click the Outbound tab. Select any existing rules and delete them. Select the “Automatic” option at the top and click Save, then select “Manual” and click Save. You should see a new set of rules which you can activate by clicking Apply Changes.

There’s lots more that could be done to pfSense to tighten up your security but this is a starting point.

Defining Exceptions

One client of mine requires me to log in to their Cisco VPN. Unfortunately this does not work through the VPN connection I just set up. It’s easy to force connections to their VPN server over the WAN interface, bypassing our VPN, by defining a new route as follows:

  • Go to System and select Routing. Click Routes and create a new route.
  • Enter the IP address of the remote host, in this case my client’s VPN IP address. Make sure the WAN gateway is selected and enter an appropriate name.
  • Click Save then Apply Changes.

Attempts to connect to this IP address from any device on your network will bypass the VPN and go directly to that IP address. Too easy!

Test

At this point you should have all your traffic going through the PIA VPN. You can confirm this by refreshing your ipchicken screen which should now show a different IP address.

Final Thoughts

The biggest concern I had with using a VPN like this was the performance penalty. I used http://speedtest.net before and after and there’s certainly a considerable penalty to be paid, but it’s not as bad as I had feared. I have 100-megabit service and without the VPN connection realize a max throughput of about 93 Mbps. Running the same test WITH the VPN enabled cut that in half. Fortunately, 40+ Mbps is still plenty fast for most of my needs. I have to wonder if the fact that my pfSense is running as a virtual machine plays much of a role here since there’s a whole lot of encryption going on. Perhaps I’d be better off using dedicated hardware, but that’s an experiment for another day.

I may give a few other services a try to see if they offer improved throughput. So far though I am impressed with Private Internet Access.

NOTE: These instructions are for pfSense 1.2.3. For 2.X versions, please see the additional requirements in this article, specifically relating to a bug in older versions of pfSense. Please see the first quote there for more information.

VMware ESXi Corrupted iSCSI Datastore

Terrible scare yesterday. I rebooted my ESXi server, which hosts several critical systems, and ESXi didn’t recognize the iSCSI datastore. It could see the iSCSI LUN, but attempts to add it as a new datastore warned that all data would be destroyed. Backups were available but I’d rather have (a) found out what the problem was/why it occurred and (b) fixed the problem. So started my 12-hour learning process.

This all came about because I rebooted the host in an attempt to get new vNICs working under a VLAN. Yes, holidays are a great time to play and learn, it would seem. As a consultant, I should spend even more time doing this sort of thing.

So why couldn’t VMware mount the datastore? Did something happen to it? I tried all manner of fixes, including ultimately reconfiguring my host from scratch to wipe out any traces of the old datastore in the hope that some config was corrupted, but no go. The last resort, it would seem, would be to repartition the iSCSI LUN, which to me seemed a last-gasp effort. Since I was at that stage, I followed the following instructions:

esxcfg-scsidevs -c (take note of the disk device)
fdisk -l /dev/disks/t10.F405E46494C4540096D427739387D25525F4A5D245638787

Hmmm, this didn’t show “fb VMFS” like it should, but rather “SFS”. A quick search told me that this indicated a Windows dynamic disk. Uh oh… Rewinding a bit, a couple of weeks ago a Hyper-V Windows Server of mine had lost its iSCSI connection. The disk was there but it couldn’t access it. I saw that it was marked as, you guessed it, Dynamic! Does that mean everything is toast? I can only guess that ESXi saw the disk as VMFS since it was first created, and continued to access it as such even once Windows had marked it as dynamic. Since the Windows server didn’t really use it, odds are that the two didn’t interfere with each other except for the partition table.

Only one way to find out. I continued with the terrifying process of repartitioning the datastore:

fdisk /dev/disks/t10.F405E46494C4540096D427739387D25525F4A5D245638787
d (deletes the partition: gulp!)
n (create new partition)
p (make primary)
enter (accept default)
enter (accept default again)
t (change partition type)
fb (VMFS)
X (expert mode)
b (change beginning of partition)
1 (first partition)
128 (select secdtor)
W (write changes and exit: double gulp!)
vmkfstools -V (discover the VMFS)

At this point, in vShpere I did a Rescan on the Storage Adapters, and after clicking on Storage, to my amazement, my iSCSI datastore was there! I added my VMs to the inventory and started them up, and all was fine. Very cool.

To finish things off, I disconnected the LUN from that rogue Windows server and removed the LUN from OpenFiler so this can’t happen again. While it’s fine for different ESXi hsots to share a LUN, it’s clearly a bad idea for Windows and ESXi to try and play together…

VMware Workstation 7.1 and Ubuntu 10.10

Ever since my work computer’s hard disk crashed and I reinstalled VMware Workstation 7.1.2 on Ubuntu 10.10, I get an annoying dialog every time I start Workstation telling me that certain modules need to be compiled. The last, VMCI Sockets, always fails with a compiler error. Apparently the kernel in 10.10 breaks the compile. Here’s how to fix it:

wget http://www.debuntu.org/sites/www.debuntu.org/files/vmware-7.1-ubuntu10.10-patch-v2.tar_.gz
tar -xzvf vmware-7.1-ubuntu10.10-patch-v2.tar_.gz
cd vmware-7.1-ubuntu10.10-patch 
sudo sh ./apply_patch.sh

Sweet! Thanks to this site for the instructions.

Asus G73JH-B1: Powerhouse Laptop

Lately I have been eyeing a new laptop. My newest laptop is an Acer netbook which was bought for travelling convenience, and it serves that purpose well, having performed admirably first in Mexico, and most recently in Minneapolis. My older laptops are more powerful but still lacking. I have a Macbook Pro from late 2008 which performs well enough with a dual-core processor and 4GB RAM, but comes nowhere close to my development desktop which boasts an Intel Core i7 processor and 12GB RAM, not to mention reasonably hot graphics and two large monitors. Then there is the older HP Pavillion with a Core Duo processor and then-whopping 2GB RAM. Veritable dinosaurs compared to what’s available today.

My thoughts turned to a new laptop when I read a review of the HP Envy series. It ranked higher than current high-end Macbook Pros, and claimed very high build quality, awesome specs, and light weight. And with available Core i7 processors and four SODIMM slots for a potential 16GB RAM, this was a virtualization enthusiast’s dream machine. Until I tried to find a 17″ version with 1920×1080 screen, that is. This configuration doesn’t seem to exist in Canada, maxxing out at 1600×900 resolution. Worse, reviews were generally poor, with myriad unresolved issues. Hard to believe considering this is HP’s high-end laptop series.

I looked at many other types of laptops, including Alienware (nice, but $2500?!), Toshiba Qosmio (10 lbs?!), and Dell Studio XPS (also costly, and maxxing out at 8GB RAM). I somehow stumbled upon Asus’ G73 series which seemed a perfect fit. It has the screen resolution I wanted, 4 SODIMM slots, Core i7 processor, dual 7200 RPM drives, was reasonably priced and at 7lbs was of acceptable mass, plus it looked great, with a Stealth Bomber appearance: matte grey finish, angular edges, quite nice actually. I immediately dismissed it due to poor reviews which caused keyboard lockups, video crashing, and other system instability, but as my search hit dead ends with other vendors, I kept coming back to the Asus.

Then during another bout of laptop-obsession, I stumbled upon a thread that described a new BIOS version for the G73 series, that being v209. People seemed to hail this as a breakthrough that made their systems live up to their potential at long last. Indeed, even people who had RMA’d their units were left wishing that they hadn’t. After some reading, my mind was made up.

Best Buy had the G73JH-A1 (Intel Core i7 720 with BluRay reader) for $1500, but alas it was some special BB-only derivative that had “HD+” (900) instead of “Full HD” (1080) like I wanted. The best local price was Memory Express for $1700. I did some price-comparison because they claim to match any online or local reseller’s prices, and in doing so I stumbled across the higher-end model, that being the G73JH-B1, which is like the -A1 except it has a faster Core i7 740 processor. Now there’s probably not a huge difference between the two, but I found a -B1 online for essentially the same price as the -A1, and after Memory Express agreed to match that price and confirmed that they had a -B1 in stock, I went down today and bought one.

Words can’t describe how nice this machine is. The backlit keyboard is beautiful and functional, the display is gorgeous, and its speed is just as quick as my high-powered desktop. The fans run virtually silent and the unit is cool to the touch. While it’s not nearly as thin and compact as the Envy that I was first drawn to, I think it will be a lot more of a fit for me as I look to take my work on the road (or to a different part of my house!) And though I have only had it operational for a few hours, the machine seems stable enough, probably due to its v211 BIOS (the latest and greatest.)

While the machine is replete with crapware, I think I will resist the urge to immediately reinstall fresh. There are some very nice utilities installed, not the least of which is facial recognition automatic authentication. I will give this Windows 7 Home Premium 64-bit a fair shake before deciding what to do.

At some point I will also install the two 4GB SODIMMs which will increase my memory from 8GB to 12GB, bringing it on par with my desktop. Except this one I can take anywhere.

VMware ESXi: at home!

A co-worker was making me jealous the other day about how he built an ESXi whitebox, and I got to thinking that I needed something like this myself to host my company’s servers. I have been an avid virtualization junkie ever since the original Virtual PC was made available to me in an old MSDN subscription, and this obsession continues to this day. Currently I use VMware Workstation 7 on my high-powered (Core i7, 9GB RAM) but wholly underutilized HTPC, and while it works well enough, it’s really not very “enterprisey”. Neither is a homebuilt ESXi server, but I can certainly make it pretty close, and it would be far superior to the HTPC which to my horror people often shut down when they are done watching something.

I have a Dell Inspiron 845 that was used by an employee for a past project. It’s a reasonably powerful machine with a quad-core, VT-enabled Intel processor and 8GB RAM, so I figured it would do the trick. According to http://vm-help.com, by simply adding an Intel 1000 GT or CT NIC, ESXi 4.0 will install without any modifications or funky drivers. I picked up a couple of these NICs for $45 each, installed one (the PCI-e CT version) in the 845, and within minutes I had my own ESXi server. Sweet! The only gotcha is that my Windows 7 host can’t run the vSphere management tool, so I need to run it under an XP VM. Oh, the irony!

Next up was to use some enterprisey storage. I have an OpenFiler server sitting in my wiring closet with a 400GB iSCSI volume that’s sitting idle, so after some frustration getting ESXi to see the iSCSI target, I now have what should be a very robust data store for my VMs. I’ll make a post later about exactly what you need to do to get this configured.

I installed the VMware standalone converter utility and migrated my VMware Workstation VMs, initially a development Oracle server and Redmine plus an (*ahem*) bittorrent server, to the new server in its iSCSI data store. Everything went exactly as I’d hoped it would, very smooth. I only needed to reset some static DHCP mappings due to MAC address changes.

I still needed a backup solution though, and last night I got one working. Briefly, it’s the ghettoVCB script which is highly regarded, and I can see why. There are some nice guides on how to get it set up, and I’ll post more on this later. Here’s hoping that tonight’s daily backup works!

VMware Workstation 7 Arrives

Gotta get me this. The new version’s licensing lets you install on either a Linux or Windows host OS, but still only one at a time. Still, it’s nice to have the option of changing if you wish.

I suspected this was coming. About two months ago, VMware sent out an email blast offering Workstation 6.5 for half price which worked out to roughly $100 at the time. I passed on this because I didn’t want to buy it only to get stung with a new version that I’d immediately have to upgrade to just a short while later.

The new version is $190 and the price for upgrading is $100, so I guess I’ll have ended up saving $10 this way.

VMware Cloning and Debian/Ubuntu udev Rules

When cloning Ubuntu 8.04 VMs under VMware, the /etc/udev/rules.d/70-persistent-net.rules file would get an entry for eth1 pointing at the newly-assigned NIC MAC address, leaving the original eth0 entry in place. This would cause networking to not work (i.e. “we’re having notwork problems haha”) and we’d have to manually edit this file, remove the old eth0 entry, and rename eth1 to eth0. PITA!

When we switched to Xen during our brief trial, this wasn’t necessary. We figured the Xen templating process had some smarts. Now that I’m playing with ESXi again, this issue reared its ugly head again. It turns out the file 75-persistent-net-generator.rules has an entry to exclude Xen NICs from getting an entry in the 70-persistent-net.rules file. Sweet! Adding the following line after ENV{MATCHADDR} is assigned causes the same effect:

ENV{MATCHADDR}=="00:0c:29:*|00:50:56:*", ENV{MATCHADDR}=""

It would be nice to use the same exclusion rule as the Xen entry, but substituting “vmware” for “xen” didn’t do the trick, and there’s little point wasting time trying to find out what the right string would be.