These days, virtualisation is all the rage. The various competing virtualisation products have reached a level of maturity where they can be reliably used for server consolidation. VirtualBox is one of the easiest to use, most featureful programs available in this space and with the ability to run on many different OSes on hardware with or without VM extensions, it is also one of the most popular. However, there is one wrinkle when it comes to using it for server consolidation: the proprietary RDP/USB2 extension pack.
The conventional wisdom when running a headless server with VirtualBox is that you need to install this proprietary extension pack from Oracle. This is fine until you want to use the server in production: as the PUEL only covers you for personal use and evaluation, you must purchase licenses. You can either pay £34 per user or £670 per “socket” (which has quite a convoluted definition). This gets you USB2 and RDP support.
However, there is another way, at least when it comes to RDP support. Chapter 7 of the VirtualBox manual covers running virtual machines on a remote host, either with VBoxManage command (which offeres similar functionality to the VirtualBox GUI) or the VBoxHeadless command, which appears to be the backend binary that VBoxManage calls out to to do the actual work. Now, chapter 7 covers in detail how to connect to remote VMs using the proprietary extensions, but there’s one feature that they conveniently failed to mention: the built-in VNC server.
man VBoxHeadless
VBOXHEADLESS(1) User Commands VBOXHEADLESS(1)
NAME
VBoxHeadless - x86 virtualization solution
DESCRIPTION
Oracle VM VirtualBox Headless Interface (C) 2008-2011 Oracle
Corporation All rights reserved.
Usage:
-s, -startvm, --startvm <name|uuid>
Start given VM (required argument)
-n, --vnc
Enable the built in VNC server
-m, --vncport <port>
TCP port number to use for the VNC server
-o, --vncpass <pw>
Set the VNC server password
-v, -vrde, --vrde on|off|config
Enable (default) or disable the VRDE server or don't
change the setting
-e, -vrdeproperty, --vrdeproperty <name=[value]>
Set a VRDE property:
"TCP/Ports" - comma-separated list of ports the VRDE
server can bind to. Use a dash between two port numbers
to specify a range "TCP/Address" - interface IP the
VRDE server will bind to
-c, -capture, --capture
Record the VM screen output to a file
-w, --width
Frame width when recording
-h, --height
Frame height when recording
-r, --bitrate
Recording bit rate when recording
-f, --filename
File name when recording. The codec used will be cho‐
sen based on the file extension
VBoxHeadless January 2011 VBOXHEADLESS(1)
Right! So we can start a virtual machine and forward its root console (or main video or out-of-band console or whatever you want to call it) over the network with a VNC server. In my example I’ll use screen to keep my VMs running when I log out. I’ll start a couple of VMs as an example:
screen VBoxHeadless --startvm 'Ubuntu' --vnc --vncport 5900 # hit "ctrl-a c" to open a new terminal within screen VBoxHeadless --startvm 'XP1' --vnc --vncport 5901 --vncpass vnc2xrdp
Great! We don’t need the proprietary expansion pack to do this. However, there are a few drawbacks to this approach:
- VNC sucks over slow network connections, it’d be much nicer to use the more modern RDP protocol
- You have to keep the command running so you have to use
screenornohupor equivalent, which is a bit less convenient than usingVBoxManage - If the VNC server crashes (which I’ve had happen only once so far in testing, when changing screen resolution in a Windows XP guest), the whole VM goes down with it
- You need to make sure you only bind one VNC server to each port. As far as I can tell if you try to bind a second one to a port that’s already in use, the VM still starts up but you have no way of interacting with it!
We can actually work around the first limitation, by using the xrdp program to “translate” the VNC protocol into RDP. I’m on Ubuntu, so I have the luxury of installing xrdp the easy way. On the VirtualBox server machine:
sudo apt-get install xrdp
Now we configure it to use the existing VNC servers that we previously spawned with VBoxHeadless. One neat thing here is that xrdp uses a single RDP port to manage multiple VNC connections:
/etc/xrdp/xrdp.ini
[globals] bitmap_cache=yes bitmap_compression=yes port=3389 crypt_level=low channel_code=1 [xrdp1] name=VBox-Ubuntu lib=libvnc.so ip=127.0.0.1 port=5900 [xrdp2] name=VBox-XP1 lib=libvnc.so username= password=vnc2xrdp ip=127.0.0.1 port=5901
So we have two VirtalBox VMs running, Ubuntu and XP1. In the example above, I started the ubuntu VNC without a password, so I’ve left out the username/password entries. The XP1 connection is protected with the password vnc2xrdp. You can also use the value ask and xrdp will prompt for a username/password for connecting to VNC. Note that VNC passwords are generally insecure, so it’d probably be best to protect the vnc ports using firewalling. It doesn’t appear to be possible to bind the VNC server to only the loopback device (at least from the man page above).
So now all that’s left to do is to connect to the RDP port using one of the myriad RDP clients for linux (I’m using Remmina, but there are heaps of options). You can then choose the VNC connection you want xrdp to connect to and you’re away!
xrdp login screen
Of course, SSH local port forwarding is your friend if you’re doing any of this through firewalls or over insecure connections. Remmina actually includes this functionality, or you can forward the remote port to your local machine with something like:
ssh -L 3389:localhost:3389 vboxservermachine
So there you are, RDP connections to remote VirtualBox VMs without the proprietary Oracle extensions; all free software!
Have fun. 🙂
malariagen informatics 
Looks like Oracle caught on…they removed the VNC options from VBoxHeadless in the newer builds. Sigh.
You’re correct, though actually what’s going on here is that the Oracle builds don’t have vnc compiled into them by default. This is the case all the way back through version 3.2 (which is as far back as I went in my testing). The code is still there, and needs to be enabled at compile time with VBOX_WITH_VNC:=1 in the LocalConfig.kmk. The Ubuntu builds (and probably the debian ones too) have this enabled by default. More here and here.
change vnc port/password don’t work in vb 4.
I’m not a fan of the squirrely language in the PUEL, but Oracle seem to be okay with deploying and using this commercially. This from their Licensing FAQ:
“What exactly do you mean by personal use and academic use in the Personal Use and Evaluation License?
Personal use is when you install the product on one or more PCs yourself and you make use of it (or even your friend, sister and grandmother). It doesn’t matter whether you just use it for fun or run your multi-million euro business with it. Also, if you install it on your work PC at some large company, this is still personal use. However, if you are an administrator and want to deploy it to the 500 desktops in your company, this would no longer qualify as personal use. Well, you could ask each of your 500 employees to install VirtualBox but don’t you think we deserve some money in this case? We’d even assist you with any issue you might have.
Use at academic institutions such as schools, colleges and universities by both teachers and students is covered. So in addition to the personal use which is always permitted, academic institutions may also choose to roll out the software in an automated way to make it available to its students and personnel.”
Sure, it’s not GPL, so they could remove rights to use it at any time, but it appears to be only mass deployment that’s expected to be paid for.