RDP VirtualBox without the proprietary Oracle extension pack

Posted: 11 April 2012 by Robert Hutton in HOWTOs, System Administration
Tags: , , , , , , , , , , , , , , , , ,

These days, virtualisation is all the rage. The various competing virtualisation products have reached a level of maturity where they can be reliably used for server consolidation. VirtualBox is one of the easiest to use, most featureful programs available in this space and with the ability to run on many different OSes on hardware with or without VM extensions, it is also one of the most popular. However, there is one wrinkle when it comes to using it for server consolidation: the proprietary RDP/USB2 extension pack.

The conventional wisdom when running a headless server with VirtualBox is that you need to install this proprietary extension pack from Oracle. This is fine until you want to use the server in production: as the PUEL only covers you for personal use and evaluation, you must purchase licenses. You can either pay £34 per user or £670 per “socket” (which has quite a convoluted definition). This gets you USB2 and RDP support.

However, there is another way, at least when it comes to RDP support. Chapter 7 of the VirtualBox manual covers running virtual machines on a remote host, either with VBoxManage command (which offeres similar functionality to the VirtualBox GUI) or the VBoxHeadless command, which appears to be the backend binary that VBoxManage calls out to to do the actual work. Now, chapter 7 covers in detail how to connect to remote VMs using the proprietary extensions, but there’s one feature that they conveniently failed to mention: the built-in VNC server.

man VBoxHeadless
VBOXHEADLESS(1)            User Commands             VBOXHEADLESS(1)

       VBoxHeadless - x86 virtualization solution

    Oracle  VM  VirtualBox Headless Interface (C) 2008-2011 Oracle
    Corporation All rights reserved.

    -s, -startvm, --startvm <name|uuid>
           Start given VM (required argument)
    -n, --vnc
           Enable the built in VNC server
    -m, --vncport <port>
           TCP port number to use for the VNC server
    -o, --vncpass <pw>
           Set the VNC server password
    -v, -vrde, --vrde on|off|config
           Enable (default) or disable the VRDE  server  or  don't
           change the setting
    -e,  -vrdeproperty,  --vrdeproperty  <name=[value]>
    Set a VRDE property:
           "TCP/Ports" - comma-separated list of  ports  the  VRDE
           server can bind to. Use a dash between two port numbers
           to specify a range "TCP/Address"  -  interface  IP  the
           VRDE server will bind to
    -c, -capture, --capture
           Record the VM screen output to a file
    -w, --width
           Frame width when recording
    -h, --height
           Frame height when recording
    -r, --bitrate
           Recording bit rate when recording
    -f, --filename
           File  name when recording.  The codec used will be cho‐
           sen based on the file extension

VBoxHeadless                January 2011             VBOXHEADLESS(1)

Right! So we can start a virtual machine and forward its root console (or main video or out-of-band console or whatever you want to call it) over the network with a VNC server. In my example I’ll use screen to keep my VMs running when I log out. I’ll start a couple of VMs as an example:

VBoxHeadless --startvm 'Ubuntu' --vnc --vncport 5900
# hit "ctrl-a c" to open a new terminal within screen
VBoxHeadless --startvm 'XP1' --vnc --vncport 5901 --vncpass vnc2xrdp

Great! We don’t need the proprietary expansion pack to do this. However, there are a few drawbacks to this approach:

  • VNC sucks over slow network connections, it’d be much nicer to use the more modern RDP protocol
  • You have to keep the command running so you have to use screen or nohup or equivalent, which is a bit less convenient than using VBoxManage
  • If the VNC server crashes (which I’ve had happen only once so far in testing, when changing screen resolution in a Windows XP guest), the whole VM goes down with it
  • You need to make sure you only bind one VNC server to each port. As far as I can tell if you try to bind a second one to a port that’s already in use, the VM still starts up but you have no way of interacting with it!

We can actually work around the first limitation, by using the xrdp program to “translate” the VNC protocol into RDP. I’m on Ubuntu, so I have the luxury of installing xrdp the easy way. On the VirtualBox server machine:

sudo apt-get install xrdp

Now we configure it to use the existing VNC servers that we previously spawned with VBoxHeadless. One neat thing here is that xrdp uses a single RDP port to manage multiple VNC connections:





So we have two VirtalBox VMs running, Ubuntu and XP1. In the example above, I started the ubuntu VNC without a password, so I’ve left out the username/password entries. The XP1 connection is protected with the password vnc2xrdp. You can also use the value ask and xrdp will prompt for a username/password for connecting to VNC. Note that VNC passwords are generally insecure, so it’d probably be best to protect the vnc ports using firewalling. It doesn’t appear to be possible to bind the VNC server to only the loopback device (at least from the man page above).

So now all that’s left to do is to connect to the RDP port using one of the myriad RDP clients for linux (I’m using Remmina, but there are heaps of options). You can then choose the VNC connection you want xrdp to connect to and you’re away!

xrdp login screen

xrdp login screen

Of course, SSH local port forwarding is your friend if you’re doing any of this through firewalls or over insecure connections. Remmina actually includes this functionality, or you can forward the remote port to your local machine with something like:

ssh -L 3389:localhost:3389 vboxservermachine

So there you are, RDP connections to remote VirtualBox VMs without the proprietary Oracle extensions; all free software!

Have fun. 🙂

  1. Alan says:

    Looks like Oracle caught on…they removed the VNC options from VBoxHeadless in the newer builds. Sigh.

    • You’re correct, though actually what’s going on here is that the Oracle builds don’t have vnc compiled into them by default. This is the case all the way back through version 3.2 (which is as far back as I went in my testing). The code is still there, and needs to be enabled at compile time with VBOX_WITH_VNC:=1 in the LocalConfig.kmk. The Ubuntu builds (and probably the debian ones too) have this enabled by default. More here and here.

  2. Lev says:

    change vnc port/password don’t work in vb 4.

  3. I’m not a fan of the squirrely language in the PUEL, but Oracle seem to be okay with deploying and using this commercially. This from their Licensing FAQ:

    “What exactly do you mean by personal use and academic use in the Personal Use and Evaluation License?

    Personal use is when you install the product on one or more PCs yourself and you make use of it (or even your friend, sister and grandmother). It doesn’t matter whether you just use it for fun or run your multi-million euro business with it. Also, if you install it on your work PC at some large company, this is still personal use. However, if you are an administrator and want to deploy it to the 500 desktops in your company, this would no longer qualify as personal use. Well, you could ask each of your 500 employees to install VirtualBox but don’t you think we deserve some money in this case? We’d even assist you with any issue you might have.

    Use at academic institutions such as schools, colleges and universities by both teachers and students is covered. So in addition to the personal use which is always permitted, academic institutions may also choose to roll out the software in an automated way to make it available to its students and personnel.”

    Sure, it’s not GPL, so they could remove rights to use it at any time, but it appears to be only mass deployment that’s expected to be paid for.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s