As we’re starting to get into cloud technology in a big way here, I decided to take part in Ubuntu’s IRC training days on the subject, Ubuntu Cloud Days. While the material covered was often more low-level than what we are likely to use in the short term, I found the sessions on Ensemble and CloudInit to be particularly useful, as they’ll be directly applicable to our upcoming GWAS work on the ec2 cloud.
- Day 1
- Getting started with Ensemble
- Introduction to CloudInit
- Orchestra and Ensemble (part 1)
- Orchestra and Ensemble (part2)
- Eucalyptus 3: cloud HA and ID management
- Day 2
Day 1
Getting started with Ensemble
Summary: Ensemble is a cloud orchestration framework. It allows you to treat services as pluggable units; it abstracts away the configuration details that would normally be required to connect them together into functioning systems. Additional units can be added at any time to handle increased load; they simply appear and configure themselves into a load-balancing configuration. Services are added to ensemble using “formulas” which specify how they will connect with other services. Formulas can be written in any language.
Here are the bullet point notes I took during the session.
- works at the “service” level, e.g.:
- mysql
- memcached cluster
- Munin – monitoring service
- Bacula – backup service
- Encapsulates these services
- deploy
- “relate” to other services
- Services are “composable”
- well defined interfaces: allows relating many services together. Services become “switchable”.
- Example commands
- ensemble deploy –repository=. mysql mydb
- . = directory containing the formula
- mysql = formula
- mydb = name of the resulting database
- ensemble deploy –repository=. drupal mywebsite
- ensemble status
- shows bootstrap node, mysql and drupal instances
- relations field is empty: services are uncoupled
- ensemble add-relation mydb:db mywebsite
- mysql creates DB
- “sends over” details – username, password, dbname etc to machine running drupal
- drupal rewrites config to use DB, then creates tables
- ensemble status now shows “relations” field which shows how the services are connected
- Now, if your site is slashdotted, you can add capacity to the site:
- ensemble add-unit mywebsite
- mywebsite = drupal, already configured
- this is DRY: don’t repeat yourself
- ensemble status
- now shows mywebsite/0 and mywebsite/1, two service unit instances
- mysql forumula supports adding “slave” nodes, so we can scale the DB using slave nodes as well.
- formulas can be written in any language
- drupal formula uses “drush”
- formulas are available at http://code.launchpad.net/principia
- a more integrated version is coming soon which will allow search and install of formulas like ppas.
- IRC: #ubuntu-ensemble
- goal is to cover all free software with formulas
Introduction to CloudInit
Summary: CloudInit allows customisation of instances during the bootstrap process. It works on ec2, openstack, eucalyptus. Many things can be configured during bootstrap, e.g. timezone, apt, hostname, ssh, mount, etc.
CloudInit wiki page.
- CloudInit allows customisation of instances during the bootstrap process
- it can perform operations on the instance pre-boot. e.g:
- packages: -apache2: installs apache2 pre-boot
- works on ec2, openstack, eucalyptus
- e.g.:
- timezone: US/Eastern
- apt_update: false
- apt_mirror: URL – custom mirror installed pre-boot
- bootcmd: run command on boot
- debconf-selections
- setting hostname
- generate ssh private keys
- adding ssh keys to user’s .ssh/authorized_keys so they can log in
- setting up ephemeral mount points
- to execute a command (runcmd:)
- configure automatic package update and upgrade (apt_update and apt_upgrade)
- example file: http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/doc/examples/cloud-config.txt
- can add custom repositories – will save bandwidth in a data centre
- byobu_by_default: system – enables byobu to all the uses by default once they login
- more info and detailed instruction at kim0 blog http://foss-boss.blogspot.com/search/label/cloud-init
- cloud-init comes pre installed if you are using ec2, and is available on all cloud environments (openstack, eucalyptus, ec2). right now ec2 is the most mature provider .. work is being done to get it to deploy on openstack, rackspace, eucalyptus, LXC containers ..etc. in case of openstack you need to install the package at time of preparing your cloud image. you can use euca tools in case of eucalyptus and openstack. on ec2 you can use web interface as well as via command line.
Orchestra and Ensemble (part 1)
Summary: There are two new deployment technologies that the Ubuntu team have been working on, ensemble and orchestra. Ensemble is for deploying services on existing ubuntu installs or on cloud instances, Ochestra is for deploying onto bare metal. This talk covers work that has been done to make Orchestra a provider for Ensemble, allowing Ensemble to deoploy to and manage bare metal the same way it currently does with EC2. So they used cobbler, qemu and libvirt, tied together with a script called cobbler-devenv. This is similar to the setup that I have here for testing the service migration, but I have used the (much easier to set up) VirtualBox.
Much of the Server Team’s focus this cycle has been on deployment. That deployment really falls into 2 different categories:
- ensemble: deploying and managing services on top of existing Ubuntu installs (or new cloud instances)
- orchestra: deploying Ubuntu onto “bare metal”.
A few weeks ago, it was decided that we wanted to make Orchestra a “provider” for Ensemble. What this means is that we wanted to allow Ensemble to deploy and manage “bare metal” machines the same way that it originally knew how to manage EC2 instances. Andres [RoAkSoAx] will talk more about that in the next session. Like anybody else, we don’t have enough hardware, and even less hardware with remotely controllable power switches and fast networks. In order to get ourselves an environment that we could develop the “orchestra provider” for ensemble I put together “cobbler-devenv”. That can be found at http://bazaar.launchpad.net/~smoser/+junk/cobbler-devenv/files/head:/cobbler-server, or via ‘bzr branch lp:~smoser/+junk/cobbler-devenv’. cobbler-devenv allows you to very easily set up a cobbler development environment using libvirt. That environment:
- includes an Orchestra server and 3 “nodes”
- includes a dhcp server and dns server
- will not interfere with physical networks *or* other libvirt networks.
The code there is currently focused on deploying cobbler and an ensemble provisioning environment, but it not much is really specific to that purpose. If you’ve not already done so, go ahead and open the cobber-server url above or branch it. The HOWTO explains how to set all this up. I’ll largely walk through that here with some more explanation as to what is going on than is in that file.
== Some configuration first ==
as prereqs, you’ll need to $ apt-get install genisoimage libvirt-bin qemu-kvm
In order to interact with libvirt, you have to be in the libvirtd group, and in order to use kvm acceleration you have to be in ‘kvm’ group. So:
$ sudo adduser $USER kvm $ sudo adduser $USER libvirtd $ sudo apt-get install python-libvirt
== building the Orchestra server VM ==
Also, note that libvirt does not work when images are in a private home directory. The images must be viewable to the libvirt user.
this cost me a fair amount of time once trying to debug why my VMs were getting “permission denied” when they were clearly readable (but the path to the images was not)
the first step in the HOWTO document is to build a cobbler server. To do that, we utilize build-image like:
$ ./build-image -vv –preseed preseed.cfg oneiric amd64 8G
Note, the above command won’t actually work right now. 😦 bug 815962 means that that doesn’t currently work, and wont until the next upload of debian-installer. You can have it use a mirror by editing preseed.cfg. It wraps all the following:
- grab the current mini-iso for oneiric
- extract the kernel and ramdisk using isoinfo
- repack the ramdisk so it has ‘preseed.cfg’ inside it, and set up the ‘late_command’ in installer to do some custom configuration for us (see ‘late_command.sh’).
- after install is done, boot the system again to do some final config that ‘late_command’ layed down.
It does this via kvm and the kvm user net, so you can build this entirely without libvirt or root access. I’m particularly proud of not needing root for this. or any network access other than to the archive. This basic setup could be used for automated building of virtual machines (as it is here). The result is that you now have a disk image that is ready to boot. We’ve built the Orchestra virtual server that will be in charge of provisioning the nodes.
$ ls -lh natty-amd64.img -rw-r--r-- 1 libvirt-qemu kvm 1.3G 2011-07-25 12:39 natty-amd64.img
Now we just we need to set up a libvirt network, and put that image on it.
m_3 asked: so ‘./build-image -vv –preseed preseed.cfg natty amd64 8G’ should work, but oneiric won’t?
build-image with ‘natty’ “should work”. i verified the install went fine, but ran into bug https://launchpad.net/bugs/804267 that caused me to not be 100% tested that path today.
TeTeT asked: so if it’s for a virtual environment, this means a non cloud environment, as otherwise installing OS is a non-issue, at least with euca and openstack?
TeTeT, right. it is for a virtual environment, and “non-cloud”
QUESTION: What would it take to install real physical boxes out of that dev-env
the initial reason I developed this was to ease the development of the “orchestra provider” for ensemble through that provider, ensemble will be able to install “bare metal” systems. we’re just creating a virtual network that would be like a physical network and sytems you would have access to, but its easier to work with the virtual. the primary goal of “bare metal provisioning” for ensemble, is actually to provision a cloud. to install real machines off of the cobbler vm, you’d have to set bridging up differnetly than i have it, and have your dhcp server point next-server to the cobbler system.
== Setting up Libvirt resources ==
Now, back at the top level directory of cobbler-devenv we have a ‘settings.cfg’ file [http://bazaar.launchpad.net/~smoser/+junk/cobbler-devenv/view/head:/settings.cfg]. The goal is that this file defines all of our network settings. It has sections for ‘network’, ‘systems’ (static systems like the Orchestra Server) and ‘nodes’. the only static system we have is ‘cobbler’, but there could be more described there. We create the libvirt resources by running ‘./setup.py’ (which should probably be renamed to something that does not look like it came from python-distutils). that script interacts with libvirt via python bindings.
$ ./setup.py libvirt-setup
That will put some output to the screen indicating that it created a ‘cobbler-devnet’ network, a ‘cobbler’ domain, and 3 nodes named ‘node01’ – ‘node03’. The libvirt xml is based on the libvirt-domain.tmpl and libvirt-network.tmpl files, which are parsed as Cheetah template files. The end result is that we have a ‘cobbler-devnet’ network at 192.168.123.1, and has statically configured dhcp entries for our cobbler server and 3 nodes, so that when they DHCP they’ll get set IP addresses. the cobbler-devnet network looks something like:
http://paste.ubuntu.com/651906/
notice how we have MAC addresses in the network setup that will match with our mac addresses in the nodes. now our network is setup, so lets put the cobbler server on it. We build a qcow “delta” image off of the pristine server image we built above so we can easily start fresh.
$ virsh -c qemu:///system net-start cobbler-devnet Network cobbler-devnet started $ qemu-img create -f qcow2 -b cobbler-server/natty-amd64.img cobbler-disk0.img $ virsh -c qemu:///system start cobbler Domain cobbler started
That will take some time to boot, but after a few minutes you should be able to ssh to the cobbler system using its IP address:
$ ssh ubuntu@192.168.123.2
the password is ‘ubuntu’, obviously you should change that). While you’re there, you can verify that ‘cobbler’ works by running:
$ sudo cobbler list
that should show you that there were some images imported for network install of Ubuntu. At this point You can also get to the web_ui of cobbler at: http://192.168.123.2/cobbler_web and poke around there. generally, we’ve got a fully functional cobbler server just waiting for something to install! Then, back on the host system we populate the cobbler server with the 3 nodes that we’ve created.
$ ./setup.py cobbler-setup
That uses the cobbler xmlrpc api to set up our nodes. Now, a ‘cobbler list’ will show our nodes. It also configures those nodes to be controllable by the “virsh” power control (that is like ipmi, but for virtual machines). We’ve got one more thing to do though before that can happen. On the host system we need to run:
$ socat -d -d \ TCP4-LISTEN:65001,bind=192.168.123.1,range=192.168.123.2/32,fork \ UNIX-CONNECT:/var/run/libvirt/libvirt-sock
socat is a useful utility, and the above command tells it to listen for ip connections on port 65001 and forward those to the unix socket that libvirt listens on. basically this makes libvirtd listen on a tcp socket. Before you go screaming how horrible that is (it would be). notice that We’ve limited the host IP to the IP range of the guest network, and told it to only listen on the guest’s interface, so it is mildly secure. Definitely much better than just listening on all interfaces. Once that is in place, you can turn the nodes on and off via the cobbler web_ui. Basically, at this point, we have modeled a lab with IPMI power control of node systems from the cobbler system. nodes can be turned on and off, and their network boot controlled via the cobbler vm system. I should have pointed out above, that our libvirt xml for the Node systems has them network booting. If you configure ‘network-boot’ for a node, and then start it, it should begin to install itself. You can try that out, and then (from the host system) watch the install with:
$ vncviewer $(virsh vncdisplay node01)
It should actually walk through a fully automated install. questions? Well, thats basically all I have.
=== Summary ===
after all of that, what we have is a well configured network with a single cobbler server that is ready to install some nodes. The nodes actually have functional static-dhcp addresses and can communicate with one another via hostnames (node01, cobbler, node02…) In the next session, Andreas will talk about how we can use ensemble to control the cobbler server and provision the nodes.That way, ensemble can control our bare metal servers just like it can request new EC2 nodes. (here, we’re just pretending that those VMs are real hardware, but ensemble doesn’t actually know the difference) so… if you want to just play with cobbler some, this is a really nice way to see how it fits all together. without having 2 or 3 spare systems sitting around.
Orchestra and Ensemble (part 2)
Summary: this session goes into more detail on how Orchestra will work under the hood. Interesting, but probably not directly useful to us here at MalariaGEN.
Hi all. My name is Andres Rodriguez, and I’m an Ubuntu Developer working on the Ubuntu Server Team as well. As Scott already mentioned today, we have been working on getting Ensemble to work with Orchestra. We’ve been using smoser’s devenv to achieve this result. Today I’m going to show you how this work can be tested as a proof of concept, as this is still work in progress. But first, lets learn a little bit more about the idea behind Orchestra and Ensemble integration. The main idea behind this was to basically use Ensemble with Orchestra/Cobbler the same way it’s been used with ec2. However, on ec2 we can request instances easily and add more and more, but in Orchestra/Cobbler we can’t. This is a limitation, however the pproach taken in this case, was to simply pre-populate the Orchestra server with “systems” (in terms of Cobbler). A system is a physical system that is somewhere in the network and that cobbler can deploy. So, we have a list of available machine ready to be deployed via PXE.
So we could say that we will have to do two things with ensemble 1. Bootstrap and 2. Deploy, in the same way we would do with ec2. Bootstrapping is when we tell ensemble to start creating the whole environment. In this case, bootstrapping means starting a machine to be the zookeeper machine, which will interface between a client machine from where we are issuing commands, and the provider (Orchestra), to deploy machine and create the relations between them. The process here was to simply select a “system” provided by Orchestra/Cobbler. This system will then be powered on by interfacing with the power management interface the hardware has configured (IPMI, WoL, ect, virsh), and will turn it on. When this machine boots up, it will find a PXE server on the network (Cobbler) and will start the installation process. Once the machine has finished installing, it will use cloud-init to install ensemble.
In case of the development environment, we use virsh as the power management interface. As smoser already explained, the cobbler devenv provides machines that are ready to be deployed via PXE
when we bootstrap with ensemble, it simply tells cobbler to start a machine. cobbler uses virsh to start it, and when the machine starts it searches for a PXE server, and installs the OS. So, as mentioned, the bootstrap process will start a new machine that we are gonna call the zookeeper. Once the zookeeper is up and running, we can start deploying machines.
So, when deploying, Ensemble will tell the zookeeper to deploy a machine with an specific service. The zookeeper will talk to the orchestra server in the same way it did when bootstrapping and will deploy a machine. It will also use cloud-init to install everything necessary to deploy the service. Now, since obviously ec2 is different from Orchestra/Cobbler we needed to make some changes in the approch taken to make things work (such as provide the meta-data for cloud-init). We needed a few things:
- Provide methods in ensemble to interface with Cobbler using its API
- Provide a custom preseed to be used when deploying machines through ensemble.
- Provide a method to pass cloud-init meta-data, and be populated before first boot so that cloud-init can do its thing.
So, how did we achieve this?
- As already explained, ensemble uses cobbler as a provider communicating with it via the cobbler API.
- Since ec2 instances a VM really quick, it was easy to pass all the necessary values through cloud-init, however, in our case, we needed to do somthing similar, and the conclusion was to do it via a modified preseed to deploy whatever it was needed the same way
- We figured out a method to pass the cloud-init meta-data through the preseed
So basically the changes in Cobbler were to provide a custom preseed to deploy the OS. This preseed contains what we call a late_command. This late_command will execute a script that will generate the cloud-init meta-data so after first boot, cloud-init will do its thing. So what we did is to generate the cloud-init meta-data with ensemble as it was always done, but, we had to figure out how to do it to the preseend. Here, we generated text that was later enconded in base64. This text was basically a shell script containing the information to populate cloud-init’s meta-data. So the late command in reality was to decode the base64 code and then, wirte the script and execute it. This decoding and writing was done by the preseed, right after finishing installing the system and before booting
so when the machine restarted, cloud-init would do its thing. So that was done by making ensemble interface with cobbler, and once the late command was generated, ensemble told cobbler “This is your late command” and cobbler simply executed it. Once the machine finished installing, we had a fully functional zookeeper (or service). So basically, we wated to achive the same as with ec2, but we just had to figure out how to do it with the preseed and now, it works in a very similar way.
So the only things to consider were to
- start a machine.
- deploy the machine using the preseed.
- ensure to pass the late_command.
And this way we would simulate the way how instnaces and cloud-init data is passed to instances in the cloud. Other than that, ensemble works pretty much exactly the same as it would with ec2, but using orchestra. Now, another change that was done is that ensemble when working on ec2, it used S3 to store some information that was using by ensemble to identify machines and place the formula meta-data. Instead, we used a WebDav service with the apache2 servcer installed by cobbler. Here, instead of obtaining and storing files on S3, we use the Orchestra server as storage for ensemble. Based on those considerations, pretty much had to ensure that the interaction between the cobbler API and ensemble provided results the way its done with ec2. So how can we really test this with the development environmentbut before?
With smoser’s cobbler devenv we can certainly simulate a physical deployment using ensemble. The good thing is that the devenv will setup everything necessary from the orchestra side of things
but, I’ll give and overview of what will orchestra do very soon. 1st. We would need to install orchestra-server, which will isntall cobbler and cobbler-web with that, we would need to configure the webdav so that we have storage up and running (remember, this is already done by the cobbler-devenv). how did we do this:
=== Setting up file storage ===
1. Enable Webdav
sudo a2enmod dav sudo a2enmod dav_fs
2. Write config file (/etc/apache2/conf.d/dav.conf)
Alias /webdav /var/lib/webdav <Directory /var/lib/webdav> Order allow,deny allow from all Dav On </Directory>
3. Create formulas directory:
sudo mkdir -p /var/lib/webdav/formulas chown www-data:www-data /var/lib/webdav sudo service apache2 restart
Now, we need to pre-populate cobbler with all the avialable systems and provide it with a power management interface to be able to start a physical mnachine. As previously explained, cobbler devenv uses virsh to simulate this behaviour. however, in cobbler, we needed to know two things
- How do we know when a system is available
- How do we know when the system has already been used and no longer available
For this, we had to look into cobbler’s management classes concepts, in this case we are using two, foo-available and foo-acquired. As the name says, one will be used to identify when a system is available to be used by ensemble, and the other one when the system has already been acquired by ensemble and might be in the process of bootstrapping or deploying a service, or even installing the OS
but, in cobbler terms, how can we add management classes and systems? Simple:
=== Setting up cobbler ===
1. Add management classes
sudo cobbler mgmtclass add --name=foo-available sudo cobbler mgmtclass add --name=foo-acquired 2. Add systems sudo system add --name=XYZ --profile=XYZ --mgmt-classes=foo-available --mac-address=AA:BB:CC:DD:EE:FF
Basically, a system is a definition for a physical machine using a OS profile, and what mangement class to use at first. The mprofile is no other than the OS that will be installed in that machine and the management class has already been explained. Of course you will have to configure the power management interface accordingly, but in the cobbler-devenv has alreayd been done. So basically, we now have a Orchestra/Cobbler server up and running and we have configured it with systems, mgmtclasses and the file store. storage* So it is time for us to install and configure ensemble to use our cobbler server. In this case, we are going to use the cobbler-devenv, however, you will notice that you can simply chnage it to be used by physical machines. If you already have an orchestra server up and running and preloaded with systems.
So first, we need to obtain the branch of ensemble that has orchestra support. NOTE: This branch contains code that is under development and is still buggy.
1. Obtain the branch:
bzr branch lp:~ensemble/ensemble/bootstrap-cobbler cd bootstrap-cobbler
now we need to create an environments.yaml file for ensemble we do this as follows:
2. Create the environments file (~/.ensemble/environments.yaml) environments:
orchestra: type: orchestra orchestra-server: 192.168.123.2 orchestra-user: cobbler orchestra-pass: cobbler admin-secret: foooo ensemble-branch: lp:~ensemble/ensemble/bootstrap-cobbler acquired-mgmt-class: foo-acquired available-mgmt-class: foo-available
Note that I’m already using the values for the cobbler-devenv such as orchestra-server IP address, user/pass for cobbler, the branch we need, and the management classes. So once this is done, and we have setup the cobbler-devenv correctly, we can start bootstrapping the zookeeper and then deploying the machines. So the first step, and from the branch we have obtained, we do the follwoing:
PYTHONPATH=`pwd` ./bin/ensemble bootstrap
This will bootstrap the zookeeper, it will take time for it to install and deploy the zookeeper running, it would probbaly take several minutes, so I will containue explaining what it needs to be done. So, when the zookeeper is up and running and cloud init has done its thing, we need to workaround something given that we just came into an error in the code. That is being examined but it is simple and doens’t actually affect the code. So we need to connect to the zookeeper machine (through ssh, or any ither method) and sudo the following (in the zookeeper machine)
sudo -i ssh-keygen -t rsa
This will create public keys that are verified by the zookeeper before deploying machines however, note that this is a work around and will be fixed soon. I’m just pointing you guys to it in case you want to test it after the session of today. Once this is done, we can start deploying machine, and we simply do the following:
PYTHONPATH=`pwd` ./bin/ensemble deploy --repository=examples mysql
This will tell zookeeper to deploy a machine, whcih will tell cobbler to start a machine via virsh, and once installed it will run late-command and populate cloud-init meta-data. On first boot cloud-init will do its thing and baaam we would have a mysql server working on a physical node. And I believe that’s all I have for you today. I think i run over the session too fast 🙂 Anyone has any questions?
< m_3> QUESTION: reboots… how robust is everthing wrt reboots? (In EC2-ensemble, we just typically throw instances away)
Well that’s indeed a limitation we have in comparison to ec2 as in physical environments (and cobbler) we are relying in the power management interface to deploy machines
< ClassBot> m_3 asked: does the cobbler instance provide a metadata server for cloud-init? bootstrap/zk node able to be rebooted without losing ensemble state?
< ClassBot> m_3 asked: reboots… how robust is everthing wrt reboots? (In EC2-ensemble, we just typically throw instances away)
now, as far as rebooting machines and keep things persistant, at the moment, we are not handling that but the first approach was to preseed all that information and use debconf populate those values and have upstart scripts initialize the services on reboot however, we discussed the possibility of actually not doing that through the preseed but rather, provide cloud-init with a command to write those persistant values so on reboot they can be used
Anyone any more questions? alright I guess there’s not. Thank you all
< ClassBot> alexm asked: RoAkSoAx: will ensemble/orchestra be in ubuntu-server manual for oneiric? a quick start guide, for instance
I surely hope so! I guess that will depend how far we can get with this in the development cycle, but I’m confident it would.
Eucalyptus 3: cloud HA and ID management
Summary: This session goes into the underlying infrastructure of a Eucalyptus cloud, how you can achieve high availability with commodity hardware and how ec2-style ID management works. It also goes into quite a lot of implementation detail.
Hello all, and thank you very much for attending this session! Today, we’re going to be discussing some new features of Eucalyptus 3. While there are quite a few, two of the most substantial are implementations of high availability and user/group identity management. We’ll start with a discussion of Eucalyptus HA, and then switch to ID management next.
Eucalyptus is designed as a collection of services which, when stitched together, form a distributed system that provides infrastructure as a service. Roughly, eucalyptus services are organized in a tree hierarchy. At the top of the tree, we have components (Cloud Controller, Walrus) that are directly accessed by users. In the middle, we have a Cluster Controller and Storage Controller which set up/manage virtual networks and storage (EBS) respectively. And and the bottom of the tree, we have Node Controllers which control and manage virtual machines.
In a nutshell, this collection of services provide users the ability to provision and control virtual infrastructure components that, within eucalyptus, we refer to as ‘artifacts’. For example, virtual machines, virtual networks, and cloud storage abstractions (EBS volumes and S3 buckets/objects). The design of Eucalyptus HA creates a distinction between the cloud service itself (eucalyptus components), and the artifacts that are created/managed by the service. The reason for this distinction is that, while the term ‘High Availability’ is generally meaningful. The requirements of making something ‘Highly Available’ varies greatly, depending on what that ‘something’ is.
In Eucalyptus 3, we have a new architecture that provides High Availability for the cloud service itself. The architecture additionally supports adding High Availability to eucalyptus artifacts, in the future
So, the core design of Eucalyptus HA is as follows: Each Eucalyptus component can run in ‘non-HA’ mode, exactly as it does today. Then, at runtime, each component service can be made highly available by adding an additional running version of the component, ideally on a separate physical system. This results in a basic ‘Master/Slave’ or ‘Primary/Secondary’ mode of operation, where the Eucalyptus HA deployment is resilient to (at least) a single point of failure (for example, machine failure). At any point in time, when running in HA mode, a component is either in ‘Primary’ or ‘Secondary’ mode. Any component in ‘Secondary’ mode is running, but is inactive until it is made Primary. Next, each component, and the system as a whole, is designed to keep ‘ground truth’ about artifacts as close to the artifacts as possible.
For example, all canonical information about virtual machine instances is stored on the node controller that is managing that VM and all canonical information about virtual networks that are active is stored with the Cluster Controller that is managing that network. When a eucalyptus component becomes active — which happens when the component first arrives, when it is ‘restarted’ or, when it is promoted from Secondary to Primary — the component ‘learns’ the current state of the system by discovering what it needs from ground truth. Other services that are ‘far’ from ground truth, then, learn about ground truth from nearer components.
I’ll use the Cluster Controller to illustrate how this design works as an example. When a cluster controller enters into a running eucalyptus deployment, there are typically many artifacts that are currently running. The very first operation that a cluster controller performs is to poll both above (Cloud Controller) and below (Node Controllers) in order to learn about the current state of all artifacts. It then uses this information to dynamically (re)create all virtual networks that need to be present in order for the currently active artifacts to continue functioning. So, whether a cluster controller is by itself (non-HA mode) and just reboots, or if a Primary cluster controller has failed and the secondary is being promoted. The operation is the same: learn about ground truth and re-create a functional environment.
All other HA eucalyptus components operate in a similar fashion, semantically. Storage controller uses iSCSI volumes as ground truth. Walrus uses shared filesystem, or a pre-configured DRBD setup for buckets/objects. Finally, while the design of the software permits a simple ‘no single point of failure’ setup with just additional physical machines (to support Primary/Secondary model). We also support deployments that have redundancy in the network infrastructure. This way, ‘no single point of failure’ can be extended to include network failures, as well, without having to alter the software/software configuration.
We’ve put a lot of effort into the new architecture to provide service high availability first, and hope that others will find the architecture ready to start adding HA for specific artifacts in near future releases. Utilizing live migration for VM HA, utilizing HA SAN techniques for in-use EBS volume access HA, etc.
This brings us to the end of the first part of our discussion, thank you very much! I would like to ask if there are any questions about Eucalyptus HA ?
Okay ; the second part here will be led by Ye Wen, who will be talking about the new user and group management functionality in Eucalyptus 3
Hello, everyone. I’m going to continue this topic by discussing another new feature in Eucalyptus 3: the user identity management.
We have a completely new design for managing user identities in Eucalyptus 3, based on the concept of Amazon AWS IAM (Identity and Access Management). In another word, we provide the same API as Amazon AWS IAM. Your existing scripts working for Amazon should be compatible with your new Eucalyptus 3 cloud. At the same time, we augment and extend IAM with some Eucalyptus-specific features, to meet the need of some customers. With IAM, you essentially partition the access to your resources (i.e. the artifacts as Dan said earlier) into “accounts”. Each account is a separate name space for user identities. Account is also the unit for resource usage accounting. Within an account, you can manage a set of users.
Users can also be organized into groups. Note that group is a concept for assigning access permissions to a set of users. So users can be in multiple groups. But users can be only in one account. Permissions can be assigned to users and groups to control their access to the system resources.
As in IAM, you write a policy file to grant permissions. We have a few extensions to the IAM concepts. I talk about a few here. In IAM, you can’t specify EC2 resources. For example, you can only say “allow user A to launch instance”, but you can’t say “allow user A to launch instance using image X”. We introduce the EC2 resources, so that you can do such things. One good use is to restrict the VM types for some users can launch instance with. Another extension is the introduction of VM expiration or lifetime. You can use an Eucalyptus-specific policy condition to specify a VM’s lifetime or when to expire.
The biggest extension probably is the introduction of resource quota. We extend the IAM policy syntax to allow the specification of resource quota. We use a special “Effect” to do that. So you can say “Effect: Limit” in a policy, which indicates the permission is a quota permission. And then you can use the policy “Resource” and “Condition” to specify what resource and how large of the quota. You can assign quota to accounts and users. And if a user is restricted by multiple quota spec, the smallest is taken into effect.
We don’t have much time left. I’ll briefly talk about another Eucalyptus 3 feature that is related to the identity management. That is we enable the LDAP/AD sync in Eucalyptus 3. To do that, you can simply write a LIC (LDAP Integration Configuration) and upload to the system. The identities in the system will then be synced from the specified LDAP/AD service. There is the question of how to map the structure of LDAP tree to the IAM account/group/user model. We leave that for offline discussion. You can send us email at wenye@eucalyptus.com for more information.
Thanks everyone for attending this class! Thank you all, and we look forward everyone trying out Eucalyptus 3 and letting us know what you think!
Day 2
Getting started with OpenStack Compute (AKA “Nova”)
Summary: An introduction to OpenStack Compute. What components comprise OpenStack (nova-api, nova-compute, nova-scheduler, nova-network, nova-objectstore). And a demo of setting up OpenStack on your own machine and running an instance within it.
Hopefully at the end of this session, you’ll have a basic understanding of the components involved in an OpenStack Compute cloud, have a cloud running on your laptop and you’ll be able to start and stop instances.
OpenStack Compute consists of a number of “internal” and “external” components. “External” components (a term I just came up with, so you probably won’t find it in any docs or anything) are things that aren’t part of Nova itself, but we need anyway. For our needs today this is just rabbitmq. RabbitMQ is a message queueing system. It’s a core component of a Nova deployment. It passes messages between the different components. It’s the only control communication mehcnaism we have.
Nova itself has 5 components that we’ll work with today:
nova-api is the frontend server. Whenever you — as an enduser — want to interact with Nova, this is the component you talk to. It exposes the native OpenStack API (which is derived from the Rackspace Cloud Servers API) as well as the EC2 API. Well, a subset of the EC2 API. It receives your request over HTTP, authenticates and validates the request and turns it into a message and puts it on the message queue for someone else to consume. You can have any number of these servers. They all act the same: Verify the request and sends it on to some other component for processing.No failover or anything is involved, all components are “hot”.
nova-compute is the component that actually runs your virtual machines. It receives a request from the scheduler, which I apparently should have talked about first, but here we are 🙂 Ok, so this request has information about which image to run, how much memory and disk and whatnot to assign to the virtual machine. nova-compute makes this happen by creating virtual disk images, shoving the AMI image into it, setting up networking and running the virtual machine. So, on the bulk of your servers in your cloud, this is the thing you run.
kim0 asked: Does rabbitmq choose a node to deliver the message to? if that node fails, does it reroute to some other node ?
It depends on the type of message. Hm… Let me explain a bit more first, and get back to this.
nova-scheduler As you might have guessed, it schedules stuff. When you ask the API server to run a virtual machine it sends the request to one of the schedulers. The scheduler is (meant to be) smart. It makes the decision about which compute node is meant to run the virtual machine. When it has decided, it sends the message on to the chose compute node. Ok, so rabbit. The API server sort of broadcasts the request to all the schedulers. Here, rabbitmq makes the decision about which scheduler gets the request. …and applies its usual logic about sending the request somewhere else if it doesn’t get acked. for the scheduler->compute message, it’s different, of course. …since it’s being sent directly to a specific node. I’m not sure what the failure modes are there. The scheulder doesn’t just schedule VM’s, though. It’s supposed to make decisions about which storage node is supposed to host your EBS-like volumes and so on.
nova-network is the network component. It hands out IP addressses and in certain network modes it acts as the gateway. It also does NAT’ing for elastic IP’s and a few other things. It doesn’t do firewalling. The compute nodes do that (in order to decentralise the firewall and spread the filtering load).
nova-objectstore is a simple ple implementation of the S3 API. You can use OpenStack without using it, but in Ubuntu we still use it because that’s how EC2 and Eucalyptus work, so we can reuse a bunch of documentation an tools and whatnot this way.
kim0 asked: Is there any way to specify locality, like launch this VM besides that VM as close as possible to that EBS volume ?
I’m not completely sure. There’s certainly been talk about supporting it, but I don’t remember seeing code to do it. That said, I’ve been on holiday and at conferences for a while, so maybe it landed recently. It’s definitely on the roadmap.
kim0 asked: Has openstack dropped the centralized DBs ? wasn’t mysql used before?
We have not. Yet. So, if you’re setting up a multi-machine cloud, you’ll need a shared database. E.g. MySQL, PostgreSQL or whatever floats your boat. Whatever sqlalchemy supports should be fine. “should” being the operative word. Data access works fine for all the different backends, but schema changes have only been tested with mysql, postgresql and sqlite. This is a common theme, by the way. We can use a bunch of different DB backends. We can also use a bunch of different hypervisors (kvm, Xen, LXC, UML, Xen Server, VMWare, Hyper-V). Different storage backends (HP SANs, iSCSI, AOE). Adding more drivers is meant to be reasonable easy, so if you want to use something else, that should be possible. Let us know if that’s the case.
kim0 asked: Is there code to accelerate certain operations via SAN (like clone volume, snapshotting ..etc)
I forget which operations are exposed in abstraciton layer, to be honest. I do believe the backend call to create a snapshot is just “create a snapshot”, so if a particular backend has nifty shortcuts to do that, it should totally be possible. Whether the drivers actually *do*… I don’t know.
< Guest95592> QUESTION : is there any docs talk about openstack with Hyper-V ? Or real use case ?
I’m not actually sure of the state of the Hyper-V support. We don’t really have the means to test it.
Ok, so let’s get practical. I assume you’ve all downloaded the image. I’m assuming you’re running Natty.
kim0 asked: Would you explain how a large scale (1000?) server deployment may look like. Guidelines would be great. Is it also an inverted tree of a bunch of clusters like eucalyptus? A lot of it will be dictated by the network structure you want.
Hmm.. This is a good question, really. I just write the software, I don’t actually run it at scale. It’s not a strict hierarchy. At all. You have the message queue, which everything shares. “a number” of API servers. 🙂 I don’t really know how many I’d set up. Ideally, they’d run as instances on your cloud so that you could scale them as appropriate. I’d be surprised if one or two of them wouldn’t be sufficient, but if you find out that you need a bunch of them, just add more and point them at the same message queue and db and you should be golden. Number of storage servers depends on how much storage you want to expose, really. …and how much you expect your users to use them. Nova doesn’t really add any overhead there. If you expect to have a lot of I/O-intensive stuff going on on your cloud, you’d probably have a lot of servers with a lot of bandwidth. If you don’t expect a lot of I/O-intensive stuff you might just spring for fewer servers with more disks in each. I don’t think there really are any good, general answers. The architecture is really flexible, though, so if you discover that you need more store or bandwidth, you just add more. Since the architecture is so flat.
Ok, so back to our demo. Everyone runs natty. Great. First:
sudo apt-get install rabbitmq-server unzip cloud-utils
This install rabbitmq and a few utilities that we’ll need in a few minutes. When that is done, do this:
sudo apt-get install nova-api nova-compute nova-scheduler nova-network nova-objectstore
I’m not completely sure this still needs to be two separate steps, but just to be safe, we do it this way. Ok, so next up, we create a user. If your name is not “soren” you can specify something else where it says “soren” 🙂
sudo nova-manage user admin soren
This creates a “project” called “soren” with the user “soren” as the admin. Next, we need to create a network. If you can’t use 10.0.0.0/8 (beacuse you already use it), make something else up.
sudo nova-manage network create 10.0.0.0/8 2 24
This creates two networks of 24 IP’s each in the 10.0.0.0/8 subnet.
sudo nova-manage project zipfile soren soren
This fetches a zipfile with from credentials in it for the user soren for the project soren. Unpack it:
unzip nova.zip
Source the novarc file:
. novarc
Now we upload the image we downloaded earlier.
uec-publish-tarball ubuntu-11.04-server-uec-amd64.tar.gz ubuntu
ubuntu-11.04-server-uec-amd64.tar.gz is the filename, “ubuntu” is the name of the S3 bucket you want to use. Next, we create a key:
euca-add-keypair mykey > mykey.priv chmod 600 mykey.priv
uec-publish-tarball may take a while. The last thing it outputs might look like:
emi="ami-3a0c3765"; eri="none"; eki="aki-279dfe6a";
We need the first ID there (ami-3a0c3765). Your id will be different. Well, probably. there’s a 1 in 2^32 chance it’ll be the same 🙂 Ok, next:
echo '#!/bin/sh' >> myuserdata echo "wget http://f.linux2go.dk:8080/$HOSTNAME/" >> myuserdata
And finally:
euca-run-instances -k mykey -t m1.tiny -f myuserdata "the ami ID we found a bit further up"
If this works, we’ll be able to see your hostname on http://f.linux2go.dk:8080/ …and you’ll also be able to ssh into the instance. Is this working for everyone? [did not, for me] I don’t really anything more, I didn’t know how long this demo thing would take.
Ubuntu Enterprise Cloud on Ubuntu 10.04 LTS
Summary: This talk walks one though installing UEC from packages on Ubutnu 10.04 LTS, instantiating an instance, an explanation of persistent storage and how to set it up (EBS backed by AoE), and elastic IPs.
Hello there, glad to have you aboard for a session on UEC on Ubuntu 10.04 LTS. Thanks to soren for providing us with info on openstack, a very interesting and recent development in cloud space. In the next hour I will write about UEC, Ubuntu Enterprise Cloud, as we find it in Ubuntu 10.04 LTS. Most of you will already know that the LTS (Long Term Support) releases are quite important to the Ubuntu universe. with 5 years of support for servers, it is a good choice for deploying it in any datacenter. Drawback is of course, that you don’t get the latest and greatest software on it, you pretty much have to live with what was there in April 2010 for Ubuntu 10.04 LTS. So with regards to UEC, we find that the base is eucalyptus 1.6.2. some of you might have attended nurmi’s talk yesterday on Eucalyptus 3. There’s of course quite a lot of features missing in 1.6.2, but if you are in an LTS environment, or plan to set one up, you still can get started with UEC. So the most useful document for getting started with it, is on the wiki:
https://help.ubuntu.com/community/UEC
It contains various info that to the best of my knowledge mostly still applies to 10.04 LTS. However, due to some problems with service discovery, one might need to manually set up a cloud when doing a packaged install. I’ve detailed the needed steps here, as I need them often when teaching the UEC class:
http://people.canonical.com/~tspindler/UEC-Jan/02-cloudPackagedInstall
Typically it takes 10-20 minutes to set up a two node cloud with UEC. One node acting as the front-end, the other as the node controller. While this is hardly sufficient for any serious deployment, it’s good to get started and you can add more node controllers later on. Once the cloud is up and running, you need to either download credentials for accessing it from the web ui or via the euca_conf command on the front-end. I prefer the later, so a
$ sudo euca_conf --get-credentials admin.zip
will store the credentials for the admin user in the admin.zip file. Next I save these credentials on a client system into ~/.euca-admin/ note that the wiki recommends saving it in ~/.euca, but if you have multiple users on one client system in one account, it’s better to have several directories. e.g. I then usually create a user ‘spindler’ that has non-admin privileges and store the credentials in ~/.euca-spindler. I think using multiple users for multi-tenancy in the cloud is a very nice feature, so I like to use it. Once the credentials are available, I source the accompanying ‘eucarc’ file and we’re ready to go. With euca-describe-availability-zones verbose I do a quick check if the cloud is operational.
$ euca-describe-availability-zones verbose AVAILABILITYZONEItorstenClusterI172.24.55.253 AVAILABILITYZONEI|- vm typesIfree / max cpu ram disk AVAILABILITYZONEI|- m1.smallI0023 / 0024 1 192 2 AVAILABILITYZONEI|- c1.mediumI0022 / 0022 1 256 5 AVAILABILITYZONEI|- m1.largeI0011 / 0011 2 512 10 AVAILABILITYZONEI|- m1.xlargeI0005 / 0005 2 1024 20 AVAILABILITYZONEI|- c1.xlargeI0002 / 0002 4 2048 20
Hope I don’t get kicked for flooding. So here you see a basic UEC in operation. The name of the cluster is ‘torstenCluster’, you see the private IP of the cluster controller. The lines below the first show how many instances of each instance type you can run. The instance types will look familiar to those having AWS background. Basically the command tells me that there’s a cluster controller operational and a node controller has been found as well. At least one node controller. If you only see 0 for free and max, registering the node controller didn’t work and you probably need to repeat that step. In theory a new node controller in the same subnet as my cluser controller would get picked up automatically. In practice it always worked for me, but I’ve read in bug reports that at times it didn’t for other users. Talking about the auto registration – there is one caveat. Don’t setup more than one UEC cloud in the same LAN. Otherwise the auto registration will bring strange fruits. You can always turn off the auto registration agents in their upstart / init jobs, though, in case you need to. e.g. you want to setup a classroom with multiple clouds and one LAN.
Back to our cloud, the next thing to do is getting an image and uploading it to the cloud. The easiest way to do so is using the web interface. However, almost as easy is downloading a tarball from uec-images.ubuntu.com and provisioning that. Thanks to the developers there are scripts that make this really easy. uec-publish-tarball is the one for publishing a tarball from uec-images in the cloud. Once this is done, the image resides in /var/lib/eucalyptus/bukkits/<bucket name>/ on the front-end. Cloud tech speaking we have stored a machine image in a bucket on S3. We need to store it in a way the node controller can later download and execute the image file. And S3 is one such storage method in UEC. Next step is to create a ssh keypair usually. This can be done with euca-add-keypair <keyname>. Best is to first check with euca-describe-keypairs which ones already exist. While the command nowadays blocks creation of another key with the same name as an existing one, this was not always the case in the past. Once we have an image and a keypair, we can start an instance. When doing all of this on the command line, rather than using hybridfox or landscape or any other mgmt tool. I find it useful to save the identifiers in variables, for example:
IMAGEIemi-ACC617F6Imaverick-2011-01-26/maverick-server-uec-amd64.img.manifest.xmlIadminIavailableIpublicIIx86_64ImachineIeki-14A41D03
This is a maverick server image for 64bit from back in January. I store the emi identifier in a variable emi:
emi=emi-ACC617F6
So when I want to run an instance of that type, I state:
euca-run-instances $emi -k <key name> -t <instance type>
I usually neglect the instance type and make do with m1.small, but at times bigger instances are worth it. The euca-run-instances command returns another identifier, this time for the instance. I save that in the variable ‘inst’ or ‘server-<name>’ or whatever else I like so I can quickly check the status of the instance with
$ euca-describe-instances $inst
If you’re out on your own with a single user and a small cloud, it might not make much sense for restricting the output of euca-describe-instances. But if you have a class full of people, everybody starting and stopping instances, it is useful to only see info on the wanted instance. Once the instance is in state running, one can ssh to it:
ssh -i <identity file> ubuntu@<public or private ip>
The -i option we need to use the right private key file, the one that was returned by euca-add-keypair earlier. If we need the public or private ip depends on where our client sits if we use the front-end as client, which a lot of people will initially do, the private ip is as good as any. If the client is on your laptop or any other system not hooked up to the clouds infrastructure, you need to use the public ip. When you have done all the work needed by the instance, you can terminate it with ‘euca-terminate-instances’.
So that covered the basic operation of a UEC cloud. First install the systems, either by CD or by package. Second, get the credentials. Third, check if the cloud is operational. Next make an image available. start an instance of that image. Test via ssh or whatever other service the instance may provide. But you don’t need to stop there, there’s plenty to still explore, like attaching persistent storage devices to the instance, allocating public ip addresses to the instance.
Before we cover the persistent storage, let’s see why we have the need for that. What happens when an instance is started? A node controller sooner or later gets tasked by a cluster controller to run an instance. That instance is of a specific emi, eucalyptus machine image. The node controller checks a local cache if the emi is there, and if not, the emi is transferred to the node controller via S3. The node controller then copies the image to an instance directory. Inside the instance directory the image is configured according to the start parameters of euca-run-instances. e.g. an ssh key is injected, e.g. a cloud init file is added to the boot sequence of the instance. Some more magic happens and the once xen like image is now a kvm image and kvm is used to boot the instance. So in a way eucalyptus and UEC is like kvm on steroids. When you want to get rid of an instance, you stop it with euca-terminate-instances and what happens on the node controller is that the kvm process stops _and_ the runtime directory containing the image is deleted. This means that any state in the instance is gone forever. Of course most services require to save some state somewhere. This could be an external database server, or a file server. or, as we look at now, EBS, the elastic block storage service.
In EBS a device appears on an instance that is mapped to a file on a storage server, the EBS server. There are two methods by which EBS is transported, ata over ethernt (AoE) or iscsi. In Ubuntu 10.04 LTS we only have AoE. This implies a restriction when using 10.04 LTS, as the EBS server has to reside on the same LAN as the node controllers it serves as AoE does not route. With euca-create-volume we can create a new volume for storage on the EBS storage server. Once this has finished, it can be assigned to an instance with euca-attach-volume
euca-attach-volume <vol id> -i $inst
Oops, there’s -d <device> missing. Device would be sdb or sdc or anything But to my finding the instance will just pick the next device name anyhow with the device attached to the instance, one can create a partition table, filesystem and whatever on the device just as with a regular device. The nice thing is, that you can snapshot these devices. e.g. you attach the device to an instance, put the needed data there, detach it and snapshot it. Based on that snapshot you can create new volumes that are clones of the snapshot – might be nice if you have read only data that you need on all instances of your application. That’s it pretty much for EBS.
Lastly I want to cover elastic IPs. With help of euca-describe-addresses you see which IPs are public in your UEC. You can either randomly get one or pick one with euca-allocate-address. Then this ip can be assigned to a specific instance with euca-associate-address. In this way you can make sure that a certain service is always reachable by the same IP, minus the time it takes to do the associate dance. Well, that was all I wanted to cover. Thanks for following. If you have any questions, either use the classbot or you can reach me as TeTeT on #ubuntu-cloud usually.
Node.js/Mongo with Ensemble
Summary: this talk is a specific example using Ensemble to provision a node.js server backed by mongo DB. It uses two types of ensemble forumulas: “canned” and “framework”, where canned formulas give you standard pieces of infrastructure, and framework formulas give you custom pieces. This demo switched back and forth between IRC and a byobu-classroom, so I tried to capture the code that was flashing past as best I could. It’s interspersed approximately chronologically with the description. Note that wordpress helpfully stripped most of the indentation.
Hi, I’m Mark M Mims (hence the m_3) On the Ensemble team working to build out the base set of formulas we have to work with in ensemble. I’d like to go into a little detail today about the different kinds of formulas ensemble can work with. at this point I hope that you’ve seen the earlier presentation on ‘getting started with ensemble’. I’ll be demoing stuff in byobu-classroom, where you can watch along. You can either open http://ec2-67-202-23-199.compute-1.amazonaws.com/ up in a browser or ssh there directly as “guest” with password “guest” So, how do you use ensemble to deploy a node.js app? the short version (we’ll go repeat in more detail) is:
$ ensemble bootstrap $ ensemble deploy --repository .. mynodeapp $ ensemble deploy --repository .. mongodb $ ensemble deploy --repository .. haproxy
to deploy all the services as we expect. here, we have a simple (exposed) mongodb service, a node.js app that sits above the mongodb, and haproxy that faces the outside world. We deploy the services, then relate them:
$ ensemble add-relation mongodb mynodeapp $ ensemble add-relation mynodeapp haproxy
then wait for the relation hooks to complete the outcome of that is shown in ssh. The point of this talk is that there are two different kinds of services involved here:
- “canned” services… like haproxy and mongodb. These are services you typically just configure and use they’re tailored to your infrastructure through configuration parameters
- what I like to call “framework” formulas. these are formulas for services that you write i.e., mynodeapp is an application that I keep under my own revision control system. a framework formula is a formula that helps you deploy such a service alongside other canned services.
Let’s look at an example. Mynodeapp is a _super_ simplistic node.js app there’s a basic http server listening on 0.0.0.0/8000. that tracks hits to the page and sticks them in a mongo database. Easy peasy. It reads some information from a config file… that’s just simple json I’ve written a basic ensemble formula to wrap the installation and deployment of that node.js application. The metadata.yaml shows where it fits within my overall infrastructure this consumes a db, and provides a http interface:
ensemble: formula name: mynodeapp revision: 16 summary: "Example node.js application" description: "Example node.js application" requires: - mongodb: - interface: mongodb provides: - website: - interface: http
I can of course attach monitoring services, remote logging services, etc there’s a little cruft in here about formula config
[[ -f "$(dirname $0)/config" ]] && source "$(dirname $0)/config" # goes away with config.yaml / config-get
app_name=${config_app_name:-"mynodeapp"} # `config-get app_name`
app_dir=${config_app_dir:-"/opt/${app_name}"} # `config-get app_dir`
app_user=${config_app_user:-"ubuntu"} # `config-get app_user`
app_scm=${config_app_scm:-"git"} # `config-get app_scm`
app_url=${config_app_url:-"http://github.com/mmm/testnode.git"} # `config-get app_url`
app_branch=${config_app_branch:-"write"} # `config-get app_branch`
install_node() {
- ensemble-log "Installing node..."
sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
apt-get -y install -qq nodejs
}
[[ -x /usr/bin/node ]] || install_node
install_npm() {
- ensemble-log "Installing npm..."
apt-get -y install -qq git-core
git clone https://github.com/isaacs/npm /tmp/npm -b 0.2
cd /tmp/npm && node cli.js install
}
[[ -x /usr/bin/npm ]] || install_npm
install_app() {
- ensemble-log "Installing ${app_name}..."
git clone ${app_url} ${app_dir} -b ${app_branch}
chown -Rf ${app_user}.${app_user} ${app_dir}
if [ -f ${app_dir}/package.json ]; then
- cd ${app_dir} && npm install
fi
cat > /etc/init/${app_name}.conf <<EOS
That’s a new feature that recently landed. So all of that should be replaced with ‘config-get <param>’. The installation of node itself and npm are pretty straightforward. Try to use packages when you can. When you’re on HEAD or living closer to the edge, you can pull and install from source if you want. That’s worth mentioning again… packages are great for stability and often what the ops guys want. Developers want the bleeding edge all the time. Ensemble formulas support either so it becomes a policy decision within your group but anyway… I’m pulling my node.js from github you can use any number of other options. The key is to install the packages first for the VCS you use. Note there’s a need for some idempotency guards. This is most important for long lifecycle services.
Ok, so to summarize the first part of what we’re looking at…
this is a formula to deploy my node.js app right alongside all the ‘canned’ service formulas
this is the ‘install’ hook that gets called upon service unit deployment
loads some configuration then goes about installing
node
npm
and then my application (pulled from github) and then delays any further startup/config until we have the right parameters to fill in from the mongodb service
So that’s simple enough now, the real magic of Ensemble is relations. This sets ensemble apart. the lines of demarcation between what’s specific to a service. and what’s really specific to the relations between the services. here, we see the mongodb-relation-changed hook
#!/bin/bash
set -eu # -x for verbose logging to ensemble debug-log
# Get the database settings; if not set, wait for this hook to be
# invoked again
host=`relation-get host`
if [ -z "$host" ] ; then
- exit 0 # wait for future handshake from database service unit
fi
relation_port=`relation-get port`
port=${relation_port:-27017}
[[ -f "$(dirname $0)/config" ]] && source "$(dirname $0)/config" # goes away with config.yaml / config-get
app_name=${config_app_name:-"mynodeapp"} # `config-get app_name`
app_dir=${config_app_dir:-"/opt/${app_name}"} # `config-get app_dir`
ensemble-log "configuring ${app_name} to work with the mongodb service"
config_file_path=$app_dir/config/config.js
if [ -f $config_file_path ]; then
- ensemble-log "Writing $app_name config file $config_file_path"
sed -i "s/mongo_host.*/mongo_host\" : \"${host}\"/" $config_file_path
sed -i "s/mongo_port.*/mongo_port\" : ${port}/" $config_file_path
fi
ensemble-log "Starting app"
service ${app_name} restart || service ${app_name} start
This isn’t called until the mongodb service is related to our mynodeapp service. start at the bottom…
note that we don’t actually start the “mynodeapp” service until we’ve set the relation parameters. This makes sense if we look at the config.js again for the app
module.exports = config = {
- "name" : "mynodeapp"
,"listen_port" : 8000
,"mongo_host" : "localhost"
,"mongo_port" : 27017
}
The default paramters the node.js app is using expect mongodb to be local. Ok, if you noticed during the install hook, I _do_ have it installed locally, But that was just for testing. In general, the mongodb service is external so the node.js app would barf if we started it with this info. So, the code before starting the service is just to get the right connection information from the db. When a relation is created Ensemble opens a two-way comms channel between the services through the use of ‘relation-get’ ‘relation-set’ the particular mongodb service we’re using is pretty simplistic.Let’s look at the other end of this relation:
#!/bin/sh relation-set host=`hostname -f`
that’s it! when a service connects to the mongodb service, mongodb just sends it’s connection point
#!/bin/bash
set -eu # -x for verbose logging to ensemble debug-log
# Get the database settings; if not set, wait for this hook to be
# invoked again
host=`relation-get host`
if [ -z "$host" ] ; then
- exit 0 # wait for future handshake from database service unit
fi
relation_port=`relation-get port`
port=${relation_port:-27017}
[[ -f "$(dirname $0)/config" ]] && source "$(dirname $0)/config" # goes away with config.yaml / config-get
app_name=${config_app_name:-"mynodeapp"} # `config-get app_name`
app_dir=${config_app_dir:-"/opt/${app_name}"} # `config-get app_dir`
ensemble-log "configuring ${app_name} to work with the mongodb service"
config_file_path=$app_dir/config/config.js
if [ -f $config_file_path ]; then
- ensemble-log "Writing $app_name config file $config_file_path"
| sed -i "s/mongo_host.*/mongo_host\" : \"${host}\"/" $config_file_path
| sed -i "s/mongo_port.*/mongo_port\" : ${port}/" $config_file_path
fi
ensemble-log "Starting app"
service ${app_name} restart || service ${app_name} start
When a service connects to the mongodb service, mongodb just sends it’s connection point a more mature version of this would have access control, ports specified, etc
2011-07-26 12:36:09,430 INFO Connecting to environment.
machines:
0: {dns-name: ec2-50-16-141-182.compute-1.amazonaws.com, instance-id: i-8d943fec}
1: {dns-name: ec2-50-17-63-99.compute-1.amazonaws.com, instance-id: i-fd68c39c}
2: {dns-name: ec2-72-44-35-213.compute-1.amazonaws.com, instance-id: i-d168c3b0}
3: {dns-name: ec2-174-129-107-151.compute-1.amazonaws.com, instance-id: i-b568c3d4}
services:
haproxy:
formula: local:haproxy-14
relations: {}
units:
haproxy/0:
machine: 3
relations: {}
state: started
mongodb:
formula: local:mongodb-4
relations: {mongodb: mynodeapp}
units:
mongodb/0:
machine: 1
relations:
mongodb: {state: up}
state: started
mynodeapp:
formula: local:mynodeapp-16
relations: {mongodb: mongodb}
units:
mynodeapp/0:
machine: 2
relations:
mongodb: {state: up}
state: started
2011-07-26 12:36:13,452 INFO 'status' command finished successfully
ensemble ssh mynodeapp/0
2011-07-26 12:37:06,181 INFO Connecting to environment.
2011-07-26 12:37:07,975 INFO Connecting to mynodeapp/0 at ec2-72-44-35-213.compute-1.amazonaws.com
Warning: Permanently added 'ec2-72-44-35-213.compute-1.amazonaws.com,72.44.35.213' (RSA) to the list of known hosts.
Welcome to Ubuntu 11.04 (GNU/Linux 2.6.38-8-virtual i686)
* Documentation: https://help.ubuntu.com/
System information as of Tue Jul 26 18:36:40 UTC 2011
System load: 0.0 Processes: 61
Usage of /: 10.8% of 7.87GB Users logged in: 0
Memory usage: 14% IP address for eth0: 10.118.251.96
Swap usage: 0%
Graph this data and manage this system at https://landscape.canonical.com/
---------------------------------------------------------------------
At the moment, only the core of the system is installed. To tune the
system to your needs, you can choose to install one or more
predefined collections of software by running the following
command:
sudo tasksel --section server
---------------------------------------------------------------------
Last login: Tue Jul 26 17:56:20 2011 from 76.77.152.109
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
ubuntu@ip-10-118-251-96:~$ cat /opt/mynodeapp/config/config.js
module.exports = config = {
"name" : "mynodeapp"
,"listen_port" : 8000
,"mongo_host" : "ip-10-113-21-60.ec2.internal"
,"mongo_port" : 27017
}
When a service connects to the mongodb service, mongodb just sends it’s connection point. A more mature version of this would have access control, ports specified, etc., but that’s enough info for a first pass at understanding ensemble relations. The mynodeapp service just grabs the relation information (here just the host) and crams it in the config file. We can go out to the live service and see what the config file looks like. First notice the relation status between the mongodb and the mynodeapp services. I can ssh directly to one unit and see that the relation hook filled in the ec2 internal address for the mongodb service and the port’s just default. Cool… questions so far? ha, ok… I continue.
Next, let’s look at what happens when we fire up additional units
ensemble:~/projects/principia/mynodeapp $ ensemble add-unit mynodeapp
2011-07-26 12:39:43,806 INFO Connecting to environment.
2011-07-26 12:39:45,985 INFO Unit 'mynodeapp/1' added to service 'mynodeapp'
2011-07-26 12:39:45,989 INFO 'add_unit' command finished successfully
1: {dns-name: ec2-50-17-63-99.compute-1.amazonaws.com, instance-id: i-fd68c39c}
2: {dns-name: ec2-72-44-35-213.compute-1.amazonaws.com, instance-id: i-d168c3b0}
3: {dns-name: ec2-174-129-107-151.compute-1.amazonaws.com, instance-id: i-b568c3d4}
4: {dns-name: ec2-184-72-92-144.compute-1.amazonaws.com, instance-id: i-8559f2e4}
services:
haproxy:
formula: local:haproxy-14
relations: {}
units:
haproxy/0:
machine: 3
relations: {}
state: started
mongodb:
formula: local:mongodb-4
relations: {mongodb: mynodeapp}
units:
mongodb/0:
machine: 1
relations:
mongodb: {state: up}
state: started
mynodeapp:
formula: local:mynodeapp-16
relations: {mongodb: mongodb}
units:
mynodeapp/0:
machine: 2
relations:
mongodb: {state: up}
state: started
mynodeapp/1:
machine: 4
relations: {}
state: null
2011-07-26 12:40:25,161 INFO 'status' command finished successfully
ec2’s been quite slow today, so we’ll wait a bit. Notice as this comes up that the services ‘mynodeapp’ and ‘mongodb’ are already related. All we did with ensemble add-unit was to add an additional service unit for the mynodeapp service (horiz scaling) so ensemble spins up a new instance, runs the install hook we saw before. let’s look again for a sec… notice there’s nothing in here that would depend on another service unit. We pulled some config from the formula, we installed stuff that’s going to be in common with all instances of our node.js app. The relation is the only real information that the node.js app “nodes” or units would share. Ensemble just calls the relation hooks after installing the new service unit, so then this new service unit should get the same relation-specific configuration that the first one had (and cram it in config.js for the app).
1: {dns-name: ec2-50-17-63-99.compute-1.amazonaws.com, instance-id: i-fd68c39c}
2: {dns-name: ec2-72-44-35-213.compute-1.amazonaws.com, instance-id: i-d168c3b0}
3: {dns-name: ec2-174-129-107-151.compute-1.amazonaws.com, instance-id: i-b568c3d4}
4: {dns-name: ec2-184-72-92-144.compute-1.amazonaws.com, instance-id: i-8559f2e4}
services:
haproxy:
formula: local:haproxy-14
relations: {}
units:
haproxy/0:
machine: 3
relations: {}
state: started
mongodb:
formula: local:mongodb-4
relations: {mongodb: mynodeapp}
units:
mongodb/0:
machine: 1
relations:
mongodb: {state: up}
state: started
mynodeapp:
formula: local:mynodeapp-16
relations: {mongodb: mongodb}
units:
mynodeapp/0:
machine: 2
relations:
mongodb: {state: up}
state: started
mynodeapp/1:
machine: 4
relations: {}
state: null
2011-07-26 12:45:18,704 INFO 'status' command finished successfully
Ugh… I thought for sure I rambled enough for ec2 to catch up. Well anyway, while we’re waiting, we can test a couple of things. Note that you can hit the first mynodeapp service unit “mynodeapp/0” on machine 2 and see the node.js app in action you can try http://ec2-72-44-35-213.compute-1.amazonaws.com:8000/ as well as http://ec2-72-44-35-213.compute-1.amazonaws.com:8000/hits
should give us some additional traffic in the db. This is open in this case, it wouldn’t be in real life. Ensemble has features to control ec2 security groups, so we’d have this webservice exposed on port 8000 only on the internal ec2 interface; you’d have to go through haproxy for public access.
2: {dns-name: ec2-72-44-35-213.compute-1.amazonaws.com, instance-id: i-d168c3b0}
3: {dns-name: ec2-174-129-107-151.compute-1.amazonaws.com, instance-id: i-b568c3d4}
4: {dns-name: ec2-184-72-92-144.compute-1.amazonaws.com, instance-id: i-8559f2e4}
services:
haproxy:
formula: local:haproxy-14
relations: {}
units:
haproxy/0:
machine: 3
relations: {}
state: started
mongodb:
formula: local:mongodb-4
relations: {mongodb: mynodeapp}
units:
mongodb/0:
machine: 1
relations:
mongodb: {state: up}
state: started
mynodeapp:
formula: local:mynodeapp-16
relations: {mongodb: mongodb}
units:
mynodeapp/0:
machine: 2
relations:
mongodb: {state: up}
state: started
mynodeapp/1:
machine: 4
relations:
mongodb: {state: up}
state: started
2011-07-26 12:47:46,033 INFO 'status' command finished successfully
Hey, ok, our new mynodeapp service unit is up, mynodeapp/1 on machine 4. Note that you can hit that one independently too: http://ec2-184-72-92-144.compute-1.amazonaws.com:8000/.
ensemble ssh mynodeapp/1
2011-07-26 12:52:10,079 INFO Connecting to environment.
2011-07-26 12:52:12,114 INFO Connecting to mynodeapp/1 at ec2-184-72-92-144.compute-1.amazonaws.com
Warning: Permanently added 'ec2-184-72-92-144.compute-1.amazonaws.com,184.72.92.144' (RSA) to the list of known hosts.
Welcome to Ubuntu 11.04 (GNU/Linux 2.6.38-8-virtual i686)
* Documentation: https://help.ubuntu.com/
System information as of Tue Jul 26 18:51:44 UTC 2011
System load: 0.0 Processes: 62
Usage of /: 10.8% of 7.87GB Users logged in: 0
Memory usage: 14% IP address for eth0: 10.36.7.159
Swap usage: 0%
Graph this data and manage this system at https://landscape.canonical.com/
---------------------------------------------------------------------
At the moment, only the core of the system is installed. To tune the
system to your needs, you can choose to install one or more
predefined collections of software by running the following
command:
sudo tasksel --section server
---------------------------------------------------------------------
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
ubuntu@ip-10-36-7-159:~$ cat /opt/mynodeapp/config/config.js
module.exports = config = {
"name" : "mynodeapp"
,"listen_port" : 8000
,"mongo_host" : "ip-10-113-21-60.ec2.internal"
,"mongo_port" : 27017
}
Ok, so let’s check out the config file in the new node and make sure that the relation added the right info. boom. So when we hit either node, they’re writing the hits into the common external mongodb datastore. sorry, I keep using bash aliases… es=’ensemble status’. Ok, so what’s missing here? Notice that we have a haproxy service up, but no relations.
ensemble:~/projects/principia/mynodeapp $ ensemble add-relation mynodeapp haproxy 2011-07-26 12:53:54,091 INFO Connecting to environment. 2011-07-26 12:53:56,220 INFO Added http relation to all service units. 2011-07-26 12:53:56,221 INFO 'add_relation' command finished successfully
This will call hooks to relate the webservice http interface provided by each mynodeapp to the haproxy balancer
#!/bin/sh
[[ -f "$(dirname $0)/config" ]] && source "$(dirname $0)/config" # goes away with config.yaml / config-get
app_port=${config_app_port:-"8000"} # `config-get app_port`
relation-set port=$app_port hostname=`hostname -f`
Really it just tells haproxy its hostname and port (8000 here). The haproxy service is written so that when a webservice joins, it adds it to the roundrobin queue. Ok, almost done. Now, we can hit ec2-174-129-107-151.compute-1.amazonaws.com on port 80 and it’s balancing between the node.js nodes. Our nodes.
Ok, so to wrap up. Two types of formulas:
- canned formulas
- “framework” formulas
mongodb and haproxy are examples of canned ones. mynodeapp is a framework one. The intended use is to fork an example and tailor to your needs. Thanks all…
OpenStack: An open cloud infrastructure project
Summary: High level overview of OpenStack, the way the project is governed and developed. Management speak. High level description of OpenStack Compute (“Nova”). Some questions about deployment.
Hi everyone. My name is Thierry Carrez, I’m the release manager for the OpenStack project. Raise your hand in #ubuntu-classroom-chat if you’re here for the OpenStack Project intro session! I’m also an Ubuntu core developer, working on Ubuntu Server, time permitting
19:01 #ubuntu-classroom-chat: * rwh raises hand
19:02 #ubuntu-classroom-chat: <+ttx> cool, I’m not alone 🙂
In this session I’d like to introduce what OpenStack is. So, what is OpenStack ? It’s an open source project that creates software to run a cloud infrastructure. It allows you to be an IaaS (infrastructure as a service) provider. In short, it’s software you can run to create a competitor to Amazon Web Services or the Rackspace Cloud. But it also scales down so that you can use it to manage your private SMB computing resources in a cloudish way. The project is built around 4 “open”:
- Open source: The software is Apache-licensed, and the whole project is open source. So no “enterprise edition” that keeps the tasty features
- Open design: We have open design summits, much like UDS, every 6 months. (The next one is in Boston, October 3-5 !). Also everyone can propose a feature, we use Launchpad for blueprints… but be ready to implement it if you do 🙂
- Open development: We use DVCS (bzr and git) with merge proposals and public comments, so you know why a particular piece of code is accepted or not. The code is written in Python, which makes it easier for everyone to read the code, understand it, investigate why it fails and propose patches
- Open community: We have community-elected project leaders and seats on the project policy board, we do meet weekly on IRC. (next meeting in 2 hours in #openstack-meeting !). We have about 100 different developers that have committed code, from more than 20 different companies
OpenStack is made of several subprojects:
- We have “core projects”, that follow all our rules and are part of the official OpenStack release
- We also have “Incubated” projects, that learn the process and should become core projects one day
- Finally we have “Related” projects, that are created in the OpenStack community but should not become core projects for one reason or another
We used to have a 3-month release cycle, but at the last design summit we decided to switch to a 6-month release schedule, with monthly milestones. Milestones can be used to evaluate new features as they land in the projects (For example we’ll release the diablo-3 milestone Thursday !). The 6-month release is aligned with the Ubuntu release cycle, so that you can get the latest OpenStack release in the latest Ubuntu release. Therefore OpenStack 2011.3 (also known as “Diablo”) should be released on September 22, and included in 11.10. Questions ?
< ClassBot> rwh asked: does Canonical have plans to launch an AWS competitor based on this?
I’m not part of Canonical (anymore) but I don’t think so.
If there are no other questions on the project itself, let’s go into more details in the 3 current core subprojects. The first one (and most mature) is Swift (OpenStack Object storage). You can use it to run a raw cloud storage provider, very much like Amazon S3. It allows your users to PUT and GET small or very large binary files, and get them properly replicated and available. It’s very mature. It’s actually the code that runs Rackspace “Cloud files”, but also Internap or Nephoscale “Cloud storage”. It scales horizontally using a share-nothing architecture, so you can add up hosts to cope up with the load. You deploy it using commodity hardware, and Swift replication takes care of the redundancy of your data. So you don’t need expensive SANs or RAID setups. Questions ?
< ClassBot> koolhead17 asked: any duration for support cycle? like ubuntu?
OpenStack is being distributed through downstream distributions, like Ubuntu. They all have their own support cycles, we don’t have a particular one. At this point we don'”t backport anything but critical issues and security issues into past releases. For long term support I advise that you opt for one of those distributions of OpenStack.
If there are no more questions, we’ll switch to Nova. Nova (Cloud Compute) is the biggest of the three current core projects. A few hours ago soren gave you the keys to it, I’ll repeat the essentials now. You can use it to run a cloud compute provider, like Amazon EC2 or Rackspace Cloud Servers. It allows your users to request a raw virtual machine (based on a disk image from a catalog). So for example users can request a VM with Ubuntu Server 11.04 on it, then access it using SSH, customize and run things on it. The architecture is a bit complex. We have several types of nodes, and you can run any number of each of them to cope with load. We have API nodes that receive requests, Scheduler nodes that assign workers… Network nodes that handle networking needs, Compute and Storage workers than handle VMs and block storage. Everything communicates using a RabbitMQ message queue. It’s *very* modular, so you have to choose how you want to deploy it. For example it supports 8 different virtualization technologies right now: QEMU, KVM, UML, LXC, Xen, Citrix XenServer, M$ HyperV and VMWare vSphere. Ubuntu by default should concentrate on two of those: KVM and LXC. Nova is still fast-moving, but it’s used in production at NASA Nebula cloud and it will be deployed this year to replace current Rackspace Cloud Servers software. Questions ?
< ClassBot> koolhead17 asked: how feasible it is to use openstack in production environment due to the nature of rapid change in architecture/feature every new release?
Depends on the component. Swift is very mature and slow moving now, you can certainly easily deploy it in production. Nova is still changing a lot, though that will calm down after Diablo. At this point running it in production requires a good effort to keep up. Some deployment options are more tested, and therefore more stable than others. So it is feasible… and will become more feasible as time passes. Any other question on Nova before we switch to Glance ?
Glance (OpenStack Image service) is the latest addition to the core projects family. It’s a relatively-simple project that handles VM image registration and delivery. So your users can use it to upload new disk images for use within Nova. It supports multiple disk formats and multiple storage backends. So disk images end up being stored in Swift or S3 (or locally if you can’t afford that). As far as stability is concerned, I’d say it’s on par with Nova, and maturing fast. Questions on Glance ? hah! crystal clear, I see.
So, what can *you*, Ubuntu user, do with it ? You can try it. We provide several distribution channels… Starting with 11.04, the latest OpenStack version is released in Ubuntu universe (And the “Diablo” release should be in main for 11.10). If you need something fresher (or running on LTS), you can use our PPAs: We have release PPAs (with 2011.2 “Cactus”) for 10.04 LTS, 10.10, 11.04 and Oneiric. We also have “last milestone” PPAs and “trunk” PPAs for the same Ubuntu releases (The “trunk” PPA contains packages built from the latest commit in the trunk branch !) See http://wiki.openstack.org/PPAs for more details.
You don’t need a lot of hardware to test it. You can actually deploy Nova + Glance on a single machine. It’s easier to use a real physical machine to be able to test virtualization correctly. Swift would prefer a minimum of 5 servers (to handle its redundancy)… but you can fake it to use the same machine, or you can (ab)use virtual machines to test it. That’s about all I had in mind for this presentation, so we can switch to general Q&A. Feel free to join our community: test, report bugs, propose branches fixing known issues. The best way to ensure it’s working for *your* use case is actually to try it and report bugs if it doesn’t :).
< ClassBot> _et asked: I am confused by the flow of control in the architecture diagram here http://ur1.ca/4sni7. Can you shed more light on how each component interacts with each other and how comm to and from outside flows?
Aw ok, let’s take an example: A call to create a server, run_instance in EC2-talk. You use a client library or CLI that will talk the EC2 or the OpenStack API. Your request will be received by a nova-api node. You can run any number of those, in order to cope with your load. nova-api will place your request in the queue. nova-scheduler pick up requests in the queue. Again, you can run any number of them in order to cope with your load. scheduler looks at your request and determines where it should be handled. Here, it determines which nova-compute host will actually run your VM. scheduler places back on the queue a message that says “compute node X should run this query”. nova-compute node X picks message in the queue. It retrieves disk image from Glance, starts your VM, updated DB so that if you run describ_instances you can see it’s running, etc. All the other arrows are for other types of requests
< ClassBot> _et asked: scheduler decides based on wat the load balancer says?
No, scheduler decides based on the scheduler algorithm you use. There are multiple scheduler algorithms available, and you can easily plug your own. The default ChanceScheduler is just a round robin
< ClassBot> BuZZ-T asked: are there any possibilities to automatically scale instances (e.g. by load)
That would be the role of tools above the IaaS layer. ScalR, Rightscale, etc all provide tools that monitor your IaaS and react accordingly, but then you tread into PaaS territory, and OpenStack is already very busy trying to cover the IaaS space.
http://wiki.openstack.org/ is a good starting point for all things. We also hang out in #openstack (support) and #openstack-dev (development) on Freenode ! Any other question ? Or let me know if any of my answers was incomplete or confusing.
< ClassBot> _et asked: how are IPs assigned? does compute do it thru scheduler? or is it the scheduler all by itself? ( if u ve time )
You define networks with scores of available IP addresses, and the network node just picks one that is available. There is an ambitious project to separate the network components into a set of projects to achieve complete and complex network virtualization. Check out Quantum, Melange and Donabe in Launchpad.
If there are no other questions, I’ll close this online event, since this is the last session. kim0 is not around, but asked me to thank all attendees. For further questions, feel free to hit #ubuntu-cloud. Where most of the people that presented hang out anyway :). Thank you all for making Ubuntu the best platform for the cloud, and on the cloud.
malariagen informatics 