CentOS 6.4 – Installing OpenNebula 4.2 (Flame)

This is my first post in a long time…. my apologies I’ve been quite busy for a few months and I’ve had no time left to write new posts or answer your comments, so thanks for your patience and understanding for all of you that have sent me emails or comments.

Today I’m writing an easy post… maybe it’s too late as OpenNebula 4.4 is so close but if you’re looking for a post about OpenNebula 4.2 and CentOS 6.4 it could help you.

OpenNebula provides an official quickstart guide for CentOS and other platforms so you may want to check them first, I keep writing these posts as they are my installation notes and maybe they are useful to you.

I’m executing the following commands as root.

1. Install the EPEL repository

rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

2. Add OpenNebula’s repository – [ Reference OpenNebula’s site ]

# cat << EOT > /etc/yum.repos.d/opennebula.repo
[opennebula]
name=opennebula
baseurl=http://downloads.opennebula.org/repo/CentOS/6/stable/\$basearch
enabled=1
gpgcheck=0
EOT

3. Let’s check that EPEL and OpenNebula repositories are ready

# yum search opennebula
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.mirror.xtratelecom.es
* epel: fedora.aau.at
* extras: centos.mirror.xtratelecom.es
* updates: centos.mirror.xtratelecom.es
=========================== N/S Matched: opennebula ============================
opennebula-common.x86_64 : Provides the OpenNebula user
opennebula-context.x86_64 : Configures a Virtual Machine for OpenNebula
opennebula-flow.x86_64 : Manage OpenNebula Services
opennebula-gate.x86_64 : Transfer information from Virtual Machines to: OpenNebula
opennebula-java.x86_64 : Java interface to OpenNebula Cloud API
opennebula-node-kvm.x86_64 : Configures an OpenNebula node providing kvm
opennebula-ruby.x86_64 : Provides the OpenNebula Ruby libraries
opennebula-server.x86_64 : Provides the OpenNebula servers
opennebula.x86_64 : Cloud computing solution for Data Center Virtualization
opennebula-ozones.x86_64 : Tool for administering
opennebula-sunstone.x86_64 : Browser based UI and public cloud interfaces.
Name and summary matches only, use "search all" for everything.

4. Install the packages you need for your OpenNebula installation architecture. In my case I’m running OpenNebula in a single machine so I’ll install opennebula-server and opennebula-sunstone

# yum install opennebula-server opennebula-sunstone

Warning: if it’s the first time using the EPEL repository you’ll need to import its GPG key so answer yes to the following question:

# Is this ok [y/N]: y

5. If you are going to use KVM virtualization, install the package opennebula-node-kvm in the machine that’s going to act as the host offering virtualization resources. This package will install qemu-kvm, libvirt and all the CentOS packages needed for virtualization. I’m using a single machine so my machine will act as a front-end, host and datastore. Please read the official documentation to understand which are the basic components for OpenNebula.

6. Let’s start the opennebula service

# service opennebula start
Starting OpenNebula daemon: [ OK ]

7. The opennebula’s sunstone service provides the graphical interface for opennebula. By default it listens on the 127.0.0.1:9869 address/port so if you want to listen in a different address edit the :host: directive in the /etc/one/sunstone-server.conf

For example if you want to listen in the 192.168.1.70 address change the host directive and save the file.

# Server Configuration
#
:host: 192.168.1.70

The service is started using the following command:

# service opennebula-sunstone start
Starting Sunstone Server daemon: VNC proxy started
sunstone-server started [ OK ]

If you change the IP address where Sunstone is listening, remember to add a firewall rule (also remember to save that rule)

#iptables -I INPUT -m tcp -p tcp --dport 9869 -m state --state=NEW -j ACCEPT
#service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]

OK, that’s only the installation part, if you want to run a VM you’ll need to configure a host please read this old blog post

Cheers! … I’ll be back!

Installing Suricata IDS from source – CentOS 6.4

This post has been updated: 31/05/2013. I’ve tested the installation steps with CentOS 6.4. It also works with CentOS 6.3

Today I’ll compile Suricata in a clean CentOS 6.4 server. I’m not compiling it with PFRING support (it would increase performance). Ok, hands on:

  1. You’ll need the EPEL repository, see the step 2 from this post.
  2. I’ll install Development Tools group and some packages needed by Suricata.
    yum groupinstall "Development Tools"
    yum install pcre-devel libyaml-devel libnet-devel libpcap-devel libcap-ng-devel file-devel zlib-devel
  3. Download Suricata from its web page. Move the tar.gz file to a suitable directory, in my case I’ve chosen /opt directory.
  4. Uncompress it (I’m compiling 1.4.3 version) and configure the compilation. I’ve set some prefixes and directories and added the  –disable-gccmarch as I was having problems (Illegal Instruction) when executing Suricata on my QEMU/KVM virtual machine (the post that helped me).
    tar xvfz suricata-1.4.3.tar.gz
    cd suricata-1.4.3
    ./configure --prefix=/usr/ --sysconfdir=/etc/ --localstatedir=/var/ --disable-gccmarch-native
    
  5. Ok. Now let’s use make, make install and if you want Suricata to create a config file and download rules from Emerging Threats use make install-full.
    make
    make install
    make install-full
    ldconfig
  6. And finally let’s try to execute suricata command.
    [root@sherlock ~]# suricata -V
    This is Suricata version 1.4.3 RELEASE

I read in ntop’s web page that virtual pf_ring would improve performance dramatically for virtualization environments like KVM but I have no money now to pay for the fee (if you want to donate let me know :-D) so I’ll try to use it for a few minutes as they suggest for evaluation purposes.

As always I appreciate any comments to improve the quality of this post. Enjoy!

Resizing a QEMU KVM Linux image using virt-resize in CentOS 6.4

Hi,
today my colleague Geoff told me that he was trying to resize an OpenNebula image. As I had never done that before, I started to review OpenNebula documentation and I found this email in the OpenNebula mail list (please suscribe!, it’s really useful!). It seems that resizing is not yet supported (maybe I’m wrong and there’s another solution!) but thanks to this issue information I found virt-resize.

This is virt-resize description: a tool which can resize a virtual machine disk, making it larger or smaller overall, and resizing or deleting any partitions contained within. Looks promising! The virt-resize information page is full of examples so it has been easy to start using it.

Ok this is what I’ve tested and please remember proceed with caution I’m no responsible of any damage caused by following this steps and try to read the documentation first I’ve just used that info :-D.

  1. I’ve installed the libguestfs-tools package in my OpenNebula host running CentOS: yum install libguestfs-tools
  2. I halted one VM as I want to increase the size of its main disk image (the VM can’t be running!)
  3. Using Sunstone I’ve found where the disk is located inside the datastore: Virtual Machines -> Select the VM -> Template tab -> Disk section (e.g: /var/lib/one/datastores/1/1a8a07d0382566a89afd96a134eb04cf)
  4. Now I’ve inspected the image file with the virt-filesystems command.
    [root@haddock ~]# cd /var/lib/one/datastores/1/
    [root@haddock 1]# virt-filesystems --long --parts --blkdevs -h -a 1a8a07d0382566a89afd96a134eb04cf
    Name Type MBR Size Parent
    /dev/sda1 partition 83 500M /dev/sda
    /dev/sda2 partition 8e 20G /dev/sda
    /dev/sda device - 20G -
  5. I want to have a 30GB size for my /dev/sda2 partition, so I create a file called newdisk with the truncate command:
    truncate -s 30G newdisk
  6. I create a backup of the OpenNebula image:   cp 1a8a07d0382566a89afd96a134eb04cf tmp
  7. Ok! This is very important I have a logical volume called /dev/vg_moriarty/lv_root, so today I’m not only resizing the /dev/sda2 partition I also want the logical volume to use the new extra space:
    virt-resize tmp newdisk --expand /dev/sda2 --LV-expand /dev/mapper/vg_moriarty-lv_root
  8. This is a screenshot showing the resizing progress.
    resizing
  9. Once the resizing progress is finished I substitute the original image with the newdisk file and I change the permissions:
    mv newdisk 1a8a07d0382566a89afd96a134eb04cf
    chown oneadmin:oneadmin 1a8a07d0382566a89afd96a134eb04cf
  10. I check if the partitions have been modified before running the VM. It seems that the /dev/sda2 is now 30GB.
  11. [root@haddock 1]# virt-filesystems --long --parts --blkdevs -h -a 1a8a07d0382566a89afd96a134eb04cf
    Name Type MBR Size Parent
    /dev/sda1 partition 83 500M /dev/sda
    /dev/sda2 partition 8e 30G /dev/sda
    /dev/sda device - 30G -
  12. Ok. I start the VM to check if it boots and the logical volume is fine (28GB!)
    moriarty_after
  13. It works!! Thanks to OpenNebula’s mailing list (Simon Boulet, Ricardo Duarte, Ruben S. Montero…) and the libguestfs creators I’ve found a way to resize a Linux image.

Please if you know a better way or OpenNebula has already a way to modify the size please let me know, I want this post to be useful for the community.

Enjoy!

P.S: Thanks Geoff!

Tip: Installing Windows Server 2012 (Evaluation) in OpenNebula 4 with KVM

Hi,
I’d like to evaluate Windows Server 2013 so I’ve decide to create a VM in my OpenNebula 4 lab. I’ve read in this page that I’d need VirtIO signed drivers for Windows in order to detect the virtual hard disk which will store the OS.

I’ve created a template with the following storage (DISKS):

  1. A CDROM (PREFIX hd) with the Windows Server 2012 ISO
  2. A CDROM (PREFIX hd) with the Windows stable drivers ISO from Fedora
  3. A OS HDD virtual disk: DRIVER: raw and PREFIX: vd

After instantiating the template, Windows starts the installation. When Windows warns you that no disk is found you can load the driver from the second CDROM.

Imagen

Then you have to browse the CDROM and select the SCSI RedHat VirtIO SCSI controller (WLH/AMD64 folder). Now the Virtual Hard Disk has been detected and you can install the Operating System.

Imagen

Enjoy!

OpenNebula 4 with KVM and Openvswitch using only one server

As I’ve only one server, I’m forced to install OpenNebula and KVM virtualization in the same machine. If you want to know how I configured and installed openvswitch read my previous posts.

Let’s begin installing some packages:

yum install qemu-kvm qemu-kvm-tools libvirt virt-manager

Install the opennebula-node-kvm rpm package (read my previous post for more information) as it’ll configure for you  qemu and a policy allowing oneadmin user to use the virtualization api.

yum localinstall opennebula-node-kvm-4.0.1-1.x86_64.rpm

Start the libvirtd service and configure it to start at boot

#/etc/init.d/libvirtd start
Starting daemon libvirtd: [ OK ]
# chkconfig libvirtd on

Warning: if you’re using SELinux run this command so the authorized keys is accesible for passwordless login using ssh. Also, I’ve change the context for the /var/lib/one/datastore directory to avoid a Permission Denied error (/var/lib/one/datastores/0/0/disk.0: Permission denied) when trying to run a VM with KVM.

chcon -v --type=ssh_home_t /var/lib/one/.ssh/authorized_keys
chcon -R --type=virt_image_t /var/lib/one/datastores

Create the /var/tmp/one directory and change the ownership

# mkdir /var/tmp/one
# chown oneadmin:oneadmin /var/tmp/one

If you’re using server names, you have to be sure that there’s an entry in your DNS or /etc/hosts for the server name, e.g I have an entry in the /etc/hosts for my server haddock.macto.local

Now as the oneadmin user, let’s create the host with a KVM hypervisor and openvswitch.and check that no errors are shown. Also try to open a ssh to check that no password is used, this will insert your host in the known_hosts file and will prevent ” Host key verification failed” error when monitoring your host.

# su oneadmin
$ ssh oneadmin@haddock.macto.local
The authenticity of host 'haddock.macto.local(192.168.1.20)' can't be established.
RSA key fingerprint is ....
Are you sure you want to continue connecting (yes/no)? yes
$ exit

$ onehost create haddock.macto.local -i kvm -v kvm -n ovswitch
ID: 0
$ onehost list
ID NAME CLUSTER RVM ALLOCATED_CPU ALLOCATED_MEM STAT
0 haddock.macto.l - 0 0 / 200 (0%) 0K / 5.6G (0%) on

OK status is on, and my host looks good in Sunstone GUI. Perfect.

If “err” is shown after executing the onehost list command, check /var/lib/one/oned.log for errors. I was having the following error because I haven’t installed opennebula-node-kvm after libvirt installation: “error: authentication failed: Authorization requires authentication but no agent is available”

If you’re running openvswitch you can avoid the following errors editing the sudoers file.

  • sudo: sorry you must have a tty to run sudo
  • sudo: Error deploying virtual machine: sudo: no tty present and no askpass program specified

Edit the sudoers file with visudo and comment the line “Defaults requiretty” , then add the following lines at the end of the file:

oneadmin ALL = NOPASSWD: /sbin/iptables
oneadmin ALL = NOPASSWD: /sbin/ebtables
oneadmin ALL = NOPASSWD: /usr/bin/ovs-vsctl
oneadmin ALL = NOPASSWD: /usr/bin/ovs-ofctl

I’ve also found sometimes this error:

WARNING **: Error connecting to bus: org.freedesktop.DBus.Error.FileNotFound: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

I decided to reboot the machine and the monitor status changed to on.

Tomorrow I’ll explain how to run a virtual machine in OpenNebula.

OpenNebula – Using SPICE for VM graphics

OpenNebula 4’s Sunstone GUI comes with NoVNC so you can open VNC connections against your VM if you configure the graphics section. NoVNC is awesome and offers secure websocket connections. However, I sometimes have problems with VNC’s mouse pointer position (it’s not a problem with OpenNebula or NoVNC just a qemu/vnc issue) and the trick about using tablet usb pointer shown in this discussion solves it.

If you want a different way to connect to your KVM guests you can try SPICE. If you’re curious about using SPICE in CentOS/RHEL please read Timothy Lee’s howto which is a great guide and helped me to understand how to use remote-viewer and what packages should I need.

Ok hands on. In the OpenNebula VM template I just set:

  • GRAPHICS=[KEYMAP=”es”,TYPE=”SPICE”,LISTEN=”0.0.0.0″]

If you wish to specify a port in the graphics section you have the explanation In OpenNebula’s documentation. If no port is specified OpenNebula will use the VNC_BASE_PORT variable set in /etc/one/oned.conf ( 5900 ) + your VM’s id e.g my VM has ID 7 so the SPICE/VNC port will be 5907.

A Spice server connection opened by qemu-kvm will be listening in any address (0.0.0.0) but in order to connect from a remote host I need an iptables rule e.g:

  • iptables -I INPUT -m tcp -p tcp –dport 5907 -m state –state=NEW,ESTABLISHED,RELATED -j ACCEPT
  • service iptables save

From my Fedora desktop I have to install virt-viewer and spice-client:

  • yum install virt-viewer spice-client

And finally I open a spice connection with remote-viewer:

  • remote-viewer spice://haddock.macto.local:5907 &

A new window is shown. Enjoy!

OpenNebula 4 Beta – running a virtual machine

Today we’re going to run a virtual machine using the KVM hypervisor and an openvswitch based virtual network. I’ve recorded a screencast so you can follow the steps easily. I’m going to configure a virtual network using openvswitch and I’ll test connectivity pinging my router 192.168.1.1.

Before the screencast a few comments:

  • I’m going to add a rule in iptables allowing traffic for VNC connection using the following commands:
iptables -I INPUT -p tcp --dport 29876 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
service iptables save
  • I’m using the ttylinux image from the OpenNebula market as it is small. I’m skipping the “Create image” step so image downloading time doesn’t affect the screencast.
  • The virtual machine template will run with 512 MB of RAM and it’ll use 50% of a CPU (maximum).
  • A virtual network called net0 is created with a fixed range (192.168.1.21,192.168.1.22,192.168.1.23) though just one IP address would be needed.
  • ttylinux image is ready for contextualization so an IP address from the net0 pool will be automatically assigned after adding a NIC to the template. My openvswitch bridge is named br0.

And finally here is the video:

OpenNebula : VNC Access

I will repeat it again, OpenNebula documentations is your friend. This is the documentation section which I’ve followed to enable VNC/Console for my VM:

http://opennebula.org/documentation:archives:rel3.0:sunstone

When creating the virtual machine template I’ve set the following values in the Add Graphics section:

  • Graphics Type: VNC
  • Listen IP: 0.0.0.0
  • Port:
  • Password:
  • Keymap:

If you want to open a vnc session in your virtual machine, you’ll need to install first noVNC using the install script available:

[root@haddock one]# cd /usr/share/one/
[root@haddock one]# ./install_novnc.sh
Downloading noVNC latest version...
######################################################################## 100,0%
Extracting files to temporary folder...
Installing Sunstone client libraries in /usr/lib/one/sunstone/public...
Installing SelfService client libraries in /usr/lib/one/ruby/cloud/occi/ui/public...
Downloading Websockify VNC proxy files
######################################################################## 100,0%
######################################################################## 100,0%
Backing up and updating /etc/one/sunstone-server.conf with new VNC proxy path...
Backing up and updating /etc/one/occi-server.conf with new VNC proxy path...
Installation successful

I’ve added a rule in my firewall so I can connect to the VNC port (29876 is the default, read the documentation section I’ve linked where an explanation is provided):

iptables -I INPUT -p tcp --dport 29876 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT

Maybe I can configure this rule somewhere in OpenNebula, I’ll investigate it.

And here is a screenshot of a vnc session!

VNC_connection

OpenNebula – KVM QEMU could not open disk image disk.0: Permission denied

I’ve been trying to figure out why this error was logged when trying to submit and run a virtual machine in OpenNebula with QEMU and KVM: “could not open disk image disk.0 Permission denied”. SELinux was complaining about dac_override and dac_read_search.

I thought it was a problem with security contexts as /var/lib/one directory is labelled with var_lib_t security context and QEMU would expect a virt_image_t context or svirt_image_t to run the virtual machine (libvirt provides documentation about QEMU and SELinux in this URL http://libvirt.org/drvqemu.html)

I changed the security context for the /var/lib/one/datastores directory:

chcon -R --type=virt_image_t /var/lib/one/datastores

Despite the security context change the error persisted, thankfully I found this message in the OpenNebula mail list:

http://lists.opennebula.org/pipermail/users-opennebula.org/2010-September/012877.html

I finally get rid of the permission denied editing my /etc/libvirt/qemu.conf and restarting the livirtd daemon (service libvirtd restart)

user="oneadmin"
group="oneadmin"
dynamic_ownership = 0

I’m still being able to run OpenNebula and SELinux, let’s see if I can keep SELinux enabled in my following tests.

Unable to add bridge brX port vnetX: Invalid argument or Operation not supported

This week I run one VM in my OpenNebula/KVM host, but as I like to shutdown my physical host every day (I don’t like to waste electricity when I’m not playing with my HP N40L), I encountered a problem when the host was up again.

The VM was in FAILED state and I found the following log when OpenNebula/KVM tried to run the VM:

error: Unable to add bridge br0 port vnet0: Invalid argument

Openvswitch wrote this log in /var/log/messages:

ovs-vsctl: 00002|vsctl|ERR|cannot create a port named vnet0 because a port named vnet0  already exists on bridge br0

It was clear that there was a problem with my openvswitch. I maybe wrong, (please understand that I’m learning how to use all this stuff ) but when OpenNebula oned daemon is stopped when my physical host is going to be halted, the virtual ports are not removed from the bridge so it’s not possible to add them when resuming operations (the virtual port is indeed removed if you first stop the VM).

This discussion helped me to understand the problem. According to that thread, Libvirt >= 0.9.11 seems to deal with abrupt restarts more nicely and it suggests not to use brcompatd mode.

If I remove –brcompat from my initd script, OpenNebula/KVM will show a different message

error: Unable to add bridge br0 port vnet1: Operation not supported

Although I compiled Libvirt 0.10.1, so no bridge compatibility mode should be needed, OpenNebula’s documentation states that “It is also necessary to install the Open vSwitch compatibility layer for Linux bridging“, so no choice for me!, I need to add –brcompat.

I’ve solved this problem using –delete-bridges (read the discussion) in my init.d script, so every bridge is removed from openvswitch’s database and, therefore, any port created by OpenNebula/KVM is also removed. Important: I’ve included the ovs-vsctl commands to add the bridge again so the network interface is ready to be used.

start() {
$exec start --system-id=random --brcompat --delete-bridges
retval=$?
[ $retval -eq 0 ] && touch $lockfile
/usr/local/bin/ovs-vsctl add-br br0
/usr/local/bin/ovs-vsctl add-port br0 eth0
return $retval
}

I’m sure it’s only a problem with my server, but if you happen to have the same problem please leave a comment if you find a better or cleaner solution.

Enjoy!