2014-08-29

Improving Performance of OpenShift VM

In previous post I've described how to install OSEoD-2.1.4-2.x86_64.vmdk image to run OpenShift Enterprise 2.1 on your Fedora/RHEL/CentOS machine using libvirt. There was information about:

  • resizing VM image
  • creating VM itself
  • configuring LVM to actually use additional VM storage
  • configuring OSE to use larger gear sizes
  • configuring static IP
  • configuring dev machine

However, after actually installing Fuse cartridge using openshift-origin-cartridge-fuse-6.1.0.redhat.390-2.el6op.noarch.rpm package, the performance was painfully slow, I wasn't able to create applications using fuse cartridge. Fabric was starting in more than 8 minutes...

I did a lot of tweaking, configuring, restarting, etc. and finally got to:

The final solution

Here's a list of tips to get RPM-based fuse cartridge running.

Optimizing VM image

In this post there's is solution to optimize qcow2 storage. I took original vmdk image and converted it into qcow2 format with preallocation set:

qemu-img convert -f vmdk -O qcow2 -o preallocation=metadata \
   /opt/vm/OSEoD-2.1.4-2.x86_64.vmdk \
   /opt/vm/OSEoD-2.1.4-2.x86_64-5.qcow2

(This time I've neither resized the image nor done any lvm/fs resizing. However I can't tell now whether this impacted the performance...)

Configuring VM

On VirtIO Disk 1 Configuration page I set:

  • Disk bus: VirtIO
  • Cache mode: none

After this, applications based in fuse cartridge where installed in less than 1 minute (which included starting the Fabric and waiting for io.fabric8.api.FabricService!).

2014-08-19

OpenShift Enterprise and DNS resolution

After installing OpenShift Enterprise using local Virtual Machine managed by libvirt we can access the applications running on OSE from host machine simply by adding correct nameserver to /etc/resolv.conf.

This will however break DNS name resolution inside guest machine. The reason is the way how libvirt set's up networking for guests.

libvirt runs dnsmasq DHCP and caching DNS server and it's visible from guest machines (by default) under 192.168.122.1 IPv4 address. Here's /etc/resolv.conf from guest machine:

# Generated by NetworkManager
search openshift.example.com
nameserver 127.0.0.1     # local named daemon configured to resolve names under openshift.example.com domain
nameserver 192.168.122.1 # DNS server from libvirt. this by default uses host's /etc/resolv.conf

If you add nameserver OSE guest IP to host's /etc/resolv.conf you may be able to resolve *.openshift.example.com addresses from your host, but it will prevent guest from resolving DNS names. Guest will try:

  • 127.0.0.1 which resolves only names under openshift.example.com domain
  • 192.168.122.1 - host's dnsmasq daemon which uses /etc/resolv.conf and ... tries guests DNS server first which fails to resolve names

I wasn't able to configure libvirt to specify other options for dnsmasq. The simplest thing to do is to restart dnsmasq daemon specyfying alternative DNS servers found in your /etc/resolv.conf without OSE DNS nameserver:

# ps -ef | grep dnsmasq | grep -v grep
nobody   32399     1  0 11:01 ?        00:00:00 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf
# kill $(pgrep dnsmasq)
# /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --local <one of nameservers from /etc/resolv.conf>

After this I was able to resolve both *.openshift.example.com names from host machine and all names from OSE VM.

2014-08-14

OpenShift Enterprise on Fedora 20

Do you want to play with OpenShift Enterprise by Red Hat on your Virtual Machine? Do you want to use Fedora 20 (or similar) without using VirtualBox? Here's an instruction. Please treat it as notes from successful installation, not as a comprehensive and detailed installation guide.

I used instructions provided by Kurt Stam in Notes on Getting started with OpenShift.

What you need to get

You need single VM image. I used OSEoD-2.1.4-2.x86_64.vmdk.

What you should have

You should have RHEL/Fedora/CentOS like system (I used Fedora 20) with some additional packages:

  • virt-manager (and related) to create/manage Virtual machines
  • qemu-img (and relaated) to tweak the original VM image

Resizing the image

When you check the image, you can see it has 20GB of virtual size.

$ qemu-img info OSEoD-2.1.4-2.x86_64.vmdk 
image: OSEoD-2.1.4-2.x86_64.vmdk
file format: vmdk
virtual size: 20G (21474836480 bytes)
disk size: 7.0G

Let's make it bigger. Because it's not possible to resize vmdk image (only raw images can be cconverted with qemu-img), we have to convert-resize-convert it:

$ qemu-img convert -p -f vmdk -O raw OSEoD-2.1.4-2.x86_64.vmdk OSEoD-2.1.4-2.x86_64.raw
    (100.00/100%)
$ qemu-img resize OSEoD-2.1.4-2.x86_64.raw +40G
Image resized.
$ qemu-img convert -p -f raw -O vmdk OSEoD-2.1.4-2.x86_64.raw OSEoD-2.1.4-2.x86_64-2.vmdk
    (100.00/100%)

Now we have 60GB virtual storage. Let's create Virtual Machine then.

Creating Virtual Machine

We can now create new VM by importing our resized VM image. Create new VM in Virtual Machine Manager and select Import existing disk image:

Select your image and choose OS Type and Version:

Set required Memory (RAM) and CPUs:

Finish. Ensure to customize configuration before install:

On configuration screen check Copy host CPU configuration and click Begin Installation:

Congratulations: you've just installed (actually: started) RedHat Enterprise Linux 6.5 with OpenShift Enterprise 2.1.4!

Configuring disk space

When we log into new VM using openshift login, we can check free space:

[openshift@vm ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_vm1-lv_root
                       18G  6.4G   10G  39% /
tmpfs                 5.0M   76K  5.0M   2% /dev/shm
/dev/vda1             485M   39M  421M   9% /boot

Let's use all available (60GB) space for root mount point. We could use the method described by Kurt Stam using GParted, or we can use LVM tools to resize lv_root logical volume. We'll do the latter.

Create new partition (we could use cfdisk to get more appealing visual experience or fdisk) with 8e type (Linux LVM). The resulting list of partitions should look like this:

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *           3        1018      512000   83  Linux
/dev/vda2            1018       41611    20458496   8e  Linux LVM
/dev/vda3           41611      124831    41943040   8e  Linux LVM

Do not worry about Partition N does not end on cylinder boundary warnings. It may require to reboot the system to see /dev/vda3 device.

Add new physical volume:

[root@vm ~]# pvcreate /dev/vda3
  Physical volume "/dev/vda3" successfully created
[root@vm ~]# pvs
  PV         VG     Fmt  Attr PSize  PFree 
  /dev/vda2  vg_vm1 lvm2 a--  19.51g     0 
  /dev/vda3         lvm2 a--  40.00g 40.00g

Currently volume group vg_vm1 contains single physical volume and has VSize equal to 19.51g:

[root@vm ~]# vgs
  VG     #PV #LV #SN Attr   VSize  VFree
  vg_vm1   1   2   0 wz--n- 19.51g    0 

Add /dev/vda3 to the volume group:

[root@vm ~]# vgextend vg_vm1 /dev/vda3 
  Volume group "vg_vm1" successfully extended

This changed the size of volume group and the status of physical volumes:

[root@vm ~]# vgs
  VG     #PV #LV #SN Attr   VSize  VFree 
  vg_vm1   2   2   0 wz--n- 59.50g 40.00g
[root@vm ~]# pvs
  PV         VG     Fmt  Attr PSize  PFree 
  /dev/vda2  vg_vm1 lvm2 a--  19.51g     0 
  /dev/vda3  vg_vm1 lvm2 a--  40.00g 40.00g

Now we should resize logical volume:

[root@vm ~]# lvscan 
  ACTIVE            '/dev/vg_vm1/lv_root' [17.51 GiB] inherit
  ACTIVE            '/dev/vg_vm1/lv_swap' [2.00 GiB] inherit
[root@vm ~]# lvextend -l +100%FREE /dev/vg_vm1/lv_root
  Extending logical volume lv_root to 57.50 GiB
  Logical volume lv_root successfully resized
[root@vm ~]# lvscan 
  ACTIVE            '/dev/vg_vm1/lv_root' [57.50 GiB] inherit
  ACTIVE            '/dev/vg_vm1/lv_swap' [2.00 GiB] inherit

The last step is to actually resize the filesystem to expand all available logical volume size. Now we have only 9.9G free space:

[root@vm ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_vm1-lv_root
                       18G  6.5G  9.9G  40% /
tmpfs                 2.9G  220K  2.9G   1% /dev/shm
/dev/vda1             485M   39M  421M   9% /boot

It's easy to resize live partition under LVM to get 48G free space:

[root@vm ~]# resize2fs -p /dev/mapper/vg_vm1-lv_root
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_vm1-lv_root is mounted on /; on-line resizing required
old desc_blocks = 2, new_desc_blocks = 4
Performing an on-line resize of /dev/mapper/vg_vm1-lv_root to 15074304 (4k) blocks.
The filesystem on /dev/mapper/vg_vm1-lv_root is now 15074304 blocks long.

[root@vm ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_vm1-lv_root
                       57G  6.5G   48G  13% /
tmpfs                 2.9G  220K  2.9G   1% /dev/shm
/dev/vda1             485M   39M  421M   9% /boot

Now we're ready to configure OpenShift Enterprise itself.

Configuration of OpenShift Enterprise

As described here we can configure Openshift Enterprise to allow medium or even large gear sizes. Do the customization only to /etc/openshift/broker.conf file and alter demo user. After the configuration we should get this (on Openshift VM):

[root@vm ~]# oo-admin-ctl-user -l demo


User demo:
                            plan: 
                consumed domains: 0
                     max domains: 10
                  consumed gears: 0
                       max gears: 100
    max tracked storage per gear: 0
  max untracked storage per gear: 0
                       max teams: 10
viewing all global teams allowed: true
                      gear sizes: small, medium, large
            sub accounts allowed: false
private SSL certificates allowed: true
              inherit gear sizes: false
                      HA allowed: true

The above won't allow us however to install applications using medium or large gear sizes, because there's only one default district with small gear size.
We can however create entirely new district:

[root@vm ~]# oo-admin-ctl-district -c create -p "large" -n default
Successfully created district: 53f1f2e2e659c5a904000001

{"_id"=>"53f1f2e2e659c5a904000001",
 "uuid"=>"53f1f2e2e659c5a904000001",
 "available_uids"=>"<6000 uids hidden>",
 "name"=>"default",
 "platform"=>"linux",
 "gear_size"=>"large",
 "available_capacity"=>6000,
 "max_uid"=>6999,
 "max_capacity"=>6000,
 "active_servers_size"=>0,
 "updated_at"=>2014-08-18 12:34:43 UTC,
 "created_at"=>2014-08-18 12:34:43 UTC}

Deactivate & remove vm.openshift.example.com node from default-small district:

[root@vm ~]# oo-admin-ctl-district -c deactivate-node -n default-small -i vm.openshift.example.com
Success for node 'vm.openshift.example.com'!
...
[root@vm ~]# oo-admin-ctl-district -c remove-node -n default-small -i vm.openshift.example.com
Success for node 'vm.openshift.example.com'!
...

Configure node profile to allow large gear sizes. To do it, change /etc/openshift/resource_limits.conf to contain:

...
#
# Standard Profile
#
node_profile=large
...

And restart the broker:

[root@vm ~]# /etc/init.d/openshift-broker restart

Finally add the vm.openshift.example.com node to the new district (see that the gear_size parameter is set to large):

[root@vm ~]# oo-admin-ctl-district -c add-node -n default -i vm.openshift.example.com
Success for node 'vm.openshift.example.com'!


{"_id"=>"53f1f2e2e659c5a904000001",
 "active_servers_size"=>1,
 "available_capacity"=>6000,
 "available_uids"=>"<6000 uids hidden>",
 "created_at"=>2014-08-18 12:34:43 UTC,
 "gear_size"=>"large",
 "max_capacity"=>6000,
 "max_uid"=>6999,
 "name"=>"default",
 "platform"=>"linux",
 "servers"=>
  [{"_id"=>"53f1f76ce659c5080c000001",
    "active"=>true,
    "name"=>"vm.openshift.example.com",
    "unresponsive"=>false}],
 "updated_at"=>2014-08-18 12:34:43 UTC,
 "uuid"=>"53f1f2e2e659c5a904000001"}

Now we can remove previous district:

[root@vm ~]# oo-admin-ctl-district -c destroy -n default-small
!!!! WARNING !!!! WARNING !!!! WARNING !!!!
You are about to delete the default-small district.

This is NOT reversible, all remote data for this district will be removed.
Do you want to delete this district (y/n): y
Successfully deleted district: default-small

See the documentation for more information.

Configuration of networking

This is not of course necessary to get working environment, but let's switch from DHCP to static IP networking.

This is the content of /etc/sysconfig/network:

NETWORKING=yes
HOSTNAME=vm.openshift.example.com
GATEWAY=192.168.122.1

And /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE="eth0"
BOOTPROTO="none"
IPADDR="192.168.122.105"
NETMASK="255.255.255.0"
NETWORK="192.168.122.0"
PEERDNS="yes"
DNS1="127.0.0.1"
DNS2="192.168.122.1"
IPV6INIT="yes"
MTU="1500"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"
UUID="0c56a8e2-4cda-4779-8cff-213584f3fc17"

The above configuration will use local DNS server as first DNS server and libvirt-provided one as second DNS server.

Ensure that /var/named/dynamic/openshift.example.com.db contains correct address for vm name:

...
ns1 IN A 127.0.0.1
vm A 192.168.122.105
activemq ...
...

And change /etc/openshift/node.conf to contain correct IP address of the node:

...
PUBLIC_HOSTNAME=vm.openshift.example.com
PUBLIC_IP=192.168.122.105
BROKER_HOST=vm.openshift.example.com
...

Configuration of developer machine

Networking

We will use host machine as the development machine, from which we will execute rhc command and manage OpenShift domains and application using web browser. First thing is to be able to access the machine using logical host names.

After switching to static-IP based networking, we can use guest's IP (here: 192.168.122.105) as DNS server - let's add it to host's /etc/resolve.conf:

$ cat /etc/resolv.conf 
# Generated by NetworkManager
...
nameserver 192.168.122.105
nameserver ...
nameserver ...
...

After this, we can resolve names under openshift.example.com domain:

$ dig vm.openshift.example.com

; <<>> DiG 9.9.4-P2-RedHat-9.9.4-15.P2.fc20 <<>> vm.openshift.example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49858
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;vm.openshift.example.com. IN A

;; ANSWER SECTION:
vm.openshift.example.com. 1 IN A 192.168.122.105

;; AUTHORITY SECTION:
openshift.example.com. 1 IN NS ns1.openshift.example.com.

;; ADDITIONAL SECTION:
ns1.openshift.example.com. 1 IN A 127.0.0.1

;; Query time: 0 msec
;; SERVER: 192.168.122.105#53(192.168.122.105)
;; WHEN: Mon Aug 18 11:51:28 CEST 2014
;; MSG SIZE  rcvd: 103

rhc utility

When connected to vm.openshift.example.com VM, we will see browser page after logging in as openshift user. There's great deal of information about further steps. There's information about accessing OpenShift Enterprise console, using JBoss Developer Studio and rhc utility.

We won't use the VM as developer machine - let's use host machine for these tasks. To access OpenShift Enterprise from Fedora 20, we need rubygem-rhc package (although the package that provides rhc tool on RHEL is rhc-1.24.3.1-1.el6op.noarch).

Let's setup local environment:

$ rhc setup --server vm.openshift.example.com -l demo

After executing this command, we'll be asked to:

  • accept self-signed certificate
  • provide password for demo user
  • generate a token (stored in ~/.openshift/ directory)
  • upload existing (or generate new first) SSH keys to OpenShift Enterprise server
  • create domain (namespace) - let's create it later (using rhc create-domain command)

The above ends client side configuration. Now we're ready to create domain and applications inside OpenShift Enterprise.

Domain and first application

Using rhc create-domain or by pointing the browser to https://vm.openshift.example.com/console/settings URL, we can set a namespace for applications.

On https://vm.openshift.example.com/console/applications page, we can create applications using available cartridges and allowed gear sizes.

I could create for example php-5.4 based application using large gear size:

Verifying the installation

Let's clone the git repository of application and add simple phpinfo() test:

$ git clone ssh://53f1f8aae659c5d75d000001@php-test.openshift.example.com/~/git/php.git/
Cloning into 'php'...
remote: Counting objects: 17, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 17 (delta 0), reused 17 (delta 0)
Receiving objects: 100% (17/17), 17.95 KiB | 0 bytes/s, done.
Checking connectivity... done.
$ mv index.php index2.php 
$ cat > index.php
<?php
phpinfo();
?>
^D
$ git commit -a -m 'Checking phpinfo()'
[master fc57849] Checking phpinfo()
 1 file changed, 3 insertions(+), 274 deletions(-)
 rewrite index.php (100%)
$ git push origin master 
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 309 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Stopping PHP 5.4 cartridge (Apache+mod_php)
remote: Waiting for stop to finish
remote: Waiting for stop to finish
remote: Building git ref 'master', commit fc57849
remote: Checking .openshift/pear.txt for PEAR dependency...
remote: Preparing build for deployment
remote: Deployment id is bcbd60fe
remote: Activating deployment
remote: Starting PHP 5.4 cartridge (Apache+mod_php)
remote: Application directory "/" selected as DocumentRoot
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://53f1f8aae659c5d75d000001@php-test.openshift.example.com/~/git/php.git/
   0224a1e..fc57849  master -> master

Here's the final result after pointing your browser to http://php-test.openshift.example.com/:

Thank you very much for your attention.