Wednesday, June 26, 2013

How to create a VM on Nutanix Cluster running KVM ?

I will go in depth on configuration modification done for running Nutanix cluster on KVM later.
(if you are interested, i like this doc

KVM Architecture Overview - Google Drive )

For now, let us take a Nutanix cluster running KVM and create VM on it.


[root@NTNX-12AM2K480036-A Create]# lsb_release   - vmware -v
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch

[root@NTNX-12AM2K480036-A Create]# virsh nodeinfo
CPU model:           x86_64
CPU(s):              24
CPU frequency:       1600 MHz
CPU socket(s):       1
Core(s) per socket:  6
Thread(s) per core:  2
NUMA cell(s):        2
Memory size:         49486468 KiB

virsh sysinfo (smbiosDump) - dmidecode on linux works as well.
<sysinfo type='smbios'>
  <bios>
    <entry name='vendor'>American Megatrends Inc.</entry>
    <entry name='version'>2.1b      </entry>
    <entry name='date'>10/28/2011</entry>
    <entry name='release'>8.16</entry>
  </bios>

Step 1.

- Login to nutanix Controller VM and Create Iscsi disk on Nutanix Container ( ncli ctr ls)
a. ncli vdisk create name=kvm-training-disk9 ctr-name=xyz  max-capacity=16

ncli> vdisk ls names=kvm-training-disk9
    Name                      : kvm-training-disk9
    Container ID              : 779
    Max Capacity              : 16 GB (17,179,869,184 bytes)
    ISCSI Target              : iqn.2010-06.com.nutanix:kvm-training-disk9-e3878625
    ISCSI LUN                 : 0









b. On KVM terminal - Verify that you are able to see the iscsi targets.

[root@NTNX-12AM2K480036-A ~]# sudo  iscsiadm -m discovery -t sendtargets -p 192.168.5.2:3260|egrep "iso|disk9" ( esxcfg-scsidevs -m)
192.168.5.2:3260,1 iqn.2010-06.com.nutanix:gasmith-training-cdrom-centos-6.4-x86_64-bin-dvd1.iso-bca6c6aa
192.168.5.2:3260,1 iqn.2010-06.com.nutanix:CentOS-6.4-x86_64-bin-DVD1.iso-c0e9bd87
192.168.5.2:3260,1 iqn.2010-06.com.nutanix:kvm-training-disk9-e3878625


c. Define the pool - same as creating datastore vmkfstools -C

 virsh pool-define-as --name kvm-training-disk9 --type iscsi --source-host 192.168.5.2 \
--source-dev iqn.2010-06.com.nutanix:kvm-training-disk9-e3878625 \
--target /dev/disk/by-path
Pool kvm-training-disk9 defined

where name can be specific to VM name,  192.168.5.2 is internal CVM IP, source dev is iqn name
of the iscsi lun, and  have it defined in /disk/by-path.

[root@NTNX-12AM2K480036-A by-path]# cd /dev/disk/by-path
root@NTNX-12AM2K480036-A by-path]# ls
ip-192.168.5.2:3260-iscsi-iqn.2010-06.com.nutanix:CentOS-6.4-x86_64-bin-DVD1.iso-c0e9bd87-lun-0
ip-192.168.5.2:3260-iscsi-iqn.2010-06.com.nutanix:kvm-training-disk9-e3878625-lun-0

d. Activate the pool

[root@NTNX-12AM2K480036-A ~]# virsh pool-list
Name                 State      Autostart
-----------------------------------------
CentOS-6.4.iso       active     no
default              active     yes
(it shows only active pools)


[root@NTNX-12AM2K480036-A ~]# virsh pool-list --all -- shows all the pool (esxcfg-scsidevs )
Name                 State      Autostart
-----------------------------------------
CentOS-6.4.iso       active     no
default              active     yes
kvm-training-disk9   inactive   no


Activate the pool
virsh # pool-start kvm-training-disk9
Pool kvm-training-disk9 started

Autostart the pool if there is a reboot
virsh # pool-autostart kvm-training-disk9
Pool kvm-training-disk9 marked as autostarted

virsh # pool-list ( esxcfg-scsidevs -m)
Name                 State      Autostart
-----------------------------------------
CentOS-6.4.iso       active     no
default              active     yes
kvm-training-disk9   active     yes

Verify the config
[root@NTNX-12AM2K480036-A ~]# virsh pool-dumpxml kvm-training-disk9
<pool type='iscsi'>
  <name>kvm-training-disk9</name>
  <uuid>3e42d29d-9037-1faa-12e1-af450904b5ab</uuid>
  <capacity unit='bytes'>17179869184</capacity>
  <allocation unit='bytes'>17179869184</allocation>
  <available unit='bytes'>0</available>
  <source>
    <host name='192.168.5.2'/>
    <device path='iqn.2010-06.com.nutanix:kvm-training-disk9-e3878625'/>
  </source>
  <target>
    <path>/dev/disk/by-path</path>
    <permissions>
      <mode>0755</mode>
      <owner>-1</owner>
      <group>-1</group>
    </permissions>
  </target>
</pool>

List the volume

[root@NTNX-12AM2K480036-A ~]# virsh vol-list --pool kvm-training-disk9 (esxcfg-scsidevs -m)

Name                 Path
-----------------------------------------
unit:0:0:0           /dev/disk/by-path/ip-192.168.5.2:3260-iscsi-iqn.2010-06.com.nutanix:kvm-training-disk9-e3878625-lun-0


[root@NTNX-12AM2K480036-A ~]# virsh vol-info --pool kvm-training-disk9 unit:0:0:0
Name:           unit:0:0:0
Type:           block
Capacity:       16.00 GiB
Allocation:     16.00 GiB


Create the VM with following config
cat ~/KVM/Create/disk9 ( chmod +x)

#!/usr/bin/env bash

virt-install \
  --description "CentOS 6.4 - minimal desktop" \
  --connect qemu:///system \
  --name kvm-training9 \
  --disk vol=kvm-training-disk9/unit:0:0:0,format=raw,cache=none,io=native,bus=virtio \
  --ram 1024 \
  --vcpu 1 \
  --graphics vmc,port=5905,listen=0.0.0.0 \
  --os-type linux \
  --os-variant rhel6 \
  --disk vol=CentOS-6.4.iso/unit:0:0:0,format=raw,io=native,bus=ide,device=cdrom \
  --noautoconsole \
  --wait 0  --network network=VM-Network,model=virtio \
  --force


run ~/KVM/Create/disk9

[root@NTNX-12AM2K480036-A by-path]# virsh list ( similar to vim-cmd vmsvc/getallvms or vm-support -V esxcli vm  process list)
 Id    Name                           State
----------------------------------------------------
 1     NTNX-12AM2K480036-A-CVM        running

52    kvm-training9                  running

virsh # dumpxml 52  ---- like vmx file <domain type='kvm' id='52'>
  <name>kvm-training9</name>
  <uuid>81f4f17f-b9e8-d533-1b89-6295c5ff6048</uuid>
  <description>CentOS 6.4 - minimal desktop</description>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.4.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-path/ip-192.168.5.2:3260-iscsi-iqn.2010-06.com.nutanix:kvm-training-disk9-e3878625-lun-0'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw' io='native'/>
      <source dev='/dev/disk/by-path/ip-192.168.5.2:3260-iscsi-iqn.2010-06.com.nutanix:CentOS-6.4-x86_64-bin-DVD1.iso-c0e9bd87-lun-0'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:3e:4c:f5'/>
      <source network='VM-Network'/>
      <target dev='vnet6'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/6'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/6'>
      <source path='/dev/pts/6'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5909' autoport='no' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>unconfined_u:system_r:svirt_t:s0:c399,c943</label>
    <imagelabel>unconfined_u:object_r:svirt_image_t:s0:c399,c943</imagelabel>
  </seclabel>
</domain>


 [root@NTNX-12AM2K480036-A Create]# ps -ef|grep qemu |grep training9- similar to vmx module in vmware
qemu     14276     1  0 15:44 ?        00:00:42 /usr/libexec/qemu-kvm -name kvm-training9 -S -M rhel6.4.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid 81f4f17f-b9e8-d533-1b89-6295c5ff6048 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/kvm-training9.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/disk/by-path/ip-192.168.5.2:3260-iscsi-iqn.2010-06.com.nutanix:kvm-training-disk9-e3878625-lun-0,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/dev/disk/by-path/ip-192.168.5.2:3260-iscsi-iqn.2010-06.com.nutanix:CentOS-6.4-x86_64-bin-DVD1.iso-c0e9bd87-lun-0,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,aio=native -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=36,id=hostnet0,vhost=on,vhostfd=39 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:3e:4c:f5,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 0.0.0.0:9 -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5


virt-top (esxtop) -1,2,3
 virt-top 17:40:34 - x86_64 24/24CPU 1600MHz 48326MB
9 domains, 8 active, 8 running, 0 sleeping, 0 paused, 1 inactive D:0 O:0 X:0
CPU: 1.0%  Mem: 22008 MB (22008 MB by guests)

   ID S RXBY TXBY RXPK TXPK DOMAIN       INTERFACE
    1 R  23K  25K   96   89 NTNX-12AM2K4 vnet0
   43 R  723    0    9    0 gasmith-trai vnet2
    52 R    0    0    0    0 kvm-training vnet6

We have virt_install in Nutanix CVM to automate these steps ( create iscsi disk, create pool and install VM)


 .CVM:10.3.202.19:~/nutanix_kvm/bin$ ./virt_install --cdrom /ImageStore/win7.iso --disk 128 --nic VM-Network --vnc_port 5999 --os_type windows --os_variant win7 --name kvm-testing-win27


2013-06-27 11:21:43 INFO batch_worker.py:190 Preparing nutanix disks: 0%
2013-06-27 11:21:46 INFO batch_worker.py:190 Preparing nutanix disks: 50%
2013-06-27 11:21:46 INFO batch_worker.py:190 Preparing nutanix disks: 100%
2013-06-27 11:21:46 INFO batch_worker.py:190 Creating libvirt storage pools: 0%
2013-06-27 11:21:50 INFO batch_worker.py:190 Creating libvirt storage pools: 50%
2013-06-27 11:21:52 INFO batch_worker.py:190 Creating libvirt storage pools: 100%
2013-06-27 11:21:52 INFO kvm_domain_template.py:156 Running virt-install

 ( connect to VNC -:99 disable Adapt and max quality in vnc viewer)
Connect to the console and install the CentOS.(virt-manager)

virsh # list --all (vmsvc/getallvms)
 Id    Name                           State
----------------------------------------------------
 1     NTNX-12AM2K480036-C-CVM        running
 40    kvm-training03                 running
 41    kvm-training6                  running
 42    kvm-training4                  running
 47    kvm-testing-win21              running
 48    kvm-testing-win24              running
 -     kvm-testing-win99              shut off



virsh # start kvm-testing-win99  -- vim-cmd vmsvc/power.on
Domain kvm-testing-win99 started