Monday, July 29, 2013

Nutanix NFS VAAI troubleshooting

vStorage APIs for Array Integration is a feature introduced in ESXi that provides 
hardware acceleration functionality. It enables your host to offload specific virtual 
machine and storage management operations to compliant storage hardware. With
 the storage hardware assistance, your host performs these operations faster and 
consumes less CPU, memory, and storage fabric bandwidth.

NAS DISK-LIB:
1. Full /Fast Copy, which is used to copy or migrate data within the same physical array 
2.  RESERVE Space ( Nutanix 3.5)
3. extended statistics.

To verify if nutanix vaai is installed:

~ # esxcli software vib list |grep vaai
nfs-vaai-plugin                1.1-101                             Nutanix   CommunitySupported  2013-05-12

~ # esxcli storage  nfs list
Volume Name   Host              Share   Accessible Mounted  Read-Only  HW Acceleration
-----------------  -----------  ------------------------  -------  ---------  ---------------------
NFS-StorageA   192.168.5.2  /NFS-A   true       true           false             Supported


NFS Uses Full File Clone Primitive and Block storage uses XCOPY primitive. 
Storage vMotion, Hot Cloning cannot leverage  Full File Clone Primitive. 
Full file clone primitive can be leveraged only when a cold migration is initiated.

Reserve Space primitive is implemented  Nutanix 3.5 software version.
More details in
http://cormachogan.com/2012/11/08/vaai-comparison-block-versus-nas/
VMWARE kb-1021976
 Troubleshooting:
VAAI clone will fail:
1. if VM has snapshot
2. if vm is powered on
3. if container name has space or other characters
4. cloning between datastores.

esxtop:
First use ‘u’ to select device view, then enable fields o and p for VAAI stats.

VAAI Logs:

grep -i VAAI var/log/vpxa.log and vmkernel.log

Possible Errors in /var/log/vpxa.log/vmkernel.log
Error:
2012-06-21T01:30:09.165Z [25DAEB90 info 'DiskLib' opID=B2F4E13F-00001118-66]
DISKLIB-LIB   : Failed to create native clone on destination handle :
One of the parameters supplied is invalid (1).


Cause:
  • Cloning to a different datastore
  • Creating a clone from a VM that has a snapshot
Error:


2012-07-18T23:15:49.378Z [4AB51B90 info 'DiskLib' opID=55E1C10D-00000188-3a]
DISKLIB-LIB   : Failed to create native clone on destination handle :
The system cannot find the file specified (25).

Cause:
  • A container name that contains a space
  • A container name that was changed
Error:


2012-07-19T01:59:02.865Z [362ACB90 info 'DiskLib' opID=4A5CFD5E-00006113-5f]
DISKLIB-LIB   : Failed to create native clone on destination handle :
The specified feature is not supported by this version (24).


Cause:
  • Cloning from the local datastore (VMFS)
Also you can vmware.log

2013-07-18T04:09:50.903Z| vcpu-0| SNAPSHOT: SnapshotDumperOpenFromInfo: Creating checkpoint file /vmfs/volumes/78993998-9e053604/ENCmsOLY1007-SCCM/ENCmsOLY1007-Snapshot168.vmsn
2013-07-18T04:09:51.494Z| vcpu-0| nutanix_nfs_plugin: Established VAAI session with NFS server 192.168.5.2
2013-07-18T04:09:51.526Z| vcpu-0| DISKLIB-VMFS  : "/vmfs/volumes/78993998-9e053604/ENCmsOLY1007-SCCM/ENCmsOLY1007-000002-delta.vmdk" : open successful (29) size = 1618106368, hd = 0. Type 8
2013-07-18T04:09:52.255Z| vcpu-0| DISKLIB-LIB   : DiskLibCreateCreateParam: vmfssparse grain size set to : 1



Thursday, July 25, 2013

CBRC storage Accelerator stats


/~ # vsish
/> cat /vmkModules/cbrc_filter/dcacheStats
CBRC cache statistics {
   Cache chunk size:4096
   Number of LRU lists:100
   Number of hash buckets:5000
   Total buffers:262144
   Buffers with memory allocated:262144
   Minimum requirement of alloced buffers:0
   VMs using cache:9
   VMs using active cache:9
   Stats counters:Data cache counters {
      Active buffers:0
      Buffer invalidations:0
      Evicts of valid buffers:0
      Evicts of valid buffers during getfreebuf:0
      VM issued read io count:1003241
      VM issued write io count:291538
      Read io count within cache limits:1001264
      Write io count within cache limits:291538
      Backend io read count:270005
      Backend io write count:291445
      Abort count:0
      Reset count:170
      Digest not found count:213818
      Skipped read count:1977
      Backened io read failures:0
      Getfreebuf failures:0
      On demand buf alloc failures:0
      Deleted during fetch count:0
      Transit during fetch count:168814
      Transit Pop count:168814
      Transit waits:0
      Timeouts in buffer transit waits:0
      Transit errors:0
      Cache fill io errors:0
      Copy to user buf errors:0
      Races in getting free buffers:56868
      Reclaimed buffers:0
   }
}

Friday, July 19, 2013

Creating service center via python REqueSTs

How to find the json config :
Python script:

import requests

def main():
  base_url = "https://10.3.101.59:9440/PrismGateway/services/rest/v1/"
  s = requests.Session()
  s.auth = ('admin', 'admin')
  s.headers.update({'Content-Type': 'application/json; charset=utf-8'})

 data1 = {
"port": 9443,
  "name": "testservice",
  "userName": "support",
  "ipAddress": "10.10.10.10"
}
s.post(base_url + 'service_centers' ,data=json.dumps(data1))
print s.get(base_url + 'service_centers/testservice', verify=False).json()

if __name__ == "__main__":
  main()

Friday, July 12, 2013

Nutanix REST-API using python Requests - Get container info

From:
http://docs.python-requests.org/en/latest/

Installing Pip and Requests on Nutanix Controller VM/or any Linux system:
curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py
 sudo python get-pip.py
sudo pip install requests
sudo pip install requests --upgrade
 cd /usr/lib/python2.6/site-packages/;sudo chmod -R 755 requests-1.2.3-py2.6.egg/
PYTHONPATH=${PYTHONPATH}:/usr/lib/python2.6/site-packages/requests-1.2.3-py2.6.egg;export PYTHONPATH


Sample Script:
 cat test_resp.py - to print container info

#!/usr/bin/python
import json as json
import requests

def main():
  base_url = "https://10.3.101.59:9440/PrismGateway/services/rest/v1/"
  s = requests.Session()
  s.auth = ('admin', 'admin')
  s.headers.update({'Content-Type': 'application/json; charset=utf-8'})

  print s.get(base_url + 'vstores', verify=False).json()

if __name__ == "__main__":
  main()


Output

./test_resp.py
[{u'protectionDomain': None, u'name': u'ctr3', u'backedup': False, u'markedForRemoval': False, u'id': 2482, u'containerId': 2482}, {u'protectionDomain': None, u'name': u'ctr4', u'backedup': False, u'markedForRemoval': False, u'id': 2483, u'containerId': 2483}, {u'protectionDomain': u'ctr5_1372277619664', u'name': u'ctr5', u'backedup': True, u'markedForRemoval': False, u'id': 1898767, u'containerId': 1898767}, {u'protectionDomain': None, u'name': u'test', u'backedup': False, u'markedForRemoval': False, u'id': 2430343, u'containerId': 2430343}, {u'protectionDomain': None, u'name': u'testing', u'backedup': False, u'markedForRemoval': False, u'id': 2430528, u'containerId': 2430528}, {u'protectionDomain': None, u'name': u'testStats', u'backedup': False, u'markedForRemoval': False, u'id': 18973731, u'containerId': 18973731}, {u'protectionDomain': None, u'name': u'dummyCTR_to_delete1', u'backedup': False, u'markedForRemoval': False, u'id': 19113654, u'containerId': 19113654}]


Get the name and size of storage pool:

#!/usr/bin/python
import json as json
import requests

def main():
  base_url = "https://10.3.101.59:9440/PrismGateway/services/rest/v1/"
  s = requests.Session()
  s.auth = ('admin', 'admin')
  s.headers.update({'Content-Type': 'application/json; charset=utf-8'})

  data = s.get(base_url + 'storage_pools', verify=False).json()
  spname= data ["entities"][0]["name"]
  size= data ["entities"][0]["capacity"]
  print spname,size/(1024*1024*1024*1024)
if __name__ == "__main__":
  main()







Thursday, July 11, 2013

Nutanix REST API Browser - How to get Container Info.

Basic Definition:
REST is an alternative to SOAP based web services. Where SOAP tries to model the exchange between client and server as calls to objects, REST tries to be faithful to the web domain. So when calling a web service written in SOAP, you may write
productService.GetProduct("1")
in REST, you may call a url with HTTP GET
http://someurl/products/product/1
CRUD - Create, Request, Update and Delete are done via http requests POST GET, PUT and DELETE respectively.
One can use curl, links, http browser to get the details of the nutanix cluster.

Here is to most inteRESTing REST API!
Now the most interesting Nutanix has became more inteRESTing!
Or we can call Nutanix REST as BeautyREST! (someone already called it sexy! http://wahlnetwork.com/2013/07/01/nutanix-and-veeam-spread-rest-api-joy/)

Nutanix REST API Browser: http://any_Nutanix_CVM 
Login as admin/admin-- Prism Nutanix Browser (HTML5) requires another couple of blogs to go through its goodness. 

To access Rest API browser:


 

Rest API Explorer: https://10.3.101.61:9440/console/api/


To get NFS datastores /Nutanix containers (vstore)
 Get Response:
 

jerome@ithaca:~$ curl https://10.3.101.61:9440/PrismGateway/services/rest/v1/vstores  --insecure --user admin:admin

[{"id":2482,"name":"ctr3","containerId":2482,"backedup":false,"protectionDomain":null,"markedForRemoval":false},{"id":2483,"name":"ctr4","containerId":2483,"backedup":false,"protectionDomain":null,"markedForRemoval":false},{"id":1898767,"name":"ctr5","containerId":1898767,"backedup":true,"protectionDomain":"ctr5_1372277619664","markedForRemoval":false},{"id":2430343,"name":"test","containerId":2430343,"backedup":false,"protectionDomain":null,"markedForRemoval":false},{"id":2430528,"name":"testing","containerId":2430528,"backedup":false,"protectionDomain":null,"markedForRemoval":false},{"id":18973731,"name":"testStats","containerId":18973731,"backedup":false,"protectionDomain":null,"markedForRemoval":false},{"id":19113654,"name":"dummyCTR_to_delete1","containerId":19113654,"backedup":false,"protectionDomain":null,"markedForRemoval":false}]


to get specific container:

 curl https://10.3.101.61:9440/PrismGateway/services/rest/v1/vstores/1898767  --insecure --user admin:admin


{"id":1898767,"name":"ctr5","containerId":1898767,"backedup":true,"protectionDomain":"ctr5_1372277619664","markedForRemoval":false}