4.0 Feature: Cluster Health Framework

Nutanix cluster health is designed to detect and analyze Nutanix cluster related failures to bring more customer visibility to the issues by providing cause/impact and resolution. Nutanix cluster health utilizes plugins from NCC ( nutanix cluster check) utility.

NCC is developed by Nutanix Engineering from inputs provided by support engineers, customers, on-call engineers and solution architects. Nutanix Engineering productised their troubleshooting scripts into NCC. Nutanix Cluster health runs NCC plugins at various intervals and provides easy access to the results through UI.  Nutanix customer will be able troubleshoot or identify the issues with the cluster and will result in a faster resolution. This will provide uniform troubleshooting tools across different Hypervisors.

Nutanix cluster health can be accessed from Prism Element (URL: CVM's ip address).

Prism Element First Page - Cluster Health Access

Nutanix Prism Element has a cluster health walk-through which shows an example disk health troubleshooting.

 

Here is the list of cluster health checks 

List of Health Check

CVM: - 

1. CPU Utilization/Load Average
2. Disk - Metadata Usage (Inode)/HDD disk Usage (df) /HDD latency(sar/iostat) smartctl status/SSD latency.
3. Memory committed ( /proc/meminfo)
4. Network - CVM to CVM connectivity (external Vswitch), CVM to host (nutanixVswitch), Gateway config, subnet config ( verify nutanix HA network config)
5. Time Drift - between CVM/Host

Host/Node:

1. CPU utilization
2. Memory swap rate
3. Network - 10 Gbe connectivity(vswitch and vmknic) / Nic Error Rate/Receive and Packet loss (ethtool)

VM : ( Nutanix provides VM centric stats - http://nutanix.blogspot.com/2013/08/how-much-of-detail-can-you-get-about-vm.html)

1. CPU utilization
2. I/O latency (vdisk)
3. Memory ( swap rate/usage)
4. Network Rx/Tx packet loss. 

CLI options:

nutanix@NTNX-13SM35300008-B-CVM:10.1.60.110:~$ ncli health-check ls |grep Name
    Name                      : I/O Latency
    Name                      : CPU Utilization
    Name                      : Disk Metadata Usage
    Name                      : Transmit Packet Loss
    Name                      : CVM to CVM Connectivity
    Name                      : Receive Packet Loss
    Name                      : CPU Utilization
    Name                      : CPU Utilization
    Name                      : Transmit Packet Loss
    Name                      : Memory Usage
    Name                      : Receive Packet Loss
    Name                      : HDD I/O Latency
    Name                      : Gateway Configuration
    Name                      : HDD S.M.A.R.T Health Status
    Name                      : Load Level
    Name                      : Memory Usage
    Name                      : CVM to Host Connectivity
    Name                      : 10 GbE Compliance
    Name                      : Time Drift
    Name                      : HDD Disk Usage
    Name                      : SSD I/O Latency
    Name                      : Memory Pressure
    Name                      : Memory Swap Rate
    Name                      : Subnet Configuration
    Name                      : Memory Swap Rate
    Name                      : Node Nic Error Rate High

Configuration of Health Checks:

1. Turn Check off
2. Parameters for Critical/Warn Threshold (if applicable)
3. Change the schedule.
4. Edit option available from CLI as well . ncli health-check edit  (interval/enable/parameter-thresholds) 

Cause/Impact and Resolution:

Each health check provides

1. Cause of the failure :  Example - disk running out of space
2. Resolution: How to fix the issue -Example: add storage capacity or delete the data.
3. Impact:  What will be the user/cluster impact ?

 History of the check and list of entities checked ( different disks/ list of CVMs)

Running the plugin manually:

NCC is superset of health check and has more plugins that can be run. NCC can be updated independently of NOS versions. ( note that the certain newer NCC plugins may be applicable only to certain NOS versions). Here are sample of few NCC options.

ncc health check

 List of Network Checks that can be run ( intrusive checks will affect the performance of the cluster.)

ncc network_checks

Sample of running a network_check .

Future Developments:

1. Unification of Cluster health, Alerts and Events. 
2. Impact, Cause and Resolution updated via NCC updates independent of NOS upgrade.
3. Provide finer Root Cause analysis and more insights into cluster health.

Here are some of the feedback that we got about  our first version of "cluster health". We will continue to improve the user experience as well as add more checks.