Tuesday, June 18, 2013

Curator does thankless job of keeping Nutanix cluster Clean and Lean.

Curator uses map reduce logic to clean up deleted vdisks, containers and update reference count.
It monitors under-replicated or over-replicated extent groups , redistribute extent groups for node and block awareness. Based on upper and lower threshold, it migrates extent groups between the tiers based on "hotness" of the data.  Partial scan is initiated every 30 minutes if there is "to Remove", "ILM needed" or "Diskspace utilization". Full scan initiated every 6 hours does additional function of updating ref. counts. For ILM and cleaning extent groups, curator finds those extent groups informs stargate to do the actual job and chronos acts as admission control on how many of these jobs are forwarded to stargate.

BTW, curator does these in the background and nutanix cluster has optimum gflags and
settings for the curator. With every release of the nutanix software, more of these configs
will be tuned automatically based on workload.

Gflags for configuring curator: ( if you need to change gflags, please contact nutanix
support or sales team).
1. Lower threshold is configured via
ncli sp edit ilm-thresh ( default 70)
2.upper ilm threshold -  --curator_tier_usage_ilm_threshold_percent 
3. how much to migrate between tiers upto lower threshold -
curator_tier_free_up_percent_by_ilm

4.  how often chronos asks stargate to work on curator jobs: chronos_master_handshake_period_msecs
5. 
--curator_next_tier_usage_ilm_threshold_percent=95 (default 90) -- migrate to next tier only if next tier has so much free space.
6. -curator_full_scan_period_secs  - how often full scan is run
7. --chronos_master_node_max_active_requests - # number of requests sent to stargate at every handshake.


How to manually run a full scan:

for svm in `svmips`; do wget -O - "http://$svm:2010/master/api/client/StartCuratorTasks?task_type=2"; done


Here is an example of hot tier usage and ILM migration activity if these params are set aggressively and this
could cause unnecessary network traffic and I/O activities. As noticed in the following figure, that before Apr17th,
there were a lot of migrate activities from the SSD tier. This picture is plots the usage of SSD tier.


How to check how much of your data was accessed in last 30 minutes in any tier ?
heat-map-analysis

 
How to find when curator full scan and partial scan was run. ?

Curator Jobs

Job idExecution idJob nameStatusReasonsZeus config validStart timeEnd timeTotal time (secs)
165656Partial ScanSucceededILMYesTue Jun 18 10:34:18 2013Tue Jun 18 10:39:51 2013333
165654Partial ScanSucceededILMYesTue Jun 18 10:03:48 2013Tue Jun 18 10:09:08 2013320
165652Partial ScanSucceededILMYesTue Jun 18 09:33:18 2013Tue Jun 18 09:38:51 2013333
165650Partial ScanSucceededILMYesTue Jun 18 09:03:17 2013Tue Jun 18 09:08:50 2013333
165647Partial ScanSucceededILMYesTue Jun 18 08:32:47 2013Tue Jun 18 08:38:07 2013320
065642Full ScanSucceededILM ToRemoveYesTue Jun 18 08:02:17 2013Tue Jun 18 08:17:50 2013933
165640Partial ScanSucceededPeriodicYesTue Jun 18 07:50:28 2013Tue Jun 18 07:55:49 2013321

Tier Usage:

Storage Pool: NTNX-SP1 ILM Down Migrate threshold: 85

Tier NameTier UsageTier SizeTier Usage Pct
SSD-PCIe1355.50 GB1481.57 GB91%
SSD-SATAN/AN/AN/A
DAS-SATA15362.15 GB51371.99 GB29%

Are all Nodes balanced disk usage ?

Storage Pool: NTNX-SP1 Tier: SSD-PCIe

Mean Usage Pct92%
Zone of Balance85% - 99%
Usage Spread Pct8%
StatusBalanced
Rack IdService VMDisk IdDisk UsageDisk SizeDisk Usage PctInside Zone of Balance
453898548849049320988.31 GB93.09 GB94%Yes
453898548223588.48 GB93.09 GB95%Yes
453898548234888.02 GB93.09 GB94%Yes
453898548285987.77 GB93.09 GB94%Yes
4538985526739568233697919588.48 GB93.09 GB95%Yes
4538985526739568433697917488.61 GB93.09 GB95%Yes
4538985526739568633697921188.58 GB93.09 GB95%Yes
4538985526739568833697918488.41 GB93.09 GB94%Yes
490725470490725463490725475163.83 GB184.21 GB88%Yes
490725470490725465490725476160.78 GB184.21 GB87%Yes
490725470490725467490725477163.83 GB184.21 GB88%Yes
490725470490725471490725478160.41 GB184.21 GB87%Yes

What are the activities done during last patial scan ?
MapReduce job 65657
Job id65657
Job namePartialScan MapReduce
StatusSucceeded
Map tasks done36/36
Reduce tasks done24/24
Start timeTue Jun 18 10:35:18 2013
End timeTue Jun 18 10:39:48 2013
Total time (secs)270

Map Tasks

Task idTask TypeDesired StatusStatusNode idStart timeEnd timeTotal time (secs)
0ExtentGroupIdMapTaskSucceededSucceeded472452227Tue Jun 18 10:35:18 2013Tue Jun 18 10:37:31 2013133
1ExtentGroupIdMapTaskSucceededSucceeded490493246Tue Jun 18 10:35:18 2013Tue Jun 18 10:37:37 2013139
2ExtentGroupIdMapTaskSucceededSucceeded490725549Tue Jun 18 10:35:18 2013Tue Jun 18 10:36:59 2013101
3ExtentGroupIdMapTaskSucceededSucceeded490725550Tue Jun 18 10:35:18 2013Tue Jun 18 10:36:51 201393
4ExtentGroupIdMapTaskSucceededSucceeded472452000Tue Jun 18 10:35:18 2013Tue Jun 18 10:36:17 201359
5ExtentGroupIdMapTaskSucceededSucceeded472581426Tue Jun 18 10:35:18 2013Tue Jun 18 10:37:06 2013108
6ExtentGroupIdMapTaskSucceededSucceeded472451186Tue Jun 18 10:35:18 2013Tue Jun 18 10:37:48 2013150
7ExtentGroupIdMapTaskSucceededSucceeded490725552Tue Jun 18 10:35:18 2013Tue Jun 18 10:36:14 201356
8ExtentGroupIdMapTaskSucceededSucceeded490725511Tue Jun 18 10:35:18 2013Tue Jun 18 10:36:34 201376
9ExtentGroupIdMapTaskSucceededSucceeded472451018Tue Jun 18 10:35:18 2013Tue Jun 18 10:36:53 201395
10ExtentGroupIdMapTaskSucceededSucceeded472451324Tue Jun 18 10:35:18 2013Tue Jun 18 10:36:57 201399
11ExtentGroupIdMapTaskSucceededSucceeded472337394Tue Jun 18 10:35:18 2013Tue Jun 18 10:36:34 201376
12ExtentGroupAccessDataMapTaskSucceededSucceeded472452227Tue Jun 18 10:35:18 2013Tue Jun 18 10:35:31 201313
13ExtentGroupAccessDataMapTaskSucceededSucceeded490725552Tue Jun 18 10:35:18 2013Tue Jun 18 10:36:14 201356
14ExtentGroupAccessDataMapTaskSucceededSucceeded490725549Tue Jun 18 10:35:18 2013Tue Jun 18 10:35:59 201341
15ExtentGroupAccessDataMapTaskSucceededSucceeded490725550Tue Jun 18 10:35:18 2013Tue Jun 18 10:35:51 201333
16ExtentGroupAccessDataMapTaskSucceededSucceeded472452000Tue Jun 18 10:35:18 2013Tue Jun 18 10:36:17 201359
17ExtentGroupAccessDataMapTaskSucceededSucceeded472581426Tue Jun 18 10:35:18 2013Tue Jun 18 10:36:06 201348
18ExtentGroupAccessDataMapTaskSucceededSucceeded472451186Tue Jun 18 10:35:18 2013Tue Jun 18 10:35:48 201330
19ExtentGroupAccessDataMapTaskSucceededSucceeded472452227Tue Jun 18 10:35:32 2013Tue Jun 18 10:36:31 201359
20ExtentGroupAccessDataMapTaskSucceededSucceeded490725511Tue Jun 18 10:35:18 2013Tue Jun 18 10:35:34 201316
21ExtentGroupAccessDataMapTaskSucceededSucceeded472451018Tue Jun 18 10:35:18 2013Tue Jun 18 10:35:53 201335
22ExtentGroupAccessDataMapTaskSucceededSucceeded490493246Tue Jun 18 10:35:18 2013Tue Jun 18 10:35:37 201319
23ExtentGroupAccessDataMapTaskSucceededSucceeded472337394Tue Jun 18 10:35:18 2013Tue Jun 18 10:35:34 201316
24VDiskOplogMapTaskSucceededSucceeded472451324Tue Jun 18 10:35:18 2013Tue Jun 18 10:35:57 201339
25VDiskOplogMapTaskSucceededSucceeded472337394Tue Jun 18 10:35:34 2013Tue Jun 18 10:36:34 201360
26VDiskOplogMapTaskSucceededSucceeded490725511Tue Jun 18 10:35:34 2013Tue Jun 18 10:36:34 201360
27VDiskOplogMapTaskSucceededSucceeded490493246Tue Jun 18 10:35:37 2013Tue Jun 18 10:36:37 201360
28VDiskOplogMapTaskSucceededSucceeded472451186Tue Jun 18 10:35:48 2013Tue Jun 18 10:36:48 201360
29VDiskOplogMapTaskSucceededSucceeded490725550Tue Jun 18 10:35:51 2013Tue Jun 18 10:36:51 201360
30VDiskOplogMapTaskSucceededSucceeded472451018Tue Jun 18 10:35:53 2013Tue Jun 18 10:36:53 201360
31VDiskOplogMapTaskSucceededSucceeded472451324Tue Jun 18 10:35:57 2013Tue Jun 18 10:36:57 201360
32VDiskOplogMapTaskSucceededSucceeded490725549Tue Jun 18 10:35:59 2013Tue Jun 18 10:36:59 201360
33VDiskOplogMapTaskSucceededSucceeded472581426Tue Jun 18 10:36:06 2013Tue Jun 18 10:37:06 201360
34VDiskOplogMapTaskSucceededSucceeded490725552Tue Jun 18 10:36:14 2013Tue Jun 18 10:37:14 201360
35VDiskOplogMapTaskSucceededSucceeded490725552Tue Jun 18 10:36:14 2013Tue Jun 18 10:37:14 201360

Reduce Tasks

Task idTask TypeDesired StatusStatusNode idStart timeEnd timeTotal time (secs)
0DiskIdReduceTaskSucceededSucceeded472452227Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:31 2013253
1DiskIdReduceTaskSucceededSucceeded472452227Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:31 2013253
2DiskIdReduceTaskSucceededSucceeded472452000Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:17 2013239
3DiskIdReduceTaskSucceededSucceeded472452000Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:17 2013239
4DiskIdReduceTaskSucceededSucceeded490493246Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:37 2013259
5DiskIdReduceTaskSucceededSucceeded490493246Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:37 2013259
6DiskIdReduceTaskSucceededSucceeded490725549Tue Jun 18 10:35:18 2013Tue Jun 18 10:38:59 2013221
7DiskIdReduceTaskSucceededSucceeded490725549Tue Jun 18 10:35:18 2013Tue Jun 18 10:38:59 2013221
8DiskIdReduceTaskSucceededSucceeded472337394Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:34 2013256
9DiskIdReduceTaskSucceededSucceeded472337394Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:34 2013256
10DiskIdReduceTaskSucceededSucceeded490725550Tue Jun 18 10:35:18 2013Tue Jun 18 10:38:51 2013213
11DiskIdReduceTaskSucceededSucceeded490725550Tue Jun 18 10:35:18 2013Tue Jun 18 10:38:51 2013213
12ExtentGroupIdReduceTaskSucceededSucceeded472581426Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:06 2013228
13ExtentGroupIdReduceTaskSucceededSucceeded472581426Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:06 2013228
14ExtentGroupIdReduceTaskSucceededSucceeded490725552Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:14 2013236
15ExtentGroupIdReduceTaskSucceededSucceeded490725552Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:14 2013236
16ExtentGroupIdReduceTaskSucceededSucceeded472451018Tue Jun 18 10:35:18 2013Tue Jun 18 10:38:53 2013215
17ExtentGroupIdReduceTaskSucceededSucceeded472451018Tue Jun 18 10:35:18 2013Tue Jun 18 10:38:53 2013215
18ExtentGroupIdReduceTaskSucceededSucceeded490725511Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:34 2013256
19ExtentGroupIdReduceTaskSucceededSucceeded490725511Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:34 2013256
20ExtentGroupIdReduceTaskSucceededSucceeded472451324Tue Jun 18 10:35:18 2013Tue Jun 18 10:38:57 2013219
21ExtentGroupIdReduceTaskSucceededSucceeded472451324Tue Jun 18 10:35:18 2013Tue Jun 18 10:38:57 2013219
22ExtentGroupIdReduceTaskSucceededSucceeded472451186Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:48 2013270
23ExtentGroupIdReduceTaskSucceededSucceeded472451186Tue Jun 18 10:35:18 2013Tue Jun 18 10:39:48 2013270

Job Counters

NameValue
MapExtentGroupIdMap535252
ReduceDiskIdExtentGroupId1070510
MapExtentGroupAccessDataMap535213
NumExtentGroupsToMigrateForILM4740
NumExtentGroupsToMigrateForDiskBalancing0
MapVDiskOplogMap764
NumHostVDiskTasks3
FgHostVDiskTaskCount3
FgDeleteToRemoveOplogMapEntryTaskCount0
FgDeleteVDiskBlocksTaskCount0
MapVDiskBlockMap0
NumExtentGroupsWithReplicaOnSameNode0
NumExtentGroupsWithReplicaOnSameRack3275
NumFixExtentGroupsTasksReplicaOnSameRack23
FgDeleteExtentGroupsWithNonEidExtentsTaskCount0
NumExtentGroupsWithNonEidExtentsToDelete0
NumInvalidExtentGroupAccessDataMapEntries0
BgFixExtentGroupTaskCount5779
BgMergeExtentGroupsTaskCount0
BgCompressExtentsTaskCount0
BgDeduplicateExtentTaskCount0
BgMigrateExtentsTaskCount0
BgCopyBlockmapMetadataTaskCount0
BgUpdateRefcountsTaskCount0
InternalError0

What are the activities done during full scan ?

MapReduce job 65643
Job id65643
Job nameFullScan MapReduce #1
StatusSucceeded
Map tasks done25/25
Reduce tasks done24/24
Start timeTue Jun 18 08:03:32 2013
End timeTue Jun 18 08:06:33 2013
Total time (secs)181

Map Tasks

Task idTask TypeDesired StatusStatusNode idStart timeEnd timeTotal time (secs)
0VDiskOplogMapTaskSucceededSucceeded490725550Tue Jun 18 08:03:33 2013Tue Jun 18 08:03:50 201317
1VDiskOplogMapTaskSucceededSucceeded490725552Tue Jun 18 08:03:33 2013Tue Jun 18 08:04:13 201340
2VDiskOplogMapTaskSucceededSucceeded490725552Tue Jun 18 08:03:33 2013Tue Jun 18 08:04:13 201340
3VDiskOplogMapTaskSucceededSucceeded472451018Tue Jun 18 08:03:33 2013Tue Jun 18 08:03:52 201319
4VDiskOplogMapTaskSucceededSucceeded472451018Tue Jun 18 08:03:33 2013Tue Jun 18 08:03:52 201319
5VDiskOplogMapTaskSucceededSucceeded490725511Tue Jun 18 08:03:33 2013Tue Jun 18 08:04:33 201360
6VDiskOplogMapTaskSucceededSucceeded490725511Tue Jun 18 08:03:33 2013Tue Jun 18 08:04:33 201360
7VDiskOplogMapTaskSucceededSucceeded472451324Tue Jun 18 08:03:33 2013Tue Jun 18 08:03:56 201323
8VDiskOplogMapTaskSucceededSucceeded472451324Tue Jun 18 08:03:33 2013Tue Jun 18 08:03:56 201323
9VDiskOplogMapTaskSucceededSucceeded472451186Tue Jun 18 08:03:33 2013Tue Jun 18 08:03:47 201314
10VDiskOplogMapTaskSucceededSucceeded472451186Tue Jun 18 08:03:33 2013Tue Jun 18 08:03:47 201314
11VDiskOplogMapTaskSucceededSucceeded490493246Tue Jun 18 08:03:36 2013Tue Jun 18 08:04:36 201360
12NfsInodeMapTaskSucceededSucceeded472452227Tue Jun 18 08:03:33 2013Tue Jun 18 08:04:30 201357
13NfsInodeMapTaskSucceededSucceeded490493246Tue Jun 18 08:03:33 2013Tue Jun 18 08:03:36 20133
14NfsInodeMapTaskSucceededSucceeded472452227Tue Jun 18 08:03:33 2013Tue Jun 18 08:04:30 201357
15NfsInodeMapTaskSucceededSucceeded490725549Tue Jun 18 08:03:33 2013Tue Jun 18 08:03:58 201325
16NfsInodeMapTaskSucceededSucceeded472452000Tue Jun 18 08:03:33 2013Tue Jun 18 08:04:16 201343
17NfsInodeMapTaskSucceededSucceeded490725549Tue Jun 18 08:03:33 2013Tue Jun 18 08:03:58 201325
18NfsInodeMapTaskSucceededSucceeded472337394Tue Jun 18 08:03:33 2013Tue Jun 18 08:04:33 201360
19NfsInodeMapTaskSucceededSucceeded472581426Tue Jun 18 08:03:33 2013Tue Jun 18 08:04:04 201331
20NfsInodeMapTaskSucceededSucceeded472452000Tue Jun 18 08:03:33 2013Tue Jun 18 08:04:16 201343
21NfsInodeMapTaskSucceededSucceeded490725550Tue Jun 18 08:03:33 2013Tue Jun 18 08:03:50 201317
22NfsInodeMapTaskSucceededSucceeded490493246Tue Jun 18 08:03:33 2013Tue Jun 18 08:03:36 20133
23NfsInodeMapTaskSucceededSucceeded472337394Tue Jun 18 08:03:33 2013Tue Jun 18 08:04:33 201360
24NfsVDiskMapTaskSucceededSucceeded472581426Tue Jun 18 08:03:33 2013Tue Jun 18 08:04:04 201331

Reduce Tasks

Task idTask TypeDesired StatusStatusNode idStart timeEnd timeTotal time (secs)
0NfsInodeReduceTaskSucceededSucceeded472452227Tue Jun 18 08:03:33 2013Tue Jun 18 08:06:30 2013177
1NfsInodeReduceTaskSucceededSucceeded472452227Tue Jun 18 08:03:33 2013Tue Jun 18 08:06:30 2013177
2NfsInodeReduceTaskSucceededSucceeded472452000Tue Jun 18 08:03:33 2013Tue Jun 18 08:06:16 2013163
3NfsInodeReduceTaskSucceededSucceeded472452000Tue Jun 18 08:03:33 2013Tue Jun 18 08:06:16 2013163
4NfsInodeReduceTaskSucceededSucceeded490493246Tue Jun 18 08:03:33 2013Tue Jun 18 08:05:36 2013123
5NfsInodeReduceTaskSucceededSucceeded490493246Tue Jun 18 08:03:33 2013Tue Jun 18 08:05:36 2013123
6NfsInodeReduceTaskSucceededSucceeded490725549Tue Jun 18 08:03:33 2013Tue Jun 18 08:05:58 2013145
7NfsInodeReduceTaskSucceededSucceeded490725549Tue Jun 18 08:03:33 2013Tue Jun 18 08:05:58 2013145
8NfsInodeReduceTaskSucceededSucceeded472337394Tue Jun 18 08:03:33 2013Tue Jun 18 08:06:33 2013180
9NfsInodeReduceTaskSucceededSucceeded472337394Tue Jun 18 08:03:33 2013Tue Jun 18 08:06:33 2013180
10NfsInodeReduceTaskSucceededSucceeded490725550Tue Jun 18 08:03:33 2013Tue Jun 18 08:05:50 2013137
11NfsInodeReduceTaskSucceededSucceeded490725550Tue Jun 18 08:03:33 2013Tue Jun 18 08:05:50 2013137
12NfsDirectoryReduceTaskSucceededSucceeded472581426Tue Jun 18 08:03:33 2013Tue Jun 18 08:06:04 2013151
13NfsDirectoryReduceTaskSucceededSucceeded472581426Tue Jun 18 08:03:33 2013Tue Jun 18 08:06:04 2013151
14NfsDirectoryReduceTaskSucceededSucceeded490725552Tue Jun 18 08:03:33 2013Tue Jun 18 08:06:13 2013160
15NfsDirectoryReduceTaskSucceededSucceeded490725552Tue Jun 18 08:03:33 2013Tue Jun 18 08:06:13 2013160
16NfsDirectoryReduceTaskSucceededSucceeded472451018Tue Jun 18 08:03:33 2013Tue Jun 18 08:05:52 2013139
17NfsDirectoryReduceTaskSucceededSucceeded472451018Tue Jun 18 08:03:33 2013Tue Jun 18 08:05:52 2013139
18NfsDirectoryReduceTaskSucceededSucceeded490725511Tue Jun 18 08:03:33 2013Tue Jun 18 08:06:33 2013180
19NfsDirectoryReduceTaskSucceededSucceeded490725511Tue Jun 18 08:03:33 2013Tue Jun 18 08:06:33 2013180
20NfsDirectoryReduceTaskSucceededSucceeded472451324Tue Jun 18 08:03:33 2013Tue Jun 18 08:05:56 2013143
21NfsDirectoryReduceTaskSucceededSucceeded472451324Tue Jun 18 08:03:33 2013Tue Jun 18 08:05:56 2013143
22NfsDirectoryReduceTaskSucceededSucceeded472451186Tue Jun 18 08:03:33 2013Tue Jun 18 08:05:47 2013134
23NfsDirectoryReduceTaskSucceededSucceeded472451186Tue Jun 18 08:03:33 2013Tue Jun 18 08:05:47 2013134

Job Counters

NameValue
MapVDiskOplogMap764
FgHostVDiskTaskCount1
FgDeleteToRemoveOplogMapEntryTaskCount0
NumHostVDiskTasks1
FgAddNfsInodeContainerIdTaskCount0
NumNfsInodesUpdatedWithContainerId0
FgDeleteNfsInodesTaskCount0
NumNfsInodesDeleted0
NumNfsVDisksProcessed879
NfsReduceChildLinkCount5853
NfsReduceParentLinkCount5859
NfsReduceAttributeCount5859
NfsReduceVDiskCount879
FgFixNfsInodeLinksTaskCount0
FgFixNfsLinkAcrossContainersTaskCount0
FgFixNfsVDiskTaskCount0
FgDeleteNfsDirectoryCount0
BgFixExtentGroupTaskCount0
BgMergeExtentGroupsTaskCount0
BgCompressExtentsTaskCount0
BgDeduplicateExtentTaskCount0
BgMigrateExtentsTaskCount0
BgCopyBlockmapMetadataTaskCount0
BgUpdateRefcountsTaskCount0
InternalError0
NameIdValue
NumNfsDirectoryInodes19930
NumNfsDirectoryInodes19940
NumNfsDirectoryInodes2876983
NumNfsDirectoryInodes3369989193
NumNfsDirectoryInodes19950
NumNfsDirectoryInodes2877211
NumNfsDirectoryInodes19960
NumNfsDirectoryInodes193305844
NumNfsDirectoryInodes1933059261
NumNfsDirectoryInodes4133650
NumNfsDirectoryInodes3234873