Enabling, Monitoring and Disabling Dedup

dedup info:
http://nutanix.blogspot.com/2013/08/introducing-nede-nutanix-elastic-dedup.html

Enabling Dedup

1.
- it can be done per container
ncli ctr edit name=xyz fingerprint-on-write=on
- it can be done per vdisk
ncli vdisk edit name NFS:2389 fingerprint-on-write=on


2. When you upgrade to 3.5.1, enable dedup right away if needed, as FP is done on write, we may not
get the benefit of dedup if enable later ( unless you vdisk manipulator tool is used).

Why it works on upgrade is curator converts 16 MB extent group to 4 MB extentgroup, so there
will be a lot write activities which will be used for Finger printing.

3. Increase  medusa_extent_group_id_map_cache_size_mb=2048 (stargate gflag) to reduce evictions. Make sure of 24G CVM

4. Content cache and extent cache are allocated based on CVM memory (ENG-10798)


 Monitoring Dedup example: - 3.5.2 prism GUI has dedup stats.

have to look around in 2009 page and Curator master logs.)

#0. Overall container usage & amount of data which is fingerprinted:

On master curator (data/logs/curator.INFO)

I1203 12:27:53.777602  curator_execute_job_op.cc:2452] ContainerEgroupFileSizeBytes[912] = 4168785526784

I1203 12:27:53.777606  curator_execute_job_op.cc:2452] ContainerEgroupFileSizeBytes[107907] = 1100420612096

I1203 12:27:53.777609  curator_execute_job_op.cc:2452] ContainerUntransformedSizeBytes[912] = 4136075722752

I1203 12:27:53.777613  curator_execute_job_op.cc:2452] ContainerUntransformedSizeBytes[107907] = 1100301139968

I1203 12:27:53.777616  curator_execute_job_op.cc:2452] ContainerInternalGarbageBytes[912] = 32709804032

I1203 12:27:53.777621  curator_execute_job_op.cc:2452] ContainerInternalGarbageBytes[107907] = 119472128

I1203 12:27:53.777623  curator_execute_job_op.cc:2452] ContainerFingerprintedBytes[912] =  3699631161344

From above, looks like customer has two containers (912 & 107907) and 912 is the one for which fingerprint on write has been turned on. Almost 90% of it is fingerprinted (3.6T out of 4.1T).

#1. Read Path Live (30sec) Metrics

wget -O- 'http://localhost:2009/h/vars?regex=stargate.content&format=text'

stargate/content_cache_adds 2921

stargate/content_cache_dedup_ref_count 2.4583     <--- effective RAM/Flash savings right now is ~2.5x

stargate/content_cache_evictions_flash 9815

stargate/content_cache_evictions_memory 15969

stargate/content_cache_flash_page_in_pct 4

stargate/content_cache_flash_spills 9815

stargate/content_cache_hits_pct 98                         <---- good

stargate/content_cache_lookups 261701

stargate/content_cache_multi_touch_flash_max 21474836480

stargate/content_cache_multi_touch_flash_usage 21474836480

stargate/content_cache_multi_touch_memory_max 1899180856

stargate/content_cache_multi_touch_memory_usage 1900703744

stargate/content_cache_page_in_from_flash 13062

stargate/content_cache_single_touch_flash_max 0

stargate/content_cache_single_touch_flash_usage 0

stargate/content_cache_single_touch_memory_max 474795208

stargate/content_cache_single_touch_memory_usage 473272320

stargate/content_cache_usage_flash_mb 20480               <---- 20G flash

stargate/content_cache_usage_memory_mb 2264            <---- 2.2G RAM

#2. Medusa ExtentGroupId Map (where SHA1s are kept) hit/miss info:

wget -O- 'http://localhost:2009/h/vars?regex=medusa.cache.extent_group_id_map&format=text'

medusa/cache/extent_group_id_map/current_size_bytes 268430856

medusa/cache/extent_group_id_map/entries 9496

medusa/cache/extent_group_id_map/evictions 293020 <<< to reduce evictions increase the extent_group_id cache.

medusa/cache/extent_group_id_map/hits   28665029

medusa/cache/extent_group_id_map/misses 923660

medusa/cache/extent_group_id_map/insertions 9417038

medusa/cache/extent_group_id_map/max_size_bytes 268435456  ---> 256MB cache size for this map

Medusa Extent ID map hit ratio is 96.8% (28665029/(28665029+923660)). 

After increasing medusa extent id cache to 2G:

medusa/cache/extent_group_id_map/current_size_bytes 1644951348

medusa/cache/extent_group_id_map/entries 75678 (increased)

medusa/cache/extent_group_id_map/evictions 0 ( zero evictions)

medusa/cache/extent_group_id_map/hits 71550917

medusa/cache/extent_group_id_map/insertions 24404990

medusa/cache/extent_group_id_map/max_size_bytes 2147483648

medusa/cache/extent_group_id_map/misses 875467    (reduced)

#3. Write Path Live (30sec) Metrics

wget -O- 'http://localhost:2009/h/vars?regex=stargate.dedup&format=text'

stargate/dedup_fingerprint_added_bytes 8355840

stargate/dedup_fingerprint_cleared_bytes 8290304



III. Disable Dedup:
ncli ctr edit name=xyz fingerprint-on-write=off

stargate.gflags: stargate_disable_dedup_on_read=true