Product : NetApp, NetApp HCI [HCI]/1.4, x86
Feature : Dedup/Compr. Process, Efficiency, Data Services
Content Owner:  Herman Rutten
Summary
Deduplication: inline (post-ack)
Compression: inline (pre-ack) + post process
Details
Deduplication can be performed in 4 ways:
1. Immediately when the write is processed (inline) and before the write is ackowledged back to the originator of the write (pre-ack).
2. Immediately when the write is processed (inline) and in parallel to the write being acknowledged back to the originator of the write (on-ack).
3. A short time after the write is processed (inline) so after the write is acknowleged back to the originator of the write - eg. when flushing the write buffer to persistent storage (post-ack)
3. After the write has been committed to the persistent storage layer (post-process).

The first and second methods, when properly integrated into the solution, are most likely to offer both performance and capacity benefits. The third and fourth methods are primarily used for capacity benefits only.

NetApp HCI leverages global inline deduplication and global inline as well as post compression techniques. Incoming writes are broken into 4K blocks, assigned a hash (BlockID), compressed and then written to the NVRAM of two storage nodes in order to protect the data. When the internal Block Service identifies that the BlockID already exists, which means that the data has already been committed to the persistent flash layer, it does not destage the data twice. When destaging unique compressed 4K blocks from NVRAM to the persistent flash layer, blocks going to the same SSD are consolidated into 1MB chunks.