Professional Documents
Culture Documents
Hardware Compression in Storage Networks and Network Attached Storage
Hardware Compression in Storage Networks and Network Attached Storage
• Introduction
• Lossless Data Compression, Background
• Lossless Compression Algorithms
• Hardware versus Software
• System Implementation
• Power Conservation and Efficiency
• Technology advances and Compression Hardware
• The Sliding Window History Buffer adds one new Byte and
drops off one Byte from the back end of the history buffer
each time a Byte is input and processed
Size - 1 2 1 0
Up to 32K
byte sliding
window Current byte to be processed
Input Output
A A
B B
C C
D D
ABC Distance=4, Length=3
F F
CDAB Distance=6, Length=4
Probability Of occurrence
/\ Symbol Code Pr
0 1 A 10 0.25
/ \ B 0 0.5
B / \ C 110 0.125
0 1 D 111 0.125
/ \
*Reduction =
A / \
½[0.25(2) + 0.5(1) + 0.125(3) + .125(3)]
0 1 = 0.875
/ \
* Reduction in data size due to Huffman
C D encoding.
• Data dependent
– Random data provides poor compression ratio
performance
– Data with repeating Byte strings, 2 Bytes or longer
provides greater compression ratio performance
– Compression ratios greater than 100:1 are possible
– May expand if attempting to compress previously
compressed data, but a system could detect this and
send the original data
• Algorithm dependent
– Size of sliding window
– Static or dynamic Huffman encoding
– Number of matches tracked
– Length of matches the algorithm will search for
• Compression levels
– Level 1, 2 and 3 supports static Huffman
– Level 4-9 supports dynamic Huffman
• Each level has limits on:
– Number of matches it will track
– Length of matches it will search for
– Lower levels better for higher throughput
– Higher levels for better compression ratio
performance
3.5
3
Compression Ratio
3.5
Compression Ratio
3
2.5 2.5
2
2 1.5
1
1.5 0.5
0
1 ALDC LZS GZIP-1 GZIP-9
0.5
0
ALDC LZS GZIP-1 GZIP GZIP-9
Coprocessor
6
Compression Ratio
0
ALDC LZS GZIP-1 GZIP GZIP-9
Coprocessor
Users
NAS Appliance
Network
o
o
o Compression
(Most critical for
storage gain)
SAN Storage
NAS Appliance
Compression
(Bandwidth
gain)
Users
NAS Appliance
Network
o
o
o
SAN Storage
Compression
(Bandwidth
gain)
NAS Appliance
• System Issues
– Varying Compressed File Sizes
– Varying Latency
– Multiple Compression Processors
• 10G Ethernet
• Fiber Transceivers at 10Gbps
• PCI express, 8-lane, 16-lane
• Scatter/Gather DMA
• Up to 10 Gigabit/s LZ1 compression boards
• Additional Notes:
– This assumes you want to apply all the gain to power savings
– HVAC power savings is typically as great as the power loading
from the equipment removed.
– Most Enterprise class systems cannot tolerate the speed of GZIP
running in software.
100%
Power Consumption
33%
0
No Compression GZIP
• Milburn, Ken (2003).JPEG2000: The Killer Image File Format for Lossless Storage
[Electronic Version] Retrieved August 18, 2006
http://www.oreillynet.com/pub/a/javascript/2003/11/14/digphoto_ckbk.html
Dr Pat Owsley
Jason Franklin
Bill Thomson
Sean Gettmann