Flux Research Group / School of Computing

Metadata Considered Harmful ... to Deduplication

Xing Lin, Fred Douglis, Jim Li, Xudong Li, Robert Ricci, Stephen Smaldone, and Grant Wallace

Proceedings of the 7th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage) 2015.



Deduplication is widely used to improve space efficiency in storage systems.  While much attention has been paid to making the process of deduplication fast and scalable, the effectiveness of deduplication can vary dramatically depending on the data stored.  We show  that many file formats suffer from a fundamental design property that is incompatible with deduplication: they intersperse metadata with data in ways that result in otherwise identical data being different. We examine three models for improving deduplication in the presence of embedded metadata: deduplication-friendly data formats, application-level post-processing, and format-aware deduplication.  Working with real-world file formats and datasets, we find that by separating metadata from data, deduplication ratios are improved significantly---in some cases as dramatically as 5.6x.