Carnegie Mellon University
Browse
- No file added yet -

DiskReduce: RAID for Data-Intensive Scalable Computing (CMU-PDL-09-112)

Download (302.41 kB)
journal contribution
posted on 2009-11-01, 00:00 authored by Bin Fan, Wittawat Tantisiriroj, Lin Xiao, Garth GibsonGarth Gibson
Data-intensive file systems, developed for Internet services and popular in cloud computing, provide high reliability and availability by replicating data, typically three copies of everything. Alternatively high performance computing, which has comparable scale, and smaller scale enterprise storage systems get similar tolerance for multiple failures from lower overhead erasure encoding, or RAID, organizations. DiskReduce is a modification of the Hadoop distributed file system HDFS) enabling asynchronous compression of initially triplicated data down to RAID-class redundancy overheads. In addition to increasing a cluster's storage capacity as seen by its users by up to a factor of three, DiskReduce can delay encoding long enough to deliver the performance benefits of multiple data copies.

History

Publisher Statement

All Rights Reserved

Date

2009-11-01

Usage metrics

    Keywords

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC