Stream-Based Lossless Data Compression Hardware Using Adaptive Frequency Table Management

In order to treat BigData efficiently, the communication speed of the inter or the intra data path equipped on high performance computing systems that needs to treat BigData management has been reaching to very high speed. In spite of fast increasing of the BigData, the implementation of the data communication path has become complex due to the electrical difficulties such as noises, crosstalks and reflections of the high speed data connection via a single cupper-based physical wire. This paper proposes a novel hardware solution to implement it by applying a stream-based data compression algorithm called the LCA-DLT. The compression algorithm is able to treat continuous data stream without exchanging the symbol lookup table among the compressor and the decompressor. The algorithm includes a dynamic frequency management of data patterns. The management is implemented by a dynamic histogram creation optimized for hardware implementation. When the dedicated communication protocol is combined with the LCA-DLT, it supports remote data migration among the computing systems. This paper describes the algorithm design and its hardware implementation of the LCA-DLT, and also shows the compression performance including the required hardware resources.