Content-Based Textual Big Data Analysis and
Compression
Fei Gao Ananya Dutta
Department of Computer Science, Department of Computer Science,
Lamar University Lamar University
fgao@lamar.edu adutta@lamar.edu
ABSTRACT
With the growing enhancement of technology and the Internet,
the number of people who are using the Internet is increasing
daily. Users are engaged in web searching and accessing
different types of websites, such as social media, banking, etc.
As a result, a large volume of data is being generated in every
day. It is necessary to load this data for analysis purposes.
However, memory space and transmission time are the most
important factors of limited processing. In most cases, we only
need to extract the important textual data from these vast raw
datasets. In this work, we propose content-based compression
(CBC) for textual data analysis on the basis of the Huffman
Code. The data is pre-analyzed to find very high frequency
words and then a shorter symbol is inserted to replace those
words. This compression approach is performed in an effort to
maintain the original format of the data so that, compressed
data structure could be completely transparent to Hadoop
platform. The algorithm is evaluated on a set of real world data
sets (e.g. Amazon movie review, food review, etc.) and a 52.4%
average data size reduction is obtained from the experiment.
Though this gain may seem modest, this can be supplementary
to all other compression optimization techniques. Furthermore,
the proposed technique can be effectively applied for the big
data optimization purpose.
CCS Concepts
• Information systems → Data management systems →
Data structures → Data layout → Data compression
Keywords
Compression; text-based encoding; Huffman Tree Algorithm;
Hadoop
1. INTRODUCTION
Compression can be viewed as an encoding technique where
the data contents can be represented in a special way that
satisfy a purpose of encoding. Furthermore, compression is a
subdivision of information theory [1] which is focused on
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and
that copies bear this notice and the full citation on the first page.
Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee. Request permissions
from Permissions@acm.org.
ICCBD '18, September 8–10, 2018, Charleston, SC, USA
© 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-6540-6/18/09…$15.00
DOI: https://doi.org/10.1145/3277104.3277107
Jiangjiang Liu
Department of Computer Science,
Lamar University
jliu@lamar.edu
encode and decode in such a way to obtain effective consequences.
The basic characteristic of compression is that it takes sequences of
characters or strings in any form, such as ASCII, and transforms the
sequences into new characters or symbols in bits where the
information remains the same but is reduced in length. The important
target area for compression is data storage and transmission. This
fact is also true for our personal computer system where we can save
memory and reduce the load of input and output channels (e.g. data
bus, address bus, etc.).
1.1 Big Data & Compression
“Big Data” is a very significant term commonly used in the area of
computer technology. The volume of data is growing rapidly. It is
equally valuable for both business and scientific research. This
motivates us to innovate new types of observations, measurements,
and storage mechanisms. By analyzing data, we can find the patterns
behind it and then design different types of decision making
applications that can be used effectively in business, healthcare,
public service, and security sectors. However, constantly increasing
data size causes constant pressure for analytical platforms that are
designed only for big data analyses, such as Hadoop and Spark,
which work under the limits of computational power, storage
capacity, etc. Therefore, if we can effectively implement the
compression technique, we will enhance the efficiency of our
analytical tools and algorithms by working on that compressed data.
1.2 Benefits of Data Compression
Compression technique speeds up the data transfer across the
network to and from the disk. By compressing information, we can
solve our storage capacity limitation issue. Performance can also be
enhanced by achieving optimization in data compression in [2].
Substantial energy and power savings can be obtained through
compressing data. We can also be benefited in cost reduction by
dealing with less data. The savings potential in data compression
comprises a better scale in terms of size, speed, and power
consumption of implementation (which will be used to do
compression and decompression) than uncompressed data processing.
Hence, these overheads will continue to decrease over time.
1.3 Compression Issues
There are currently several compression methods available, such as
7zip, lzw, and snappy. However, these algorithms can squeeze the
data contents at block level and disregard the order of information.
Thus, the data format and structure are jumbled, and compressed
data may lose its original information.
The popular 7zip, bzip, gzip, and zip compression methods use a
dictionary for their compression. Therefore, even though the
compression ratio is good for a very large file in those methods,
carrying a dictionary adds overhead that can be observed during the
compression for a comparatively small file.
7
1.4 Our Contribution
The target of this work is to design a content-based
compression method for big textual data analysis in a local
system. We have extended the Huffman Code and replaced a
word/text with a symbol after performing OR and bitwise left
shift operations on a Huffman encoded string, while
maintaining the original properties of the input file, such as
data ordering, format, structure, and punctuation.
The principle behind our implementation is to translate any
type of data into a code as well as the special characters. If we
can substitute the word with a shorter code and keep all other
regular expressions unaffected, then the compressed data
structure will be completely transparent to any analytic
platform, such as Hadoop. Thus, for any textual data analysis
programs, such as sentiment, semantic analysis, and Page Rank
calculation, we can implement them on the compressed data as
well. Our compression algorithm is a loss-less data
compression scheme and case-insensitive to the strings.
The organization of this paper is as follows: Section 2
describes related work in the data compression techniques,
which includes previous data compression techniques. Section
3 presents the implementation of our proposed algorithm on
data compression. Section 4 illustrates the experimental results
based on applying our algorithm and analysis of the results. In
Section 5 we delineate the future works that can be done as an
extension to the present work.
2. RELATED WORKS
In this chapter, we provide a brief overview of prior related
works on the data compression techniques of potential big data
analysis so that we can obtain some ideas about the past and
current ongoing research works in this field.
A more recent work on content-based compression for big data
was proposed by [3]. They proposed a new approach called the
Content-aware Partial Compression (CaPC), which uses a
dictionary-based approach. Basically, it consists of two
independent parts. All recent compression methods destroy the
original data structure and format after compression, but CaPC
will maintain the original data structure even after the
compression. This guarantees that the Map Reduce program
can work with CaPC-compressed data directly without
decompression. Its experimental results show 30% average
data size reduction. And on a Hadoop cluster for some I/O
intensive jobs it increases up to 32% performance.
A survey studied compression algorithms that have been
implemented in Hadoop package recently [4]. This survey
depicts the brief overview of the most currently used
compression methods: LZO, GZIP, BZIP2, LZ4, and Snappy.
They also have performed a comparison among these methods
in the Hadoop platform for a very large size of data.
A decision algorithm was developed to help Map-Reduce users
identify when and where to use compression [5]. Their
experiment results show that compression can improve
performance and energy efficiency for Map-Reduce workloads.
Therefore, compression provides 35-60% energy savings for
reading heavy jobs as well as jobs with highly compressible
data. For some jobs, using compression enhances energy
savings up to 60%. Based on their measurements, they
construct an algorithm which examines per-job data
characteristics and IO patterns. It also determines the field in
which to use compression and the exact time. Also,
compression for I/O performance was studied and proposed to
alleviate I/O bottleneck for big data [7,8].
3. CBC DESIGN AND MPLEMENTATION
3.1 CBC Compression Criteria
We were motivated by the content-aware compression technique and
focused to solve their issue with generality [3]. Since preserving the
original data format and structure are two of the objectives, any
meta-characters must not be encoded and should remain unchanged
during the compression.
3.2 Word Count Job
When we pass an input file, we need to calculate the frequency of
every word. These frequencies will be used as the weightage for each
word to build a Huffman tree. Moreover, we have also designed
partial-compression where we can establish a constraint depending
on these frequencies. The constraint in our partial-compression is as
follows “Add up the frequencies of the words, calculate the
percentage, and choose how many percentage of
high frequency words that you want to select for partial
compression”.
3.2.1. Data Structure
Our main objective is to design an algorithm that can support small
and large volumes of data sets. Thus, we need to select data structure
in such a way that it can support unlimited number of data. We have
used java TreeMap data structure for this purpose. By using
TreeMap data structure, we can efficiently store data in key/value
pairs. It allows rapid retrieval and achieves the mechanism of log (n)
time complexity. It also guarantees that the elements will be sorted in
ascending order rather than any other data structure.
3.2.2. Regular Expression
When we count frequency for a given input file, we take only the
words and keep the special characters in their place as in original
form. Therefore, words with and without special characters are the
same for counting frequency. Moreover, we do not count the
frequency of these special characters.
3.2.3. Customized Sorting
After we continuously update the TreeMap with frequency count, we
will perform an additional sorting mechanism to sort in ascending
order after all the words are included with their frequencies in the
TreeMap data structure.
3.3 Huffman Tree Encoding Mechanism
Our content-based compression is based on Huffman encoding,
which uses variable-length code for high frequency symbols[6].
3.3.1. Data structure
We have used a customized Linked List to support large data sets for
compression. We access first two elements of a linked list to make
the first two leaf nodes of the tree, then we reset the next head of a
linked list, omitting the first two elements as they are already
accessed.
Every combined parent node has compared with other elements in
the linked list. The built Huffman Tree has been stored in a hash map
with encoded words.
3.3.2. Partial and Full Encoding Methods
To implement a partial compression technique, we have summed up
all the frequency counts then calculated the percentage. We have
taken three measurements as threshold values, i.e. 90%, 50%, 60%.
8
We send these threshold values as an encoding limit to the
Huffman Tree algorithm. This indicates that input files can be
compressed up to 90%, 50%, or 60% of high frequency words.
Excluding this variant, we are able to apply the full Huffman
encoding and compress an entire input file.
3.4 Partial Compression
3.4.1. Partial Compression
If we proceed with the following random example, “quick
brown fox quick brown fox jumped over a lazy lazy dog,”
then with the 90% partial Huffman encoding we get the
following encoded sequence: “100 brown 01 111 01 jumped
101 110 00 00 110”. Our next approach for compression is
divided into the following steps:
• First, when we obtain an encoded word, we start
counting. If there is no encoded word, we will use OR
and the bitwise left shift operation. Furthermore, we pad
with zeroes to make it 8 bits long due to the system’s
required word length. When we send it to the system
background, it generates a character or symbol to store in
a file.
• If we obtain a non-encoded word in any place of the
sentence, we will directly write it into the files. In this
case, the count will be zero.
• If we obtain consecutive numbers of encoded words then
use OR and bitwise left shift operation, then we can
concatenate all the encoded words up to 8 bits in length.
Next, we use a count to keep record of the encoded
words, which will help during compression. Then we
send this concatenate string to the system and it will
generate one symbol and store it to the file. Hence, the
key factor works for the compression we will receive:
“01 111 01.”–These are the encoded strings for three
different words, but we will get only one symbol by our
compression technique. “01 111 01” are the encoded
strings for “fox quick brown fox” respectively. These
words take total 18*8 bit/character = 144 bits to
represent. However, through our compression, we will
take only 8 bits instead of 144 bits. Hence, the total
improvement is ((144-8)/144) * 100 = 94%.
• If we obtain only one encoded string at the last of the
sentence, we need follow the first step.
After performing the partial compression, we receive `brown
ªjumpedü as our partially compressed string in the compressed
file.
3.4.2. Full Compression
For full compression, if we proceed with the following
example, “Hi how are you?” the full encoded string we will get
from Huffman encoding is “01 111 00 110?” Now we can send
this encoded string for full compression. It will follow the next
step:
The encoded string will be divided into an 8 bits’ length string
using OR and bitwise left shift operation and sent to the system.
The system will generate symbols to represent these strings.
For the above sentence, the compressed code will be y€?. So
we can see that the original sentence takes 15
characters*8bits/character = 120 bits to represent. Whereas by
our compression, it will only take 3 characters*8bits/character
= 24 bits to represent and has ((120-24)/120) *100 = 80% total
improvement. We have decided to keep the special characters in the
original input structure, so we did not encode and compress any
special character.
3.5 Decompression Mechanism
For decompression, after we fetch any symbol from the compressed
file, we send it to the system which will return the bits representation
of that symbol. During compression, we keep track of how many
words are compressed in a given symbol. According to the count, we
divide the bits obtain from the system. If we obtain excessive zeros,
we will truncate those zeroes following AND and bitwise left shift
operation. Thus, we can receive the actual encoded string.
Furthermore, we send the encoding string to the Huffman tree
structure. In the Huffman tree, it traverse from root to the leaf to find
the actual word and return to system. Decompressed text for y € ?
mentioned in above example is “Hi how are you?”
We need to take special care at the time of decompression for
partially compressed files due to the technique. This technique is
used to identify whether the present string is compressed or not. So
we have extended our decompression method as follows:
When decompression technique fetches any character, meanwhile, it
checks the corresponding value stored in another file (array list,
linked list, and other storage data structure). For the non-encoded
word, corresponding value stored in that file is zero. So whenever we
fetch zero for any given string, we can decide that the string is not
encoded. Therefore, we do not need to send it to Huffman Tree.
Consequently, decompressed text for partially compressed string `
brown ªjumped ü mentioned in above example is “quick brown
fox quick brown fox jumped over a lazy lazy dog.”
4. RESULTS ANALYSIS AND
COMPARISON
We analyzed in various ways for the efficiency of different
compression methods. Then we measured the relative time
complexity, the space complexity, processing time (i.e. how fast the
algorithm performs), how closely the reconstruction matches the
original data, and the amount of compression gained. In this work,
we will be mainly concerned with the last two criteria. Let us take a
brief look at what those are.
A very logical way of measuring how well a compression algorithm
compresses a given set of data is to look at the ratio. We are
calculating it by:
(1)
Compression ratio is the ratio of how much compressed data is
achieved from the original file size. The lower compression ratio will
indicate the higher performance. In this section, we compared
compression ratio for our compression mechanism with others.
4.1 Results Analysis of CBC (Partial and Full)
Compression Technique with Huffman
Encoding
In our work, we have focused on both partial and full compression
techniques and their performances. Furthermore, we have compared
our partial and full compression performance with Huffman
encoding for different sized data files. We categorized the data files
in four parts: very small, small, medium, and large in each category.
9
When we compared the compression ratio of our CBC partial Table 1. Average compression time and ratio
and full compression with Huffman encoding technique, we Compression Average Compression Average
received the following result. Method Time (millisecond) Compression Ratio
Figures 1 & 2 show that five different ranges of very small CBC(Full) 21.91 0.37
data files, the CBC partial and full compression display a lower CBC(90%) 18.16 0.45
compression ratio compared to the Huffman encoding CBC(70%) 16.21 0.63
technique. Then, we analyzed the CBC partial and full CBC(50%) 13.85 0.77
compression technique with the Huffman encoding technique Huffman 48.9 0.97
for small sized files. We have categorized the small file sizes
in five different range and this is discussed as follows.
Therefore, we can summarize that our partial and full CBC
CBC-90% and CBC-full compression demonstrate a better compression technique are better in both compression ratio and
performance than Huffman encoding. Besides, CBC-70% and execution time than Huffman encoding technique.
CBC-60% partial encoding methods give us a faster execution
speed compared to the existing Huffman encoding technique 4.2 Results Analysis of CBC Full Compression
shown in Table 1.
Technique with Other Compression Techniques
As from our previous analysis, though partial and full CBC
Comparison of Compression Ratio of Different
compression methods are beneficial for better performance, the CBC
Algorithms (Very Small Size File) full compression method is much better than the CBC partial
2.5
compression. Thus, we decided to analyze the CBC full compression
Compression Ratio

2 scheme with other popular compression schemes.

1.5

0.5

0
Huffman CBC-50% CBC-70% CBC-90% CBC-Full

Original File Size = 150 bytes Original File Size = 300 bytes
Original File Size = 500 bytes Original File Size=700 bytes
Original File Size=1000 bytes

Figure 1. Comparison of compression ratio for very small

size data files

Figure 3. Comparison of compression ratio for medium size data

files

Figure 2. Comparison of compression ratio for small size

data files

Figure 4. Comparison of compression ratio for large size data

files
We have compared CBC full compression technique with 7zip,
Bzip2, Gzip, LZW, and Tar in five different sizes of data files. We
can see from the result, though the CBC full compression could not
give as small of a compression ratio as 7zip, Bzip2, and Gzip, it is

10
still giving a low compression ratio and better performance
than Tar and almost the same performance as LZW
compression schemes.
4.3 Results Analysis for Repeating
Compression for Very Large Data Files
The research is focused on implementing an algorithm to
reduce data file size for data that varies from gigabyte size to
petabyte. In fact, we could not gain optimal compressed size
by implementing the compression methods, such as 7zip, bzip2,
etc. Besides, they cannot perform well repeatedly due to
overwriting and increasing the size. For example, when we
compress one file using the 7zip method twice, the compressed
file size remains the same. If we apply the bzip2 on that 7
zipped compressed file, it will increase the size of the existing
compressed file. So now we will experiment with whether or
not we can reduce file size further with the CBC full
compression following with other popular compression
methods.
Figure 5 shows that executing the CBC full compression along
with other compression gives a much better result than when
performed alone.
5. CONCLUSION AND FUTURE WORKS
In our compression method, we basically targeted text data for
effective data analysis due to frequent use. Thus, we implemented
our experiment only with text-based unstructured real time raw data
sets and limited our works by textual compression technique. In the
future we can extend our work for application specific image, audio,
or video data compression.
Generally textual data does not contain many special characters that
require numbers of bits that are needed to be compressed. We did not
account for any specific special character in our compression method
due to keeping format, so the data structure was not destroyed.
However, we may utilize special characters for compression in our
future consideration of files such as xml. In the future, we can focus
on compression and decompression time by optimizing our
algorithm and programming structure.
Until now, compression technique was vastly accomplished and
effectively used for memory utilization and reducing the
transmission time. We have many compression techniques which
give us the best performance in both ways. But compression and
decompression times and analysis are adding an overhead for a big
data file, about which we should be concerned. In our study, we have
designed a compression methodology which will take care of these
three dimensions, such as memory space, transmission time, and
analyzation time all at the same time. Our proposed model is limited
to compress and analyze text-based data, and it sacrifices the
computation time to compress for achieving a better compression
performance and direct access to the compressed data. As the
frequent word is replaced by a symbol in our compression, the code
does not contain or match any internal information about that word.
Hence, it supports the high-level data abstraction as well. In
conclusion, we obtained an extremely good performance of
compression from small to medium sized file and a moderate
performance for a large data file.

6. REFERENCES
[1] Lelewer, Debra A and D. S. Hirschberg. 1987. “Data
Figure 5. Comparison of compression ratio for repeated Compression.” ACM Computing Surveys (September, 1987):
compression 261-296, Volume 19 Issue 3.
doi:10.1145/45072.45074.http://dl.acm.org/citation.cfm?id=450
4.4 Analysis of Supported Compression 74.K. Elissa, “Title of paper if known,” unpublished.
Methods in Hadoop with CBC [2] Thirunavukarasu, B., V. M. Sudhahar, U. VasanthaKumar, T.
There are four compression methods available in Hadoop to Kalaikumaran, and S. Karthik . 2014. “Compressed Data
work on a vast data set. Among them, three can give a good Transmission Among Nodes in BigData.” American Journal of
compression, but they cannot provide processing time and Engineering Research (AJER) (2014): 209-212, Volume-03,
execution speed benefit except storage utilization before Issue-06, e-ISSN : 2320-0847. p-ISSN : 2320-0936.
decompression.
[3] Dong, Dapeng and J. Herbert. 2014. “Content-aware Partial
Table 2 shows a 52.4% reduction in compression size for Compression for Big Textual Data Analysis Acceleration.”
performance analysis done in Hadoop. 2014 IEEE 6th International Conference on Cloud Computing
Technology and Science, Singapore (2014): 320 – 325.
Table 2. Compression result of Map-Reduce program in Accessed December 15-18, 2014. doi:
Hadoop 10.1109/CloudCom.2014.76.
Compression File Size Can Execute Map- [4] Lovalekar, Sampada. 2014. “A Survey on Compression
Technique Reduced Reduce Program Algorithms in Hadoop.” International Journal on Recent and
on Hadoop Innovation Trends in Computing and Communication (2014):
7zip 77% No 479 – 482, Volume: 2 Issue: 3, ISSN: 2321-8169.
Bzip2 77.6% No [5] Chen, Yanpei, A. S. Ganapathi, and R. H. Katz. 2010. “To
Gzip 67.6% No
Compress or Not To Compress - Compute vs. IO tradeoffs for
CaPC 30% Yes
CBC 52.4% Yes
MapReduce Energy Efficiency.” Proceedings of the first ACM
SIGCOMM workshop on Green networking - Green

11
 Networking (2010): 23-28. Accessed March 29, Heterogeneous Source Mining Algorithms, Systems,
2010.Technical Report No. UCB/EECS-2010 36. Programming Models and Applications - BigMine '12 (2012):
doi:10.1145/1851290.1851296. 45-52. Accessed December 15-18, 2014. doi:
[6] Huffman, David. “A Method for the Construction of 10.1109/CloudCom.2014.76.
Minimum-Redundancy Codes” Proceedings of the IRE 40 [8] Zou, Hongbo, Y. Yu, W. Tang, and H. M. Chen. 2014.
(9): 1098-1101. http://doi:10.1109/JRPROC.1952.273898. “Improving I/O Performance with Adaptive Data Compression
[7] Xue, Zhenghua, J. Li , Y. Zhang , G. Shen , Q. Xu, and J. for Big Data Applications.” 2014 IEEE International Parallel &
Shao. 2012. “Compression-Aware I/O Performance Distributed Processing Symposium Workshops (2014): 1228-
Analysis for Big Data Clustering.” Proceedings of the 1st 1237. Accessed May 19-23, 2014. doi:
International Workshop on Big Data, Streams and 10.1109/IPDPSW.2014.138.

12