Design and Implementation of HDFS Data
Encryption Scheme using ARIA Algorithm on
Hadoop
Youngho Song, Young-Sung Shin, Miyoung Jang, Jae-Woo Chang*
Dept. of Computer Engineering
Chonbuk National University, Republic of Korea
{songyoungho, twotoma, brilliant, jwchang}@jbnu.ac.kr
*Corresponding author
Abstract—Hadoop is developed as a distributed data
processing platform for analyzing big data. Enterprises can
analyze big data containing users’ sensitive information by using
Hadoop and utilize them for their marketing. Therefore,
researches on data encryption have been widely done to protect
the leakage of sensitive data stored in Hadoop. However, the
existing researches support only the AES international standard
data encryption algorithm. Meanwhile, the Korean government
selected ARIA algorithm as a standard data encryption scheme
for domestic usages. In this paper, we propose a HDFS data
encryption scheme which supports both ARIA and AES
algorithms on Hadoop. First, the proposed scheme provides a
HDFS block-splitting component that performs ARIA/AES
encryption and decryption under the Hadoop distributed
computing environment. Second, the proposed scheme provides a
variable-length data processing component that can perform
encryption and decryption by adding dummy data, in case when
the last data block does not contains 128-bit data. Finally, we
show from performance analysis that our proposed scheme is
efficient for various applications, such as word counting, sorting,
k-Means, and hierarchical clustering.
Keywords— Hadoop Security, Data Encryption, HDFS Data
Encryption, ARIA Encryption Algorithm, Hadoop Encryption
Codec
I. INTRODUCTION
Recently, due to the prevalence of smart phones, the
number of data generated by Social Networking Services
(SNS) has rapidly been increasing. The data including
multimedia contents with various forms are grown to big data
scale. Meanwhile, the MapReduce framework can support
parallel data processing based on both map and reduce
functions for a distributed big data processing computing
environment. Hadoop [1] is an open-source implementation of
the MapReduce framework. Because Hadoop is used for the
major projects of Facebook and Amazon, it is considered as a
standard distributed parallel processing system for dealing with
big data. Currently, companies analyze customers’ information
and location data collected in various fields and utilize them for
a marketing activity. Therefore, personal sensitive data can be
disclosed while analyzing customers’ data. To solve this
problem, studies on the security of Hadoop [3-6] have actively
done. First, Kerberos [3, 4] protects the disclosure of data
according to network attacks. Hadoop can block the accesses of
malicious attackers by using user authentication through
978-1-5090-3015-6/17/$31.00 ©2017 IEEE
Kerberos. Second, studies on data encryption using a
compression codec have been done to prohibit data disclosure
in HDFS (Hadoop Distributed File System). Liu Yi[5]
proposed a SplittableCompressionCodec to solve the problem
that the existing compression codec cannot support the
encryption on distributed data due to using only a single map
for input data compression. The SplittableCompressionCodec
can compress and decompress HDFS data, in a parallel way,
which consist a set of blocks because multiple maps are
allowed to process HDFS blocks. In addition, S. Park[6]
proposed a scheme which can support the AES encryption of
HDFS data by using the SplittableCompressionCodec and Java
cipher library. However, both studies have a limitation that
they support only the AES encryption algorithm.
However, the ARIA algorithm[9, 10] is widely used in
many public institutions because the Korean government
selected it as a standard data encryption scheme for domestic
usages. The ARIA algorithm performs data encryption based
on a 128-bit block, like the AES algorithm. In this paper, we
propose a HDFS data encryption codec scheme using both
ARIA and AES encryption algorithms for Hadoop. First, the
proposed encryption scheme provides a HDFS block-splitting
component that performs both ARIA encryption and
decryption in distributed computing environment by dividing
data into HDFS blocks. Second, we provide an encryption
scheme supporting a variable-length data processing that
performs the encryption and decryption of variable-sized data
by adding dummy data. Finally, we provide a codec selection
component that selectively utilizes the existing AES codec and
the proposed ARIA codec.
The structure of this paper is as follows. In Section 2, we
introduce the HDFS encryption scheme on the existing AES
algorithm. In Section 3, we present the encryption scheme of
HDFS data using the ARIA algorithm. In Section 4, we
compare the performance of the proposed ARIA-based
encryption scheme with the exiting AES-based encryption
scheme. Finally, we conclude the paper with future work in
Section 5.
II. RELATED WORK
A. Encryption Algorithm
The advanced encryption standard (AES) algorithm [7, 8] is
a symmetric key-based encryption one that was announced by
84 BigComp 2017
US National Institute of Standards and Technology(NIST). The
AES encryption algorithm is widely used in various data-
related fields as US federal information processing standard.
Meanwhile, the ARIA algorithm [9, 10] is a block cipher
algorithm designed in 2003 by a large group of researchers
from government agency, academy, and research institutes in
South Korea. In 2004, the Korean Agency for Technology and
Standards selected it as a standard cryptographic technique.
The ARIA algorithm processes data with 128-bit block size
and uses keys with 128/192/256-bit size. The ARIA algorithm
provides bit operations such as XOR and can support data
protection by executing the encryption of 12/14/16 rounds
depending on performance requirements. Because the ARIA
algorithm performs data encryption based on 128-bit block,
like AES algorithm, it can substitute the AES algorithm. Table
1 shows the comparison of the AES algorithm and the ARIA
algorithm.
TABLE I. COMPARISON OF THE AES AND ARIA
AES ARIA
Period 1999 2004
National Institute of
National Security
Main Agent Standards and
Research Institute
Technology
National service
Target Administrative service
Private service
Standard of domestic
Characteristic and international
AES alternative
domestic technology
encryption
B. HDFS data encryption methods based on AES encryption
algorithm
Liu Yi et al.[5] proposed a SplittableCompressionCodec to
perform the existing encryption in distributed environment.
The existing compression codec suffers from an efficiency
problem because it performs encryption with a single map
function. the SplittableCompressionCodec supports parallel
compression and decompression functions for HDFS data that
can be divided into several blocks by assigning each HDFS
block to a map function. Because Hadoop supports the
SplittableCompressionCodec in 1.x and 2.x version, there is no
compatibility issues. The user only needs to indicate
CompressCodec class name in the configuration file to make
use of it.
Fig. 1. Encryption using Hadoop compression function
S. Y. Park et al.[6] proposed a scheme which can support
the AES encryption of HDFS data using the JAVA cipher
library. Because the cipher class is related to encryption library,
85
the cipher can inherit such encryption algorithms as AES, DES,
and RSA (Fig. 1). To override the cipher class to the Splittable-
CompressionCodec, the scheme performs both encryption and
decryption based on the selected encryption algorithm. The
limitations of the existing encryption schemes are as follows.
First, the exiting AES encryption schemes support the
encryption of data with a fixed 128-bit block size. As a result,
the exiting AES encryption cannot support schemes the
encryption of a variable-length data block. Second, the existing
encryption schemes cannot support the ARIA encryption
algorithm that is a standard encryption algorithm in South
Korea.
III. DESIGN OF PROPOSED ARIA ENCRYPTION SCHEME
A. System Architecture
Though the ARIA encryption algorithm is designated as a
standard encryption algorithm by South Korea, to the best of
our knowledge, there is no research that utilizes ARIA data
encryption algorithm on Hadoop. In this paper, we design a
HDFS data encryption scheme using both ARIA and AES
encryption algorithms. Figure 2 shows the overall system
architecture of the proposed HDFS data encryption scheme.
The encryption of the data stored in the master local file system
is performed as follows. i) The user specifies the preferred
encryption algorithm between the ARIA and the AES
algorithms. ii) The data encryption codec encrypts data in
HDFS based on the selected encryption algorithm. iii) The
encrypted data are distributed into data nodes with the 64MB-
unit of HDFS blocks. The data decryption is done in the
reverse order of the data encryption.
Fig. 2. Structure for encrypting HDFS data
To process the encrypted data in the MapReduce, it is
required not only to decrypt data before computation, but also
to encrypt the result before storing it to the HDFS. The overall
processing procedure of the encrypted data in MapReduce is
described in Figure 3. i) The input data of Map function is
decrypted before the Map function is processed. ii) When the
Map function is completed, the output data is encrypted and
stored back into the HDFS. iii) The Reduce function is run
after decrypting the intermediate result of the Map function in
HDFS. iv) When the Reduce function is completed, the final
result is encrypted and stored into HDFS. Based on the
sensitivity of the intermediate result of the Map function, the
step ii) and iii) can be omitted.

Fig. 3. Processing of Hadoop mapreduce on encrypted HDFS

B. Encryption procedure of the proposed scheme in HDFS

The data encryption of the proposed scheme is performed
as shown in Figure 4. i) The user selects the preferred Fig. 4. Procedure of data encryption
encryption CODEC between the existing AES and the
proposed ARIA, and calls the Hadoop encryption component
to encrypt input data. To convert the input data into a HDFS
block with 64MB unit, Hadoop encryption component calls the
HDFS block splitting component. ii) Because the existing
HDFS encryption scheme cannot process a variable-length data,
we newly design a variable-length data processing component
that can check whether the last data sub-block has a 128-bit
unit or not. The variable-length data processing component is
called by the HDFS block splitting component. If the size of
data stored in the last sub-block is a 128-bit unit, the variable-
length data processing component adds dummy data to make
the last sub-block have a 128-bit unit. iv) The encryption
component performs the encryption of each data sub-block by
using the selected encryption CODEC (ARIA or AES). Finally,
the encrypted sub-blocks are merged into the encrypted HDFS
block, which is stored into the corresponding data node by the
HDFS block splitting component.
Fig. 5. Hadoop encryption component
1) Hadoop encryption component: The users select a
preferred encryption algorithm to encrypt their data from the
Hadoop encryption component. Figure 5 shows an overall
architecture of the Hadoop encryption component. In the
figure, the users select an encryption algorithm between ARIA
and AES algorithms and override the encryption CODEC
through the algorithm selection module. The data block
generated by HDFS block splitting component is encrypted by
the CODEC and the encrypted data is returned to the users.
Fig. 6. HDFS block splitting component
2) HDFS block splitting component: Because the existing
ARIA algorithm normally performs the encryption of the 3) Variable-length data processing component: The AES
entire database, the algorithm cannot perform the encryption and ARIA algorithms only support the encryption of a fixed-
of data in HDFS, which is stored in a distributed manner. To length data block with 128-bi unit. As a result, the existing
solve this problem, we apply the SplittableCompressionCodec AES and ARIA algorithms using Hadoop have a problem that
[5] into our encryption module in order to divide data into they cannot support the encryption of variable-length data. To
HDFS block with 64MB unit and to encrypt data by using the solve this problem, we propose a variable-length data
same encryption key. processing component that automatically checks the data

86
block size and adds dummy data if the data is not composed of
a 16-byte unit. In addition, the proposed component can be
applied to the existing AES encryption scheme. Figure 7
shows the processing of variable-length data.
(a) Encrypt data that is less then 16 bytes
(b) Encrypt data that is more then 16 bytes
Fig. 7. Encryption of variable-length data
Algorighm 1 shows a variable-length data processing
algorithm that performs the encryption of data with sub-block
unit. The algorithm divides the input file into a set of 16-byte
sub-blocks and performs the encryption of each sub-block.
First, the variable-length data processing algorithm checks if
the sub-block to be encrypted is the last sub-block. If the sub-
block is not, the algorithm reads the sub-block with 16-byte
length and allocates it to the encryption sub-block. Second, the
sub-block is the last one, the algorithm checks whether or not
the last sub-block is shorter than 16-byte unit. If it is shorter,
the algorithm appends a dummy data at the end of the last sub-
block to make it to be a 16-byte unit(line 3~8). The algorithm
reads the sub-block with 16-byte length and allocates it to the
encryption sub-block. Finally, the algorithm encrypts the
entire block by merging a set of the encrypted sub-blocks.
87
Algorithm 1: Variable-length data processing algorithm
Input : Plain text file
Output : OutputStream that contains encrypted block
C A : 01: CurrentOff = 0
02: while CurrentOff >= EOF do
03: if CurrentOff + 16 >= EOF then
04: CodecBlk = Read InputData to EOF
05: CurrentOff = CurrentOff + Read Bytes
06: if CodecBlk.size % 16 != 0 then
07: Fill CodecBlk
08: end if
09: else
10: CodecBlk = Read InputData 16 Bytes
11: CurrentOff = CurrentOff + 16
12: end if
13: encrypt(CodecBlk, OutputStream)
14: end while
15: return OutputStream
4) Encryption component: The encryption component
performs the encryption of intermediate data generated by the
variable-length data processing component. There are two
types of the encryption based on the selected encryption
scheme, i.e, ARIA and AES. First, when the user selects the
ARIA algorithm as an encryption one, the encryption
component performs the encryption by using the encryption
key being received from the user. The encryption component
returns the encrypted data to the user through three components,
such as variable-length data processing component, HDFS
block splitting component, and Hadoop encryption component.
Second, when the user selects the AES algorithm an encryption
one, the encryption component performs the encryption by
using the encryption key from the user. The encryption
component returns the encrypted data to the user through the
same three components.
C. Decryption procedure of proposed scheme
The data the decryption procedure of the proposed scheme
is performed as shown in Figure 8. i) The user selects a
preferred encryption algorithm between the AES and the ARIA
and decrypts input data by calling the Hadoop decryption
component. To divide the input data into HDFS blocks with
64MB unit, the Hadoop decryption component uses the HDFS
block splitting component. ii) The HDFS block-splitting
component merges into a single block the encrypted data being
distributed in HDFS. iii) The variable-length data processing
component splits the input single block into sub-blocks with
128-bit unit. iv) The decryption component performs the
decryption of each sub-block by using the selected decryption
algorithm, i.e. ARIA or AES. Finally, the decrypted sub-blocks
are merged into the HDFS block by using the variable-length
processing component. The HDFS block is stored into the
corresponding data node through the HDFS block
splitting/merging component.
 encryption of constant values, such as encryption key and
block size.

IV. PERFORMANCE EVALUATION

Fig. 8. Procedure of data decryption
D. Detailed design of proposed scheme
In this section, we describe the detailed design of the
proposed HDFS data encryption scheme using ARIA algorithm.
Figure 9 shows the overall architecture of the proposed ARIA
CODEC. In the figure, i) AriaParallelCryptoCodec class that
implements the Hadoop encryption component calls the
AriaCryptoOutputStream. ii) AriaCryptoOutputStream class
that implements the HDFS block splitting/merging component
returns the encrypted data by splitting the original data into
HDFS blocks with 64MB unit. iii) AriaEncrypt class divides
the HDFS block into sub-blocks with 16-byte unit and checks
if the last sub-block has a 16-byte unit. The variable-length
data processing component in ARIAEncrypt class performs the
encryption of the sub-blocks without 16-byte unit. iv)
ARIAEngine class performs the encryption of sub-blocks
through the ARIAEncrypt class.
A. Experimental environment
We implement the proposed ARIA-based HDFS encryption
scheme on Java Eclipse ver. 1.8.24. In addition, we use a JAR
file format to aggregate encryption class files so that users can
apply the ARIA encryption CODEC without modifying the
Hadoop internal source codes. Moreover, we use two types of
typical queries: string data processing algorithm and scientific
data analysis algorithm. For the string data processing, we use
the WordCount and the sorting algorithms, both of which are
the most typical ones. For the scientific data analysis, we use k-
Means clustering algorithm [12]. and the hierarchical clustering
algorithm [13]. The k-Means clustering algorithm is the typical
scientific data processing one based on the nearest neighbor
data search. The hierarchical clustering algorithm requires
more complex and intense data computation then k-Means
clustering. Table 2 shows four applications used for the
performance evaluation.
TABLE II. APPLICATIONS OF PERFORMANCE EVALUATION
Application Reason of select
Most basic application that can understand the
Word Count
operation process of Mapreduce framework
Sort
String types bigdata processing Application that
has large number of arithmetic operations
k-Means
The basic algorithm for solving Clustering
problem
Hierarchical Application that perform complex clustering
Clustering amount of calculation than k-Means

Table 3 shows the dataset description used in our

experiments, where both Wikipedia English dump data [14]
and MetaWiki stub dump data [15] are the XML data set.
MODIS AQUA data [16] includes the seaweed information of
marine science for a year.

TABLE III. DATA USED FOR APPLICATIONS

Application Type of data Size

Dump of English Wikipedia in

WordCount 86GB
25.07.2015
Dump of English Metawiki
Sort 6.45GB
Stub in 10.08.2015
368.05MB
k-Means MODIS AQUA data in 10.2015
15 million records

Fig. 9. Detailed design of our ARIA-based HDFS data encryption scheme Hierarchical 1.34GB
MODIS AQUA data in 2012
Clustering 75 million redocrds

We run the four algorithms on a cluster with three data

The decryption procedure has the reverse procedure of nodes under the experimental environment described in Table 4.
encryption. In this figure, the AriaConstant class manages the

88
 TABLE IV. PERFORMANNCE EVALUATION ENVIRONMENT
Item Performance
CPU Intel(R) Core i3-3240 CPU @ 3.40GHz
Memory 12GB
OS Ubuntu 12.04.2
B. Performance evaluation of WordCount
Figure 10 shows the performance evaluation result using
the WordCount algorithm. NoEncrypt and AES require
12307.8 and 13190.4 seconds to perform the WordCount
algorithm, respectively. On the other hand, the ARIA algorithm
requires 13556.2 seconds, which indicates a slight performance
declination compared to the NoEncrypt. In addition, our ARIA
encryption shows a similar performance to that of the AES
algorithm.
Fig. 10. Result of wordcount Application
Fig. 11. Result of sort Application
C. Performance evaluation of sorting algorithm
Figure 11 shows the performance evaluation result using the
sorting algorithm. NoEncrypt and AES show 1947.6 and
2016.6 seconds on the average, respectively. On the other hand,
the ARIA algorithm requires 2081.6 seconds, which indicates a
slight performance degradation compared with the NoEncrypt.
In addition, our ARIA encryption shows a similar performance
89
to that of the AES algorithm. From the experiment, we can
show that the encryption cost for string data do not have much
influence on the performance between the AES encryption
algorithm and the ARIA encryption algorithm.
D. k-Means algorithm
Figure 12 shows the performance evaluation result of the k-
Means algorithm. NoEncrypt and AES require 440 and 462.4
seconds to perform the k-Means algorithm, respectively. AES
encryption shows approximately 5% performance degradation
compared to the NoEncrypt. On the other hand, ARIA
algorithm requires 478.4 seconds, which indicates about 9%
performance degradation compared to the NoEncrypt and 3%
degradation compared to AES encryption, respectively. The
performance trend for the k-Means algorithm using ARIA
encryption is similar to those of WordCount and the sorting
algorithm. We analyze that the encryption and decryption costs
are relatively higher than the computation cost because the k-
Means algorithm requires less computation overhead than other
clustering algorithms.
Fig. 12. Result of k-Means clustering
Fig. 13. Result of hierarchical clustering
E. Hierarchical clustering algorithm.
Figure 13 shows the performance evaluation result of the
hierarchical clustering algorithm. NoEncrypt and AES require
1460 and 1470.2 seconds to perform hierarchical clustering,
respectively. On the other hand, ARIA algorithm requires
1460.6 seconds, which indicates about 2% performance
degradation compared to the NoEncrypt and 1% performance
degradation compared to AES encryption. Because the
hierarchical clustering algorithm requires high computation for
clustering, the performance degradation of the ARIA algorithm
is significantly reduced. For clustering algorithms with high
computation, we show that the performances of the AES and
ARIA encryption schemes have a less dependence on both data
encryption and decryption costs.
V. CONCLUTION AND FUTURE WORK
In this paper, we proposed a HDFS data encryption scheme
using the ARIA encryption algorithm on Hadoop. The ARIA
encryption algorithm has gained its importance since it
becomes the standard encryption scheme in South Korea. The
proposed scheme provided the HDFS block-splitting
component that supports encryption and decryption of data in
Hadoop by dividing the data into HDFS blocks. We also
introduced the variable-length data processing component that
supports the encryption of not only the data with a 128-bit unit
but also variable-length data. In addition, we can choose one of
encryption schemes between the international standard AES
and the Korean standard ARIA. We showed from the
performance evaluation that our proposed scheme can support
the ARIA-based encryption for various applications, such as
word counting, sorting, k-Means, and hierarchical clustering.
Our ARIA-based encryption scheme showed that it has only 2-
3% performance degradation on query processing, compared to
AES encryption. However, our scheme can support the
encryption of variable-length data while providing the same
level of data protection with the AES algorithm. Future work is
to apply our ARIA-based encryption scheme to the real world
applications, such as location-based services and financial
information processing.
[6] Seonyoung Park and Youngseok Lee, “A Performance Analysis of
Encryption in HDFS,” Journal of KISS : Databases, Vol.41, Issue.1,
2014, pp.21-27
[7] Byeong-yoon Choi. “Design of Cryptographic Processor for AES
Rijndael Algorithm,” The Journal of The Korean Institute of
Communication Sciences, Vol.26, Issue.10, 2001, pp. 1491-1500
[8] Yong Kuk Cho, Jung Hwan Song, and Sung Woo Kang, “Criteria for
Evaluating Cryptographic Algorithms based on Statistical Testing of
Randomness,” Journal of the Korea Institute of Information Security and
Cryptology, Vol.11, Issue.6, 2001, pp.67-76.
[9] ARIA Development Team, Block Encryption Algorithm ARIA
[Internet], http://glukjeoluk.tistory.com/attachment/ok110000000002.pdf.
[10] Korea Internet & Security Agency, ARIA specification [Internet],
http://seed.kisa.or.kr/iwt/ko/bbs/EgovReferenceDetail.do?bbsId=BBSM
STR_000000000002&nttId=39&pageIndex=1&searchCnd=&searchWrd
=.
[11] Jeffrey Root, Intel Ⓡ Advanced Encryption Standard Instructions(AES-
NI) [Internet] https://software.intel.com/en-us/articles/intel-advanced-
encryption-standard-instructions-aes-ni.
[12] Weizhong Zhao, Huifang Ma, Qing He, “Parallel k-means clustering
based on mapreduce,” In: IEEE International Conference on Cloud
Computing. Springer Berlin Heidelberg, Vol.5931 p. 674-679, 2009.
[13] Hui Gao, Jun Jiang, Li She, Yan Fu, “A New Agglomerative
Hierarchical Clustering Algorithm Implementation based on the Map
Reduce Framework,” International Journal of Digital Content
Technology and its Applications, Vol.4 Issue.3, 2010, pp.95-100
[14] [Internet] https://dumps.wikimedia.org/enwiki/
[15] [Internet] https://dumps.wikimedia.org/metawiki/
[16] MODIS [Internet] https://modis.gsfc.nasa.gov/

ACKNOWLEDGEMENT
This work was partly supported by the Human Resource
Training Program for Regional Innovation and Creativity
through the Ministry of Education and National Research
Foundation of Korea (NRF-2016H1C1A1065816). This work
was also supported by Institute for Information &
communications Technology Promotion(IITP) grant funded by
the Korea government(MSIP) (No. R0113-16-0005,
Development of a Unified Data Engineering Technology for
Large-scale Transaction Processing and Real-time Complex
Analytics).

REFERENCES
[1] Jeffrey Dean, Sanjay Ghemawat, “MapReduce: Simplified data
processing on large clusters,” Communications of the ACM, Vol.51,
Issue.1, 2008, pp.107-113.
[2] Hadoop [Internet], http://hadoop.apache.org.
[3] Sudheesh Narayanan, “Securing Hadoop : Implement robust end-to-end
security for your Hadoop ecosystem,” 1st Vol, PACKT Publishing, 2014
[4] So Hyeon Park and Ik Rae Jeong, “A Study on Security Improvement in
Hadoop Distributed File System Based on Kerberos,” Journal of the
Korea Institute of Information Security and Cryptology, Vol.23, Issue.5,
2013, pp.803-813
[5] Liu Yi, Hadoop Crypto Design [Internet],
https://issues.apache.org/jira/secure/attachment/12571116/Hadoop
Crypto Desi gn.pdf.

90