International Conference on Smart Systems and Inventive Technology (ICSSIT 2018)
IEEE Xplore Part Number: CFP18P17-ART; ISBN:978-1-5386-5873-4
Block Level based Data Deduplication and Assured
Deletion in Cloud
Sneha C. Sathe
M.E student: Department of Information Technology
Ramrao Adik Institute of Technology
Nerul, Navi Mumbai
snehasathe47@gmail.com
Abstract—With the progression of web, an ever increasing
number of individuals have begun to outsource information on
cloud. However, sending information to an outsider creates
opportunity for attackers to easily retrieve data. Because of
simplicity, individuals regularly tend to store excess information
on the cloud. This leads to wastage and increase in the cost of
storage space.As nobody knows where the information is put
away on cloud; it is likewise vital to certainly erase information
from cloud. Hence along with security, data de-duplication and
assured deletion are the major concerns in cloud storage. In this
paper, we put-forth an application based on block level based file
deduplication scheme over fragmented blocks that addresses the
issue of duplication in the cloud storage platform. We also focus
on guaranteed erasure using user defined policies. Thus on
expiry of a policy, the keys are deleted therefore making the file
inaccessible to anyone including the proprietor.
Keywords—Block level deduplication Fragmentation, Policy
assured deletion
I. INTRODUCTION
Cloud computing is defined as the conveyance of
administrations by means of web through a cloud service
platform. Organizations and business firms are currently
moving to distributed storage keeping in mind the end goal to
store the organization reports, introductions, business-basic
information and reinforcement records. Scalability,
availability, elasticity, pay-per-use are a portion of the
advantages of distributed computing. Google Cloud, Amazon
Web Services, Microsoft Azure, IBM Bluemix are a few who
provide cloud computing services[9]. Because of its dispersed
and framework less nature, clients don't know where their
information is put away on the cloud. Hence, cloud storage
suffer from security, integrity, confidentiality issues. It is
therefore required to secure the data before uploading to the
cloud. As clients spare excess information on the cloud, the
lease of distributed storage increments. Consequently to limit
the cost factor, it is important to check the document before
transferring for duplication.
A. Data deduplication in cloud
Data deduplication is a strategy by which the document is first
checked for a copy duplicate on the cloud. It helps to eliminate
978-1-5386-5873-4/18/$31.00 ©2018 IEEE
Nilima M.Dongre
Assistant Professor: Department of Information Technology
Ramrao Adik Institute of Technology
Nerul, Navi Mumbai
nilimarj@gmail.com
redundant copies of data from being uploaded. Using
deduplication, it becomes easier to store a single copy of the
file and eliminate the multiple copies from being transferred
in the distributed storage framework.
In Single Block level File Deduplication, the complete file is
considered as a single block. A single fingerprint is made for
the complete file and compared with the already stored
fingerprints. This makes more storage room as less number of
files are being put away and calculation time is lessened.
Fixed sized block-level deduplication divides a file into equal
sized chunks and compares the fingerprints of each chunk with
already stored chunks. This technique helps to find the number
of chunks that are duplicate in a file. But because of its settled
size nature, it can't de-tect deduplication on a more extensive
scale due to content shifting problem. Consequently, content
level deduplication is used in which which real substance of
document is checked.
B. Data Deletion in Cloud
Data is replicated across several locations; whenever the
proprietor wants to delete the data, he should be assured of
data deletion over all areas in the cloud. The cloud specialist
co-ops and the clients have service level agreements (SLA) for
the same. Nonetheless, as client isn't sure whether his
information is erased or not, guaranteed erasure is a basic
worry in distributed storage frameworks [10]. Policy based file
assured deletion ensures assured way of deletion when the
files associated with access policies are revoked.
Our contributions are as follows:
 Develop a cloud based application that stores file in
fragmented and encrypted format on storage nodes.
The main focus is to store only one fragment on a
particular storage node so that even if a node is
compromised, no information leakage is observed.
Attribute based file access is provided so that only
those users whose attributes match can have access to
the file.
406
 International Conference on Smart Systems and Inventive Technology (ICSSIT 2018)
IEEE Xplore Part Number: CFP18P17-ART; ISBN:978-1-5386-5873-4
 Provide a block level deduplication scheme that works
on fixed blocks to check duplicate blocks on cloud
using SHA-512 algorithm.
 Provide a policy based assured deletion scheme to
assuredly delete data from cloud based application.
II. LITERATURE REVIEW
A. Fragmentation and Replication
Fragmentation deals with creating chunks of a file and then
storing it on cloud. Correspondingly replication makes in
excess of one duplicate of the document in the distributed
storage for simple recovery at a later stage. Authors in [1]
have proposed DROPS that arrangements with the security
issues in cloud. The data is divided and put in the storage
nodes. Each node is selected using centrality measure thus
stores only a single fragment. Consequently, regardless of
whether a node is endangered; no imperative information is
uncovered.
B. Deduplication Techniques
Deduplication on cloud can be performed at file- level,
block- level and content level. Authors in [2] has proposed a
framework that spotlights on file and block level deduplication
techniques. It gives an approach to discover copies by looking
at records utilizing MD5 Algorithm. It supports threshold
based deduplication check where a percentage value is set for
duplication check. If the duplication percentage is greater than
50%, then file is refrained from being transferred to the cloud.
Haonan Su, Dong Zheng, Yinghui Zhang in [3] have
proposed a scheme that realizes variable-size block-level
deduplication using Rabin fingerprinting. The client side first
partitions the file F into a finite set of blocks Bi (where i = 1, 2
...). The user computes each block fingerprint f(Bi), and
forwards the block fingerprints f(Bi) to the Cloud server for
duplicate checks. f(Bi) acquired is sent to the Third Party who
performs the randomization of the convergent keys. In the
event that copy square is discovered, at that point client is
informed for the same else document is transferred after
encryption.
Ankush R. Deshmukh , Prof. R. V. Mante and Dr. P. N.
Chatur [4] proposed a system that deals with document level
deduplication and destruction in cloud using time variant
encryption. Attribute based encryption(ABE) permits access
and decoding on a man's part/benefits inside an association, as
opposed to by a man's particular personality.
Authors in [8] have proposed a plagiarism system that uses
Rabin-Karp algorithm in search plagiarized content in
assignments submitted by students in college or university. It
fastens the processing of string comparison by matching given
pattern with different i/p document’s string/substring by using
hash esteems. It uses hash function for every string/substring
in text. If hash estimation of given pattern and substring
matches, it implies two strings are comparable.
978-1-5386-5873-4/18/$31.00 ©2018 IEEE
C. File Assured Deletion
Data deletion is an important issue in cloud. Even if the
user deletes the data, he is not assured of complete and secure
deletion of data. M. Vanitha and Dr. C.Kavitha[5] has
proposed a strategy to distinguish singular client's information
and their encryption enters keeping in mind the end goal to
give an answer for erase information from cloud supplier's
multi-tenant storage architecture using a policy file which is
kept up by the key administrator. Subsequently, when the
policy gets revoked, the data files become unrecoverable.
Authors in [6] have presented a FadeVersion, secure cloud
reinforcement framework. The framework does not store
excess information in different renditions of reinforcements. It
gives security to information, rendition control of record and
gives guaranteed erasure of documents. Diverse information
objects are made for a document and encoded with an
arrangement of information keys additionally scrambled with
set of control keys. A master key is used as it is easier to
maintain all the control keys. On the off chance that a strategy
for document is disavowed, at that point the separate control
key is erased. If the data object is linked only with the revoked
policy, at that point it will be guaranteed erased. However if
the data object is associated with both the revoked policy and
another active policy, at that point it can at present be gotten
through the active policy.
Ashfia Binte Habib, Tasnim Khanam, Rajesh Palit[7] have
proposed a methodology which is an alternative and simpler
way for ensured erasure of documents by means of
cryptographic algorithms and other tools without the help of a
key manager. Firstly, the user who wants to upload a file will
provide a secret phrase. A random key (key 2) will be
generated by which the document will be encoded. Secondly,
using the phrase another key (key 1) will be generated which
will further encrypt key 2. Finally, encrypted file and key 2
will be transferred to the cloud. Consequently erasure of key,
prompts guaranteed erasure of document.
III. PROPOSED SYSTEM
We have proposed a cloud based application through
which clients can store information on cloud. The application
performs duplication check before transferring the record on
the cloud. Block level based deduplication seems to be one of
the best and most useful technique used for deduplication
check in cloud. Block level based technique allows better
accuracy while finding redundant data in cloud. Unlike file
based technique; fixed size block level deduplication allows to
check duplicates for redundant blocks of data. AES encryption
is provided on the fragments before being uploaded to the
cloud. Attribute based encryption is used in order to allow
only authorized users having a particular set of policies to
access the file. Convergent key is used in order to share
ownership access of file. Also with respect to deletion, policy
based file assured deletion is being used that works best on
user defined policies.
407
 International Conference on Smart Systems and Inventive Technology (ICSSIT 2018)
IEEE Xplore Part Number: CFP18P17-ART; ISBN:978-1-5386-5873-4
Fig. 1. System Flow
A. System Description
 Registration: The user has to first register for
uploading and downloading the file. During registration
user has to provide a username and password for his
account which would be required during login. The
user only needs to enter his original password during
registration. During login, a session grid password
mechanism will be implemented that avoids shoulder
surface attack.
 Fragmentation of file: When the user tries to upload
the file on cloud, the application fragments the file into
fixed size blocks. The fragments are distributed across
the storage nodes such that no single node contains
more than one block of the file. As a result, even a
successful attack on a particular node won’t leak any
significant information.
 Duplication Check: Before uploading the file to the
cloud, the file is checked for block level deduplication.
The hash value of each block is computed using SHA-
512 and is stored in a temporary database. Each hash
value is compared with the hash values of the already
978-1-5386-5873-4/18/$31.00 ©2018 IEEE
existing file fragments in the cloud. The count for the
total number of duplicate blocks is obtained. A
threshold value is predefined, and if count of duplicate
blocks exceeds the given threshold, the entire file is
considered to be a duplicate and not allowed to be
uploaded on the cloud. Or else, the file is directly
allowed to be uploaded on the cloud and the uploader
becomes the owner of the file.
 File Upload After duplication check, the fragments
are encrypted using Symmetric encryption algorithm
i.e AES and then uploaded in encrypted format on the
cloud. The file is accessed by the owner and other users
based upon their specfic attributes. Hence a key is
generated on the concept of Attribute Based
Encryption. The user who wishes to download the file
would require the key for decryption of the file and
then only he can download the file. Controlled
replication is being used so that only a single replicated
copy of file is stored into the cloud to improve the
security and recovery aspects of the file. If a file is
found to be duplicate, then the user cannot directly
upload the file on cloud. The user needs to acquire
permissions from the actual owner of the file and is
only then permitted to upload the file. The user sends
request to the owner of the file by sharing ownership
access of the file using convergent key concept. The
owner downloads the file and only if he wishes to
allow the user to upload the file, a random dekey is
generated for each user. The dekey helps the owner to
identify which user’s duplicate file is being allowed to
upload on the cloud. Upon receiving permissions, the
user is allowed to upload the complete file.
 Download file with decryption If any user other than
the file owner wants to download the file from the
cloud, he needs the key for downloading the file. Only
users whose attributes matches can download the file.
As the file is saved in the cloud in encrypted format,
the file should be first brought into its decrypted format
to get the file in its original format.
 File Assured Deletion: If a policy is revoked, then the
application completely destroys the keys that are
related to the particular file after expiration of
particular time. If the owner implicitly issues request
for file deletion, the owner is first verified based on his
attributes and if authorized, the keys are deleted. As the
database does not contain the keys, we ensure that file
F, which is tied to policy, is assuredly deleted. The
policy revocation operations do not involve
interactions with the cloud. Also assured deletion is
performed if the user deletes the file.
IV. ADVANTAGES OF PROPOSED SYSTEM
 Distributed access control of data stored in cloud so
that only authorized users with valid attributes can
access them.
408
 International Conference on Smart Systems and Inventive Technology (ICSSIT 2018)
IEEE Xplore Part Number: CFP18P17-ART; ISBN:978-1-5386-5873-4
 The identity of the user is protected from the cloud
during authentication.
 Achieve protection from cloud client’s perspective, no
changes on the cloud provider side. Data is protected
from attacker as fragmented blocks are stored on
different storage nodes.
 Provides a deduplication and deletion mechanism
along with a fragmentation and single replication for
cloud storage platform.
 Work on today’s cloud as an overlay.
V. CONCLUSION
Huge measure of information is being transferred on
cloud. This enables simple entry to the clients who can get to
their information living at any area at any point of time.
However vast amount of duplicate data is being uploaded that
prompts a lessening in the storage room and increment in lease
of the distributed storage space. Secure deletion is also another
important aspect as as clients don't know whether their
information is erased forever from cloud or not. Alongside this
security of information being put away on cloud is a test. Our
proposed system therefore aims at creating an application that
will perform fixed size block level deduplication check on the
fragmented blocks and then encrypting the blocks before
uploading it to the cloud. Also, it performs policy based file
assured deletion in order to securely delete file from the cloud.
As a result, it tries to achieve accuracy, security and other
design goals for the system.
978-1-5386-5873-4/18/$31.00 ©2018 IEEE
References
[1] Mazhar Ali, Kashif Bilal, Samee U. Khan, Bharadwaj Veeravalli, Keqin
Li and Albert Y. Zomaya,”DROPS: Division and Replication of Data in
Cloud for Optimal Performance and Security”, 2015 IEEE
TRANSACTIONS ON CLOUD COMPUTING.
[2] Namrata P. Kawtikwar, Prof. M. R. Joshi, Nur’aini Abdul Rashid, ”Data
Deduplication in Cloud Environment using File-Level and Block-Level
Techniques”, 2017 Imperial Journal of Interdisciplinary Research (IJIR)
Vol-3, Issue-5, 2017 ISSN: 2454-1362.
[3] A.Mounika and G. Murali, An Enhanced Approach for Securing
Authorized Deduplication in Hybrid Clouds”, 2016 International
Conference on Communication and Electronics Systems (ICCES).
[4] Ankush R. Deshmukh , Prof. R. V. Mante and Dr. P. N. Chatur, ”Cloud
Based Deduplication and Self Data Destruction” 2017 International
Conference on Recent Trends in Electrical, Electronics and Computing
Technologies.
[5] M.Vanitha and Dr. C.Kavitha,”Secured Data Destruction in Cloud
Based Multi- Tenant Database Architecture”, 2014 Inter-national
Conference on Computer Communication and Informatics (ICCCI),
Coimbatore, India.
[6] Arthur Rahumed, Henry C. H. Chen, Yang Tang, Patrick P. C. Lee, and
John C. S. Lui, ”A Secure Cloud Backup System with Assured Deletion
and Version Control”, 2011 International Conference on Parallel
Processing Workshops.
[7] Ashfia Binte Habib, Tasnim Khanam, Rajesh Palit, ”Simplified File
Assured Deletion (SFADE) - A User Friendly Overlay Approach for
Data Security in Cloud Storage System”,2013 IEEE 978--4673-6217-
7/13.
[8] https://www.investopedia.com/terms/c/cloud-computing.
[9] https://www.ibm.com/cloud/learn/what-is-cloud-computing.
409