Deduplication of Data in the Cloud-based
Server for Chat/File Transfer Application
Ankit Rai , Arpit Diwan , Harsh Tripathi
ABSTRACT
With the enormous use and collection of
data, cloud storage is gaining the
popularity among the computer users. But
most of the data in the cloud storage is
redundant. Cloud storage providers are
using deduplication as the mechanism to
keep only one copy of the file, by which
cloud storage providers are minimizing the
storage and management overheads for
data. However, deduplication is raising so
many security concerns and challenges.
This paper addresses the techniques to
provide secure deduplication of data, by
taking the popularity of data items into the
count, and assuming that data items
require different levels of security based
on popularity. We are proposing a
mechanism to ensure secure data
deduplication leveraging the advantages of
dynamic perfect hash techniques
Keywords
Cloud Storage, Convergent key
Encryption, Dynamic Hashing, data
deduplication
INTRODUCTION
With the increasing demand of data
rates throughout the world, cloud storage
systems are becoming a necessity for
computer users. All the users are
uploading their data to the cloud storage
and are accessing the remotely stored
data upon need [18]. But 70% of the
data stored in cloud storage is redundant
[10], this is not permitting us to utilize the
storage space efficiently. Data
deduplication is a process of eliminating
redundant items from the cloud storage.
Data Deduplication can take place in
several ways: client side deduplication
i.e. Prior to uploading data to the cloud
storage, the user can verify the
redundancy of the data. In Server side
deduplication, after uploading data to the
cloud storage server, the server will
check for redundant data items, removes
them. Data Deduplication can happen at
several granularities, i.e. file level, and
block level. Storing the data in remote
storage locations i.e. cloud storage
raises several security challenges and
concerns [1][2]. The general solution to
ensure confidentiality of the data is to
encrypt the data prior to uploading to the cloud
storage. But uploading an encrypted copy of the
data to the cloud storage does not support
eliminating redundant data in the cloud, as the
cloud storage provider is not able to identify
whether two different cipher text files correspond
to the same file or not [9]. The solution by which
this problem can be solved is convergent
Encryption. In which the encryption key is obtained
by applying the file to the hash function. The
resulting hash code will be used as the key for
encrypting the file. As the properties of hash
functions ensure that, no two files will produce the
same hash code, if two users encrypt the files with
the resulting convergent key, it will produce the
same cipher text. This will help the cloud service
provider to eliminate redundant copies of the files.
Convergent encryption is introduced to make data
deduplication and data confidentiality compatible.
Convergent key encryption is vulnerable to well-
known problems like brute force attack and
dictionary attacks [17][11].
This paper proposes a mechanism to provide data
deduplication considering, how popular the data
item is. In general, a data item owned by several
users, like a popular audio track requires less
security. A data item which does not belong to
many users like a research proposal requires
better security. Deduplication will be applied only
to the popular file. If the file is not popular, it will
not be deduplicated, will be encrypted using the
conventional symmetric key encryption. The
popularity of a file will be considered a factor for
data deduplication.
2. Data deduplication techniques: various data
deduplication techniques have been developed by
several researchers, helping the users and cloud
storage service providers, in utilizing the storage
space effectively. Most of the works did not
consider security as a concern. But data
deduplication without security is of no use.
Recently Harnik et al [1] identified several attacks
that lead to the leakage of data in client side
deduplication systems. To overcome these
attacks, proof of ownership concept is introduced
[3] [10][5]. But this mechanism will not provide
confidentiality to the data stored when the cloud
service provider is honest-but-curious.
As convergent Key encryption is not able to
provide semantic security, is susceptible to various
means of guessing the content. Bellare [8][9] et al
developed a mechanism under the name message
locked encryption and showed that it provides
confidentiality for messages which are
not predictable only. Later Bellare et al.
Presented DupLESS [4], a server-aided
encryption scheme for ensuring
deduplication in cloud storage. But this
scheme is offering server-side
deduplication. But over the year's client
side deduplication proves to produce
better results when compared to the
server-side deduplication. Convergent
encryption is suffering from various
attacks like poison attack, Sybil attack
etc [17].
Armknecht et al [7] presented a
transparent encryption scheme for
deduplication storage with built in Proof
of ownership. It attests the users in
utilizing the cloud storage space
efficiently. The user will pay only if
his/her contents were stored in cloud
storage.
Jin li et al [6] developed an efficient
mechanism to manage the convergent
keys in deduplication storage. Instead of
every user generating the convergent
keys, this mechanism will reduce the key
management overhead a lot in client-
side block level deduplication. When
every user has to generate the
convergent key and store, the key
management will become an overhead
task for the users.
Pierre meye et al [20] presented a two-
phase data deduplication technique
addressing the weakness of convergent
key encryption and ensuring proof of
ownership, making inter-user
deduplication transparent.
-Memory based CF:
Memory-based CF algorithms use the entire or a
sample of the user-item database to generate a
prediction. Grouping of users is done on the basis of
similar interests. By identifying the so-called
neighbours of a new user (or active user), a
prediction of preferences on new items for user can
be produced. In neighbourhood-based CF algorithm,
a prevalent memory-based CF algorithm, following
steps take place: calculate the similarity or weight,
w[i,j], which reflects distance, correlation, or weight,
between two users or two items, i and j; produce a
prediction for the active user by taking the weighted
average of all the ratings of the user or item on a
certain item or user, or using a simple weighted
average[9].
-Similarity Computation:
Similarity computation between items or users is a
critical step in memory-based collaborative filtering
algorithms. For item- based CF algorithms, in similarity
computation between item i and item j is first to work
on the users who have rated both of these items and
then to apply a similarity computation to determine
the similarity, w[i,j], between the two co-rated items
of the users [9].
-Correlation-Based Similarity:
In this case, similarity w[u,v] between two users u
and v, or w[i,j] between two items i and j, is
measured by computing the Pearson correlation or
other correlation-based similarities. Pearson
correlation measures the

LITERATURE SURVEY
In the recent times, recommender system
has been extensively studied and improved.
There are numerous techniques and
methods to implement the system. In this
section we will review some of the existing
work related to the proposed method. We
will limit our scope to reviewing
advancement made in collaborative
filtering techniques.
Collaborative Recommendation systems give
recommendation to a user if similar users
liked an item. Examples of this technique
include nearest neighbour modelling [1],
Matrix Completion [2], Restricted
Boltzmann machine [3], Bayesian matrix
factorization [4] etc.
Types of collaborative filtering based on
the implementation are:
extent to which two variables
linearly relate with each other
[10].
Other correlation-based
similarities include: Constrained
Pearson correlation, a variation
of Pearson correlation that
uses midpoint instead of
mean rate; Spearman rank
correlation, similar to Pearson
correlation, except that the
ratings are ranks; and
Kendall’s τ correlation,
similar to the Spearman rank
correlation, but instead of
using ranks themselves, only
the relative ranks are used to
calculate the correlation
[11,12].
-Model based CF:
Unlike memory-based CF,
model-based approach does
not use the whole data set to
compute a prediction. Instead, it
builds a model of the data
based on a training set and
uses that model to predict
future ratings. For example,
clustering based CF method
builds a model of the data set as
clusters of users, and then uses
the ratings of users within the
cluster to predict. A very
successful model- based
method is the Singular Value
methods, and has better performance
than Bayesian-clustering and vector
cosine memory-based CF algorithms.
Baseline Bayesian model uses a Bayesian
belief net with no arcs (baseline model) for
collaborative filtering and recommends
items on their overall popularity. [22]
-Hybrid Recommender:
Most of the hybrid recommenders are built
by combining a CF algorithm with a
content based algorithm [13]. However,
with two CF algorithms, hybrid algorithm
can be implemented. One way to do it, as
[14] suggests (and the approach this
project has chosen), is to weight every
prediction from the two algorithms as the
final prediction according to
rˆui = αrˆui1 + βrˆui2 where rˆui1 is the
predicted rating from the memory based
algorithm, rˆui2 is the predicted rating
from the model based algorithm and rˆui
is the hybrid prediction. α is the weight
of the
memory prediction and β is the weight of
Decomposition (SVD) [19], which represents
the data by a set of vectors, one for each
item and user, such that the dot product of
the user vector and the movie vector is
the best approximation for the training set.
The model building process is
computationally expensive and memory
intensive. After model construction,
predictions are done very fast with small
memory requirement. It achieve less
accurate prediction than memory-based
methods on dense data sets where a
large fraction of user-item values are
available in the training set, but perform
better on sparse data sets.
-Simple Bayesian CF Algorithm:
The simple Bayesian CF algorithm uses a
naive Bayes (NB) strategy to make
predictions for CF tasks. Assuming the
features are independent given in the class,
the probability of a certain class given all
of the features can be computed, and then
the class with the highest probability will
be classified as the predicted class [20].
-Other Bayesian CF Algorithms:
Bayesian belief nets with decision trees at
each node. This model has a decision tree
at each node of the BNs, where a node
corresponds to each item in the domain
and the states of each node correspond to
the possible ratings for each item [21].
Results show that this model has similar
prediction performance to Pearson
correlation-based CF
the model prediction. Furthermore, requirement is α +
β = 1 where 0 ≤ (α, β) ≤ 1. The values of α and β are
determined numerically. This simple approach
intuitively results in a program running time of the
both algorithms combined. Hybrid Recommenders
Incorporating CF and Content Based Features. The
content-boosted CF algorithm uses naive Bayes as
the content classifier, it then fills in the missing
values of the rating matrix with the predictions of the
content predictor to form a pseudo rating matrix, in
which observed ratings are kept untouched and
missing ratings are replaced by the predictions of a
content predictor. Then predictions over the resulting
pseudo ratings matrix using a weighted Pearson
correlation- based CF algorithm, which gives a
higher weight for the item that more users rated,
and gives a higher weight for the active user. [18]
Other recommender systems include demographic-
based recommender systems, which use user profile
information such as gender, postcode, occupation,
and so forth [15]; utility-based recommender systems
and knowledge-based recommender systems, both
of which require knowledge about how a particular
object satisfies the user needs [16, 17].
We will not discuss these systems in
detail in this work.
Further collaborative filtering can be divided
into to two subcategories: User based and
Item Based
In user based, the focusing on the
“nearest neighbour” approach for
recommendations, which looks at the
rating patterns of other users and finds the
“nearest neighbours”, i.e. users having
ratings closer to yours. The algorithm then
gives you recommendations based on the
ratings of these neighbours [5]. Amazon
developed Item based collaborative filtering
in 1998[6]. Unlike user based
collaborative filtering, item-based filtering
looks at the similarity between items, and
perform this task by noting down how
many users that bought item A also
bought item B. If the correlation is high
enough, a similarity can be considered to
exist between the two items, and they can
be seen as similar to one another. Item B
will from there on be recommended to
users who bought item A and vice versa
[6]. Similarity can be computed in a number
of ways Using the user ratings, using
some product description, Using co-
occurrence of items in a bag or in the set
of a user past purchased products.
Item-based methodology will be used in
the proposed model thus we will look
more into the work performed using
them.
Bayesian networks create a model based
on a training set with a decision tree at
each node and edges representing user
information. The model can be built off-line
over a matter of hours or days. The
resulting model is very small, very fast,
and essentially as accurate as nearest
neighbour methods [4].
Clustering techniques work by identifying
groups of users who appear to have
similar preferences. Once the clusters are
created, predictions for an individual can
be made by averaging the opinions of the
other users in that cluster. Some
clustering techniques represent each user
with partial participation in several
clusters. The prediction is then an average
across the clusters, weighted by degree of
participation. Clustering techniques usually
produce less-personal recommendations
than other methods, and in some cases, the
clusters have worse accuracy than nearest
neighbour algorithm [7].
Horting is a graph-based technique in
which nodes are users, and edges
between nodes indicate degree of
similarity between two users. Predictions
are produced by walking the combining the opinions of the nearby
graph to nearby nodes and users. Horting differs
from nearest neighbour as the graph may Nevertheless, the more users there are in the
be walked through other users who have system, the
not rated the item in question, thus
exploring transitive relationships that
nearest neighbour algorithms do not
consider. In one study using synthetic data,
Horting produced better predictions than a
nearest neighbour algorithm [8].
The item-based approach performs poorly
for datasets with browsing or
entertainment related items such as
MovieLens, where the recommendations
it gives out seem very obvious to the target
users. Such datasets see better results
with matrix factorization techniques. Matrix
factorization can be seen as breaking
down a large matrix into a product of
smaller ones. This is similar to the
factorization of integers, where 12 can be
written as 6 x 2 or 4 x 3. In the case of
matrices, a matrix A with dimensions m x
n can be reduced to a product of two
matrices X and Y with dimensions m x p
and p x n respectively.
There are number of ways to implement
matrix factorization which improves the
performance of item-based
recommender system, one of the
popular algorithms to factorize a matrix
is the singular value decomposition (SVD)
algorithm. SVD came into the limelight
when matrix factorization was seen
performing well in the Netflix prize
competition. Other algorithms
include PCA and its variations, NMF, and so
on. Auto encoders can also be used for
dimensionality reduction in case you want to
use Neural Networks.
We aim to use the output of matrix
factorization as the input to the deep
neural network for recommendations.

PROBLEM FORMULATION
User based CF: User-Based Collaborative
Filtering is a method used for predicting
the items to a user on the basis of ratings
given by the other users to that item who
have similar taste with that of the target
user.
Many websites use collaborative filtering for
building their recommendation system.
Drawbacks:
Sparsity: The percentage of user rating
items is very low.
Scalability: The more K neighbors we
consider (under a certain threshold), the
better my classification should be.
greater the cost of finding
the nearest K neighbors
will be.
Cold-start: New users don’t
have much information about
their taste to be compared with
other users.
New item: New items when
added will lack ratings to
create a solid ranking.
Item based CF: the similarities
between different items in the
dataset are calculated by using
one of a number of similarity
measures, and then these
similarity values are used to
predict ratings for user-item
pairs not present in the dataset.
Some of the problem faced in
this method arises when there
was only one common user
between movies. These
problems arise due to the
sparsity of the dataset itself.
SVD: Singular value
decomposition (SVD) is a
collaborative filtering method for
movie recommendation. (SVD)
generates correlation and
features from the user-item
matrix. For example, if items are
1,000,000 ratings for 3952 movies by
6040 users. Each rating is in integer form
between 1-5 (worst-best). To measure the
performance of the online prediction
method, we have shuffled the rating
matrix. We are using about 80% of the
data as training and validation set, and the
remainder as testing set. In evaluating, we
are using root mean square error (RMSE),
which emphasizes more on prediction with
larger error.
For MovieLens 1M dataset, we are using
network with a (6040+3952) rating vector
as the input, and 6 fully-connected hidden
layers with activations each. Fig. (a) shows
the RMSE value for test data versus the
number of training epochs. We further
examine our algorithm with the existing
one using the MovieLens 1M datasets.
movies then SVD would look into by
generating factors like action vs comedy,
Hollywood vs Bollywood, or Marvel vs
Disney. Mainly, we will focus on the latent
factor model for the Singular Value
Decomposition (SVD) approach. The main
drawback of SVD is that there is not much
explanation why an item is recommended to
a user. This turn out to be a huge problem
when wants to know why a particular item
is recommended to them.
The advantages of using deep neural
networks to assist representation learning
are in two-folds: (1) it reduces the efforts
on hand-craft feature design , (2) it aids
recommendation model system to take in
wide variety of content such as text,
images, audio, and even video
EXPERIMENT
Dataset Description:
MovieLens is the recommender system
that recommends movies on the basis of
preference of the user using collaborative
filtering approach. MovieLens was created
in 1997 by GroupLens Research, a research lab
in the Department of Computer Science
and Engineering at the University of
Minnesota, in order to gather research data
on personalized recommendations.
We evaluated the proposed neural
network model on MovieLens dataset 1M.
As shown in Table 1, the MovieLens 1M
dataset contains
Table 3 shows the RMSE values of some basic
models and
neural-network models. We referred to the results of
experiment in [26] for other conventional CF
algorithms.
Our proposed neural-network model achieves much
better performance than traditional methods such as
the user-based CF, the item- based CF, and the SVD
method. The results of our method are superior to the
auto encoder- based model and the restricted
Boltzmann machine, which are based on deep
learning.
Architecture:
Movielens Dataset -> user based embedded layer (u)
-> item based embedded layer(m) -> dot product of m
and n -> 6 fully connected layer dense neural
layers -> output layer
DETAILS
We’re using embedding to
represent each user and each
movie in the data. These
embedding are vectors (of size
n=50 factors) that start as
random numbers but are the
best fit by the model to capture
the essential qualities of each
user/movie. We will accomplish
this by computing the dot
product between a user vector
and a movie vector to get
predicted rating. Achieved dot
product is feed into the deep
neural network.
Deep neural network consists
of six Fully connected dense
REFERENCE
[1] Robert M Bell and Yehuda Koren.
Improved neighbourhood-based collaborative
filtering. In KDD’13 CUP, 2007.
[2] Jasson DM Rennie and Nathan Srebro.
Fast maximum margin matrix factorization
for
collaborative prediction. In ICML’05, pages
713– 719.
[3] Ruslan Salakhutdinov, Andriy Mnih, and
Geoffrey Hinton. Restricted boltzmann
machines for collaborative filtering. In
layers and dropout layers with the
probability of 0.4 is used for the purpose
of regularization after each neural layer.
Adam’s optimiser is used for optimizing the
algorithm with learning rate of 0.001.
CONCLUSION
We conclude that the proposed method
has out-performed the traditional
methods. With more research on fining
tuning the hyper- parameters and building
the deeper network architecture we can
achieve even better results for validation
error.
ICML’07, pages 791–798.
[4] Ruslan Salakhutdinov and Andriy Mnih. Bayesian
probabilistic matrix factorization using markov chain
monte carlo. In ICML’08, pages 880–887.
[5] R. Devooght, H. Bersini (2017) Collaborative Filtering
with Recurrent Neural Networks (Accessed 15-02-
2017) [6]https://www.google.com/patents/US626664 9
(patent of item based collaborative filtering)
[7] Breese, J. S., Heckerman, D., and Kadie, C. (1998).
Empirical Analysis of Predictive Algorithms for
Collaborative Filtering. In Proceedings of the 14th
Conference on Uncertainty in Artificial Intelligence, pp.
43-52.
[8] Aggarwal, C. C., Wolf, J. L., Wu K., and Yu,
P.
S. (1999). Horting Hatches an Egg: A New
Graph- theoretic Approach to Collaborative
Filtering. In Proceedings of the ACM KDD’99
Conference. San Diego, CA. pp. 201-212.
[9] B. M. Sarwar, G. Karypis, J. A. Konstan,
and J.
Riedl, “Itembased collaborative filtering
recommendation algorithms,” in Proceedings
of the 10th International Conference on
World
Wide Web (WWW ’01), pp. 285–295, May
2001
[10] P. Resnick, N. Iacovou, M. Suchak, P.
Bergstrom, and J. Riedl, “Grouplens: an open
architecture for collaborative filtering of
netnews,” in Proceedings of the ACM
Conference on Computer Supported
Cooperative Work, pp. 175–186, New York,
NY, USA, 1994.
[11] K. Goldberg, T. Roeder, D. Gupta, and
C.
Perkins, “Eigentaste: a constant time
collaborative filtering algorithm,” Information
Retrieval, vol. 4, no. 2, pp. 133–151, 2001.
[12] J. L. Herlocker, J. A. Konstan, L. G.
Terveen, and J. T. Riedl, “Evaluating
collaborative filtering recommender systems,”
ACM Transactions on Information Systems,
vol. 22, no. 1, pp. 5–53, 2004.
[13] R. F. A. Elkhleifi and F. B. Kharrat,
“Improving collaborative filtering algorithms,” in
2016 12th International Conference on
Semantics, Knowledge and Grids (SKG), Aug
2016, pp. 109–114.
[14] G. Badaro, H. Hajj, W. El-Hajj, and L.
Nachman, “A hybrid approach with
collaborative filtering for recommender
systems,” in 2013 9th International Wireless
Communications and Mobile Computing
Conference (IWCMC), July 2013, pp. 349–
354
*15+ B. Krulwich, “Lifestyle finder: intelligent
user profiling using large-scale demographic
data,” Artificial Intelligence Magazine, vol.
18, no. 2, pp. 37–45, 1997.
*16+ R. Burke, “Hybrid recommender systems:
survey and experiments,” User Modelling
and User-Adapted Interaction, vol. 12, no.
4, pp. 331–370, 2002
[17] R. H. Guttman, Merchant
differentiation through integrative
negotiation in agent- mediated electronic
commerce, M.S. thesis, School of
Architecture and Planning, MIT, 1998
[18] P. Melville, R. J. Mooney, and R.
Nagarajan, “Contentboosted collaborative
filtering for
improved recommendations,” in Proceedings
of the 18th National Conference on Artificial
Intelligence (AAAI ’02), pp. 187–192, Edmonton,
Canada, 2002.
[19] G. Linden, B. Smith, and J. York,
“Amazon.com recommendations: item-to-
item
collaborative filtering,” IEEE Internet
Computing, vol. 7, no. 1, pp. 76–80, 2003.
[20] K. Miyahara and M. J. Pazzani,
“Improvement of collaborative filtering with
the simple Bayesian classifier,” Information
Processing Society of Japan, vol. 43, no.
11, 2002.
[21] J. Breese, D. Heckerman, and C. Kadie,
“Empirical analysis of predictive algorithms
for collaborative filtering,” in Proceedings
of the 14th Conference on Uncertainty in
Artificial
Intelligence (UAI ’98), 1998.
[22] D. Heckerman, D. M. Chickering, C. Meek,
R. Rounthwaite, and C. Kadie, “Dependency
networks for inference, collaborative filtering,
and data visualization,” Journal of Machine
Learning Research, vol. 1, no. 1, pp. 49–75,
2001.
[23] D. Goldberg, D. Nichols, B. M. Oki, and
D. Terry, “Using collaborative filtering to weave
an information tapestry,” Communications of
ACM, vol. 35, no. 12, pp. 61–70, 1992.
*24+ P. Resnick and H. R. Varian,
“Recommender systems,” Communications of
the ACM, vol. 40, no. 3, pp. 56–58, 1997.
[25] Scalable deep learning-based
recommendation Systems, Hyeungill Lee,
Jungwoo Lee, Seoul National University, Seoul,
Republic of Korea
[26]O. Yuanxin, R. Wewnge, X. Zhang,
Autoencoder colaborative filtering, pp. 284–29