NLP Cross-Domain Recognition of Retail Products
Tobias Petterson∗ Rachid Oucheikh∗
tobias.pettersson@itab.com Department of Computing, Jönköping
ITAB Shop Products AB, Sweden and AI Lab, Jönköping University
Department of Computing, Jönköping Jönköping, Sweden
AI Lab, Jönköping University, Sweden rachid.oucheikh@ju.se
and Department of Engineering,
University of Skövde
Sweden
ABSTRACT
Self-checkout systems aim to provide a seamless and high-quality
shopping experience and increase the profitability of stores. These
advantages come with some challenges such as shrinkage loss. To
overcome these challenges, automatic recognition of the purchased
products is a potential solution. In this context, one of the big
issues that emerge is the data shifting, which is caused by the
difference between the environment in which the recognition model
is trained and the environment in which the model is deployed. In
this paper, we use transfer learning to handle the shift caused by
the change of camera and lens or their position as well as critical
factors, mainly lighting, reflection, and occlusion. We motivate the
use of Natural Language Processing (NLP) techniques on textual
data extracted from images instead of using image recognition to
study the efficiency of transfer learning techniques. The results
show that cross-domain NLP retail recognition using the BERT
language model only results in a small reduction in performance
between the source and target domain. Furthermore, a small number
of additional training samples from the target domain improves
the model to perform comparable as a model trained on the source
domain.
CCS CONCEPTS
• Computing methodologies → Object detection; Object identi-
fication; • Theory of computation → Online learning algorithms;
• Networks → Network reliability.
KEYWORDS
Product recognition, text classification, retail, transfer learning,
domain adaptation, NLP, BERT
ACM Reference Format:
Tobias Petterson, Rachid Oucheikh, and Tuwe Löfström. 2022. NLP Cross-
Domain Recognition of Retail Products. In 2022 7th International Conference
on Machine Learning Technologies (ICMLT) (ICMLT 2022), March 11–13, 2022,
∗ Both authors contributed equally to this research.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
fee. Request permissions from permissions@acm.org.
ICMLT 2022, March 11–13, 2022, Rome, Italy
© 2022 Association for Computing Machinery.
ACM ISBN 978-1-4503-9574-8/22/03. . . $15.00
https://doi.org/10.1145/3529399.3529436
237
Tuwe Löfström
Department of Computing, Jönköping
AI Lab, Jönköping University
Jönköping, Sweden
tuwe.lofstrom@ju.se
Rome, Italy. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/
3529399.3529436
1 INTRODUCTION
Self-service technologies aim to increase the profitability of orga-
nizations by shifting work from attendants to customers, which
become an active participant in the delivery of a service which they
ultimately consume. Self-scan and Checkout (SCO) services are one
type of these technologies that are increasingly being integrated
into many supermarkets. SCO brings many benefits as they can
increase profits, reduce service time, optimize the number of staff
needed, and provide high-quality service to customers.
Despite these advantages, SCO adoption creates some technical
and financial challenges, such as minimizing stock loss. Further-
more, the stores are growing and offering a wider range of products,
and as the number of items ordered gradually increase, there is
a need to ensure the reliability of the checkout process. Early on
in using SCO systems, retailers affirmed that they could increase
their exposure to retail losses by creating new loss factors, both
malicious and non-malicious [3]. Retailers calculate their stock loss,
typically called ‘shrinkage’, through measuring the difference be-
tween the amount of stock they think they should have and what
they actually have, usually ascertained through regular physical
audits/stock counts. The difficulty lies in trying to understand the
cause of any missing stock, particularly when stock audits are car-
ried out infrequently (often annually). The time lag between a loss
event happening and it eventually being recorded can be consider-
able, making identification of the cause very challenging. According
to Efficient Consumer Response (ECR) self checkout report [2], re-
tail stores with 50% of transactions being processed through self
checkouts can expect their shrinkage losses to be 75% higher than
the average rate found in grocery retailing.
To reduce the shrinkage, product recognition technologies could
be used to detect fraudulent behavior. Product recognition has
mainly used image-based recognition models [6, 20]. However, a
problem in the SCO environment is the collection of image data.
Products are held and moved by customers, generating occlusion
and motion blur in the image data. In other controlled environ-
ments, such as a semi-automatic checkout1 , the products are regis-
tered through a barcode recognition system that provides image
data on products without occlusion and motion blur. A potential
workaround to the data shortage in the SCO environment, making
it easier to collect more high-quality training data, could be to apply
1 https://itab.com/products-and-services/products/checkouts-self-service-
systems/hybrid-semi-automated-checkout-solutions/scanmate
ICMLT 2022, March 11–13, 2022, Rome, Italy
data from the semi-automatic checkouts to the SCO environment.
This can be reflected as a data distribution shift between the semi-
automatic checkout, in which the system is trained, and the SCO
target domain, in which it is deployed.
In this study, we are interested in fixed SCO systems as described
by Beck [2]. A fixed SCO is when a customer performs the scanning
of products in a designated machine. In a fixed SCO, a camera can
be placed where barcode readers typically are mounted, either be-
low or front-facing the product that is scanned. The camera stream
provides images of the product when positioned at the counter.
Using the captured images and machine learning techniques, the
product can be classified into its correct category. Automatising
the recognition process has a number of benefits for both super-
markets and their customers, some of which can be related to the
increase of revenue, identification of scanning errors, and detection
of malicious behavior in the checkout.
We have opted to use text extracted from product images. This is
well motivated by our previous paper [11] in which we showed that
the use of Optical Character Recognition (OCR) and NLP techniques
are promising for the recognition and verification of retail products.
This success is due to the following reasons: Text extraction using
OCR is fairly robust to changes in scale, rotation, and illumination.
Furthermore, text recognition accuracy is high even on images
containing few textual elements.
Different camera placement and resolution, lighting conditions,
and color information creates a significant domain shift between
images in the source and target domain. The goal of this paper is
to exploit the source domain knowledge and apply it in the target
domain. The source domain consists of text extracted from RGB
images while the target domain extracts text from a monochrome
camera. Consequently, the objective of this paper is to assess the
efficiency of transfer learning from a source domain, using an RGB
camera, to a target domain, using a monochrome camera, by us-
ing OCR and deep learning approaches for NLP. Furthermore, we
evaluate two different kinds of models, a Convolutional Neural
Network (CNN) with GloVe embeddings and BERT, for the product
classification.
2 BACKGROUND
It is common that the deployment of learning models is performed
in environments which are different from the training environment.
This is reflected by the fact that the data collected and used for
the training is not perfectly matching the data in the deployment
environment. Usually, statistical learning theory is based on the
assumption that both training and test data are derived by the
same process (i.e., on the i.i.d. assumption), which is not practical
for real world data in which the model can be used in similar but
not identical environments. Naturally, the performance of trained
machine learning models may decrease when the distributions of
the training and test data differ.
Another challenge which may worsen the case is that the col-
lection and labeling of data sets for each deployment environment,
which are large enough to train an efficient model, is costly and
we often have insufficient data that may lead to overfitting or un-
desirable behavior. Transfer Learning (TL) is an important new
238
Pettersson et al.
area of research which alleviates these issues and ultimately tack-
les the data scarcity problem. The TL paradigm aims to leverage
the information gained in one or more learning tasks having a
larger dataset to improve performance in another related task with
a smaller dataset [10, 22]. Thus, TL helps bridge the distribution
gap between datasets retrieved from the source domain D S and the
target domain DT .
More formally (inspired by the description in [12]), a domain
D = {X, p(X )} can be defined as a space of all feature vectors X and
a marginal probability distribution p(X ) where X = {x 1 , . . . , x n } ∈
X is a particular learning sample with n feature vectors. For a
specific domain, the goal is to learn an objective predictive function
f that predicts a label yi from label space Y for the corresponding
sample x i . This is called a task and can be noted T = {Y, f (x)}.
Given source and target data, transfer learning is used when there
is a discrepancy in either the domains (DS , DT ) or tasks (TS ,
TT ). Source and target domains are said to be different if their
feature spaces are not the same (XS , XT ) or their marginal
probability distributions are unequal (p(X S ) , p(XT )). In such
cases, an effective way to learn the target task is to explicitly map
the data to a common or domain invariant representation. This
learning strategy is called Domain Adaptation (DA).
The prevalent assumption is that the source and target label
spaces are similar. DA methods are classified further into three
categories depending on the availability of labels in the target
domain [19]. Supervised DA methods assume that the target data
is labeled (but generally small). In the semi-supervised case, in
addition to a small amount of labeled target data, a larger amount
of unlabeled target data is available. In the unsupervised setting,
no labeled data is available for the target domain. In this paper, we
use supervised DA techniques and we experiment with varying a
small amount of unlabeled target data to evaluate the performance
of the retrained model.
2.1 Transfer Learning in NLP
The arrival of Transfer Learning and rapid improvements in the
performance of language models present a great progress in the
NLP domain. Global Vectors (GloVe) is one of the models allowing
to learn distributed word representation in multi-dimensional space
[13]. It is a global log-bilinear unsupervised regression model that
aims to obtain vector representations for words. This is achieved
by mapping words into a meaningful space where the distance
between semantically similar words is small. The model is trained
on word-word co-occurrence statistics from a large corpus. The
resulting representations capture meaningful linear substructures
of the word vector space and the obtained pre-trained word em-
beddings can be used for new tasks. However, these embeddings
do not capture the context of words. Recently developed language
models, such as BERT [5] and ULMFiT [7], can learn the context of
each word and catch the nuances and inter-dependencies among
the various words of the text. The introduction of such language
models has the advantage of allowing data scientists to ensure rea-
sonable accuracy using pretrained models without the need for
an excessive amount of manually labeled data and to achieve the
state-of-art performance for various NLP tasks. One of the possi-
bilities to use these pretrained language models is to build new
NLP Cross-Domain Recognition of Retail Products
neural network structures on top of them. Basically, there are three
ways to train these stacked neural networks: stack-and-finetune,
stack-only and finetune-only. In the stack-only method, the pre-
trained model works as a feature extractor and its final layer is
removed and replaced by one or several layers. The weights of
the pretrained model are frozen during training for the new task
and only the additional layer weights can be updated. Finetune-
only methods use only the pretrained model and train it further on
the new domain. Peters et al. [14] compared to the stack-only and
finetune-only methods and concluded that finetune-only is better
than the stack-only option. Finally, stack-and-finetune combines
both methods, fine-tuning the pretrained language model while
also training the stacked model.
An important feature of the recent language models which led
to their success is the attention mechanism. Instead of encoding a
single vector to represent the sequence, the attention mechanism
computes a context vector for all tokens in the input sequence and
for each token in the output [23]. The decoder computes a relevancy
score for all tokens on the input side. These scores are then nor-
malized by performing a softmax operation to obtain the attention
weights. These weights are then used to perform a weighted sum
of the encoder’s hidden states, thus obtaining the context vector c t .
Then, the attention vector is obtained by performing a hyperbolic
tangent operation on the concatenation of the context vector and
the target hidden state. This attention vector generally provides a
better representation of the sequence than traditional fixed-sized
vector methods by identifying the relevant input tokens while gen-
erating the output token. Using this mechanism, Bahdanu et al.
[1] were able to achieve state-of-the-art performance in machine
translation tasks.
The Transformer architecture, building upon the significant im-
provements achieved by the attention mechanism and consisting
of an encoder and a decoder, was proposed in Vaswani et al. [18].
The encoder includes a multi-head attention layer, residual con-
nections, normalization layer, and a generic feed-forward layer.
The decoder is identical to the encoder except that it contains a
”masked” multi-head attention layer. The Transformer has achieved
new state-of-the-art results in various tasks such as machine trans-
lation, entailment, and so on.
Based on transformers, Bidirectional Encoder Representations
from Transformers (BERT) was developed to learn contextual em-
beddings for words [5]. BERT has at its core a Transformer language
model with a variable number of encoder layers and self-attention
heads. It is the first unsupervised language model which can be pre-
trained from unlabeled data extracted from a large corpus. During
the training, some of the words in the input are randomly masked
out and a condition is used for each word bidirectionally, i.e., in
both left and right contexts to predict the masked words. Being
bidirectional, BERT can learn efficiently the contextual embeddings
and achieve the state of the art in various tasks such as named
entity recognition and question answering.
239
ICMLT 2022, March 11–13, 2022, Rome, Italy
2.2 Transfer learning for retail product
recognition
Transfer learning is exploited for retail product recognition using
image datasets. Tonioni and Di Stefano [17] have used an image-
to-image translation by means of a generative adversarial network
(GAN) to address the domain shift and generate resilient cross-
domain feature embedding of product images. In the case where
the source datasets includes subsets with different distributions,
Thota et al. [16] proposed a multi-source deep learning-based do-
main adaptation system which serves to identify and verify the
presence and legibility of date information from the use-by on food
packaging pictures. The method incorporate discrepancy losses,
i.e., maximum mean discrepancy (MMD) [9] and correlation align-
ment (CORAL) [15], to extract domain-invariant representations
for all domains. Furthermore, it aligns distribution of all pairs of
source and target domains in a common feature space, along with
the class boundaries. Zhang et al. [23] developed a dual pyramid
scale network to learn the multi-scale feature of the data using both
detection and counting views. Furthermore, an iterative knowl-
edge distillation training strategy is applied to leverage both image
and instance levels and thereby narrow the semantic gap between
source domain and target domain. To tackle the cross-domain fine-
grained product recognition, Wang et al. [21] develop a CNN based
model which consists of two specialized components. The first
component is an adversarial component which allows to handle
the domain shift by gradually minimizing the discrepancy between
different domains and ensuring alignment in both domain-level and
class-level. The second component is a self-attention module de-
signed for fine-grained recognition by capturing the discriminative
image regions.
3 THE PROPOSED APPROACH
Domain adaptation is a common challenge for image recognition
systems. The domain shift between a source and target domain can
be caused by the change of background, pose, or type of camera.
This may lead to a significant degradation in performance for the
new domain, caused by different data distributions between training
and tests [4]. To mitigate this, re-calibration of the model to the
target domain has to be performed by collecting data from the
target domain. However, this is time-consuming since it requires a
lot of manual work.
The setup of the training procedure of our proposed approach is
illustrated in figure 1. The images from the source domain consist of
three-channel RGB data, while the target domain captures images
from a monochrome camera from another viewpoint. The image
data from both domains are processed by an OCR module that
detects words in an image and outputs it in a textual data format.
The source textual data is used for training a product recognition
model from scratch with RGB textual data, while the target domain
uses textual data from both the source and target domain. Further
details about the training and testing of the product recognition
model are explained in section 4.2.
In this study, we have used two machine learning models, namely,
CNN with GloVe and BERT. The architecture of CNN consists of 3
one-dimensional convolutional layers separated by two max pool-
ing layers, followed by one global max pooling and two dense layers.
ICMLT 2022, March 11–13, 2022, Rome, Italy
Source
Textual Data
Training
RGB
Source Images
OCR
Training
RGB+LS
Target Textual Product
Data Recognition
Model
Target Images
Figure 1: Training procedure of the proposed approach
The first layer is an embedding layer which receives as input the
sequenced GloVe embeddings for the text data. This stack-only
method was used in our previous paper [11] and yielded the best
classification results compared to several other text classification
models. BERT is a state-of-the-art model for NLP tasks. We used
a pretrained BERT model2 for evaluation of the text recognition
performance. The model used default configurations with a clas-
sification head consisting of a linear layer. During training of the
model, both the classification and the underlying BERT model are
trained, hence this is a stack-and-finetune model.
4 EXPERIMENTAL SETUP
4.1 Dataset description
The dataset has been collected using automatic scanning tunnels
deployed at a retailer at the checkout counter in grocery stores.
Figure 2 shows a schematic of the acquisition process.
Scanning Tunnel
RGB
Camera
Belt Direction
Product1 Product2 Product3
Line Scan
Camera
Figure 2: Picture of the data acquisition setup. Products are
passing through the scanning tunnel, registered by a bar-
code recognition system, enabling RGB and Line scan cam-
era data to be collected.
Two cameras have been used to collect image data: An RGB
camera 3 with resolution 2592x1944 has been placed in the top
of the tunnel towards the checkout belt. The second camera is a
monochrome line scan camera4 with a horizontal resolution of 4096
pixels. The line scan camera is located below the tunnel scanner
and the products that are passing through are scanned through a
small gap between the two checkout belts. The camera placement
and lighting conditions of the line scan camera are similar to those
of a front-faced barcode scanner with a fixed SCO system. The class
labeling is performed by a separate barcode recognition system
integrated in the tunnel scanner.
2 https://huggingface.co/transformers/model_doc/bert.html
3 DFK 33GP006 GigE color industrial camera: https://www.theimagingsource.com/
products/industrial-cameras/gige-color/dfk33gp006/
4 Teledyne DALSA LA-GM-04K08A Line scan GigE camera https://www.1stvision.com/
cameras/models/Teledyne-DALSA/LA-GM-04K08A
Pettersson et al.
The dataset includes textual descriptions of 81 retail products
extracted from two different types of images, namely, RGB and line
scan (LS). Thus, we have two different domains representing the
Final
Result same products. Figure 3 shows example images from the dataset in
both domains. There are a few characteristic differences between
the images in the domains. Due to low lightning conditions, the
images of the line scan camera have to be monochrome. In addition,
the placement of lightning generates glare in products. The RGB
camera images consist of images with more poses because it is
not faced down as the line scan camera images. Furthermore, the
resolution of the textual details is worse compared to the line scan
camera images.
Textual descriptions have been extracted from both the RGB and
the LS domains using OCR via Google Vision API5 . Datasets for
the domains have been extracted and some statistics are presented
in table 1. The RGB domain consists of a training set, denoted
Train RG B and a test set denoted T est RG B . Line scan data include a
training set Train LS and a test set T est LS . All datasets have been
extracted to have a balanced distribution for all classes.
In SCOs, products are often occluded when the product is being
scanned. To simulate the removal of text snippets at the edges
of a product when occluded by the hand of a person, a test set
T est LS _CC is created. It divides the words for each sample in T est LS
into multiple clusters using k-means based on the center point of
the word in the image. A mean center point is calculated using
the position of all words in a sample. The cluster centers closest
to the mean center point are then included in the sample. Two
different occlusion factors (OF) of 0.25 and 0.5 has been extracted,
meaning that either 25% or 50% of the text snippet clusters have
been removed. This emulates the case where a customer is holding
a standard size package with either one or two hands. We evaluate
the cluster sizes of 4 and 8 with k-means to see if different cluster
size affects our emulation method.
The size of the text snippets has been measured to analyze the
characteristics of the RGB and the LS domains and are available
in table 1. It shows that the LS domain contains 27% more words
per sample and the average number of characters per word is 12%
higher. The difference between the domains is expected, because
smaller characters are distinguishable in the LS domain due to
higher resolution. Qualitative inspection of the dataset in both
domains shows that misspellings are frequently occurring in the
extracted words. Misreadings are also frequent, where non-text
parts of the image are interpreted as text. No method for correcting,
removing, or cleaning the textual data has been done in this work.
Table 1: Datasets extracted from RGB and LS domain.
Dataset Classes Samples Average # words Average #
per sample characters per
word
T r ain RG B 81 16200 22.2 16.4
T est RG B 81 4050 21.9 16.2
T r ain LS 81 4050 28.2 18.4
T est LS 81 39926 27.8 18.2
T est LS _CC 81 39926
OF0.25 : 21.9 OF0.25 : 18.6
OF0.5 : 14.6 OF0.5 : 18.9
5 https://cloud.google.com/vision
6 Some samples from 3 classes were not possible to collect due to corrupt image data
from the LS camera.
240
NLP Cross-Domain Recognition of Retail Products
(a)
Figure 3: Example images (a), and (b) from the dataset. Each sample shows an image captured by the RGB camera (left) and
the line scan camera (right). Note that the example images in (a) is different samples and is displayed to show the domain
difference.
To get more insight into the data distribution in the space, we
used t-distributed stochastic neighbor embedding (t-SNE) to visual-
ize both domains. Figure 4 shows the source domain, represented in
red color, and the target domain, drawn in blue, where each point
in the scatter plot represents a projection in 3D of a sentence em-
bedding representing one product. The shift between the domains
is visually clear and it is evident that the data distributions of the
domains are different with an overlap in some regions of the space.
Figure 4: Visualization of source (red) and target (blue) do-
mains using T-SNE
In addition to the visualization, the gap between both domains
is measured using the word moving distance (WMD) technique [8]
and is presented in table 2. WMD measures the similarity between
documents by computing the minimal traveling distance between
them using word2vec embeddings. The mean of the distances be-
tween T rain RG B and the target domain is 1.32 while it is just 1.11
between T rain RG B and T est RG B . The relative difference of these
distances, which is 16% reflects the gap existing between the two
domains and supports the visualization obtained by t-SNE.
Table 2: WMD distance between source and target domains
Domains Mean Variance
T r ain RG B –> T r ain LS ∪ T est LS 1.32 0.33
T r ain RG B –> T est RG B 1.11 0.31
241
ICMLT 2022, March 11–13, 2022, Rome, Italy
(b)
4.2 Experiments
Three experiments are conducted to evaluate cross-domain recog-
nition of retail products using NLP. They are described in the fol-
lowing points and a summarization of the training and test dataset
for the experiments can be seen in table 3.
Experiment 1. The goal of the first experiment is to evaluate
the performance of the model pretrained using RGB data on the
LS data without any retraining of the model. The objective is to
get insights on the gap between the two domains. Thus, we train
the CNN and BERT models on a training set including only textual
data from the RGB domain using Train RG B . Then we evaluate it
on T est RG B and T est LS and report the obtained accuracies.
Experiment 2. The goal of the second experiment is to explore
how additional training samples from the target domain improve
the model performance compared to experiment 1. Thus, we train
the model with Train RG B and an additional increasing number of
samples from Train LS , using random selection of 2, 5, 10, 20, 30
and 50 samples for each training run. Due to the random selection
of samples for Train LS , 5 iterations are performed for each training
run and the mean accuracy is reported.
Experiment 3. In a realistic context such as a fixed SCO system,
the products are often occluded when scanned. In this experiment,
we simulate how occlusion affects the performance of the model
by removing parts (either 25% or 50%) of the textual data. Since
clustering is used to find the clusters defining what to occlude in
T est LS _CC , 5 iterations are evaluated for each case and the mean
result is reported.
Table 3: Setup of datasets for the experiments
Experiment Training Dataset Test Dataset
1 T r ain RG B T est RG B
T est LS
T r ain RG B +
2 {2, 5, 10, 20, 30, 50} samples T est LS
from T r ain LS
T est LS _CC with occlusion
3 T r ain RG B factors of {0.25, 0.5} and
cluster sizes of {4, 8}
For the BERT model, we used a sequence length of 128, which
included most of the text snippets for the samples. On the few
ICMLT 2022, March 11–13, 2022, Rome, Italy
samples that contained longer sequences than 128, the sequence
was cut.
5 RESULTS AND DISCUSSION
The first experiment compares the performance of the two differ-
ent kinds of models that we have used, the CNN with GloVe and
BERT. The results are reported in table 4. Once the CNN model is
trained using T rain RG B , it is evaluated on the RGB domain using
the T est RG B set and it achieves the accuracy 81.3%. Without any
retraining of the model, the same pretrained model achieves 65.1%
in the LS domain. On the other hand, the BERT model achieves an
accuracy of 88.2% on the test set from the source domain and as
much as 80.3% in the target domain.
In summary, the results of this experiment show that when using
the model that has been pretrained on the source domain directly
on the target domain, the accuracy is reduced by approximately 16
percentage points for CNN and 8 percentage points for BERT. In
addition, the BERT model shows a much higher performance than
CNN.
Table 4: Accuracy results for Experiment 1 using BERT and
CNN-based model
Model T est RG B T est LS
CNN 81.3% 65.1%
BERT 88.2% 80.3%
Table 5 shows the results of the second experiment in which
we train the models with varying amounts of data from the target
domain. The goal is to determine the amount of data required for a
reasonable performance, which should preferably be similar to the
one obtained in the source domain. As explained in the previous
section, the test dataset is fixed and the training data from the target
domain used for fine-tuning is varying and includes 2, 5, 10, 20, 30
and 50 random text samples per class. Both models learn from a
relatively small amount of data from the new domain. The accuracy
of the CNN model increases by 4.4 percentage points by adding only
2 samples per product and achieves 80.2% by adding 30 samples for
each class, which is close to the accuracy obtained on the source
domain. Retraining the BERT model with only 2 data points for each
product yields an accuracy of 82.9% which represents an increase
of 2.6 percentage points. Adding 20 samples in the training data
results in an accuracy of 86.0% whereas 50 samples were enough
to achieve 88.1%, which is comparable to the accuracy obtained
in the source domain. The CNN model also achieves an accuracy
comparable to that obtained in the source domain when 50 samples
are added to the training data, although it remains below the BERT
accuracy.
Table 5: Accuracy obtained in experiment 2 using BERT and
CNN-based model
#Samples per
class from 2 5 10 20 30
T r ain LS
CNN 69.5% 74.2% 76.3% 79.0% 80.2%
BERT 82.9% 84.1% 85.3% 86.0% 86.8%
The results from experiment 3 are summarized in table 6, includ-
ing results for the BERT model. They show that the classification
Pettersson et al.
accuracy of 80.1% achieved in experiment 1 for T est LS drops around
5 percentage points when having an occlusion factor of 0.25 and
around 15 percentage points with an occlusion factor of 0.5 for
T est LS _CC . The classification difference is not very significant with
an OF of 0.25. This scenario is comparable to a hand covering a
medium sized package, showing that the BERT model is robust
to occlusion. The number of clusters was varied to investigate if
different cluster size affects our emulation of occlusion by a hand
covering a product. The results with cluster sizes of 4 and 8 show
almost the same accuracy for OF 0.25 and 0.5.
Table 6: Accuracy obtained in experiment 3 for the BERT
model testing the impact of occlusion using T est LS _CC
Dataset Clusters Occlusion factor Accuracy
4 0.25 76.2%
4 0.5 67.9%
T est LS _CC
8 0.25 77.3%
8 0.5 67.9%
Summarizing the results, the experiments show that NLP can
leverage the knowledge acquired in different environments. Specif-
ically, the BERT model can achieve good performance for retail
product recognition on a dataset with a large number of products,
and it is robust for domain change. Retraining the model on 20
samples per product from the target domain causes a drop of only
2 percentage points in the accuracy and 50 samples were enough
to obtain a performance comparable to that obtained in the source
domain.
6 CONCLUSIONS
A common aspect for many learning tasks including retail product
recognition is that the available labeled data used for the training
do not match the actual data in the deployment environment. This
paper aims to handle this domain shift by transferring the learning
acquired in the source domain to the deployment domain which
has different features such as sensors, occlusion, viewing angles,
and lighting conditions. The results show that cross-domain NLP
retail recognition preserves similar performance when using the
BERT language model. We also show that a small amount of extra
training data from the target domain increases the performance to
the same level as the source domain. In the future, the effects using
and combining different OCR engines could also be explored for
real-time applicability.
ACKNOWLEDGMENTS
This work was supported by the Swedish Knowledge Foundation
(DATAKIND 20190194), the company ITAB, and Smart Industry
Sweden (KKS-2020-0044).
REFERENCES
[1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2016. Neural Machine
Translation by Jointly Learning to Align and Translate. arXiv:1409.0473 [cs.CL]
50
[2] Adrian Beck. 2018. Self-checkout in retail: Measuring the loss. Technical Report.
Efficient Consumer Response Community. https://doi.org/10.13140/RG.2.2.14100.
80.7% 55686
88.1% [3] Adrian Beck and Matt Hopkins. 2017. Scan and rob! Convenience shopping,
crime opportunity and corporate social responsibility in a mobile world. Security
Journal 30, 4 (oct 2017), 1080–1096. https://doi.org/10.1057/sj.2016.6
[4] Gabriela Csurka. 2017. Domain Adaptation for Visual Applications: A Compre-
hensive Survey. ArXiv abs/1702.05374 (2017).
242
NLP Cross-Domain Recognition of Retail Products
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT:
Pre-training of Deep Bidirectional Transformers for Language Understanding.
arXiv:1810.04805 [cs.CL]
[6] Venugopal Gundimeda, Ratan S. Murali, Rajkumar Joseph, and N. T. Naresh Babu.
2019. An Automated Computer Vision System for Extraction of Retail Food
Product Metadata. In First International Conference on Artificial Intelligence and
Cognitive Computing, Raju Surampudi Bapi, Koppula Srinivas Rao, and Munaga
V. N. K. Prasad (Eds.). Springer Singapore, Singapore, 199–216. https://doi.org/
10.1007/978-981-13-1580-0_20
[7] Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-
tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers). Association
for Computational Linguistics. https://doi.org/10.18653/v1/p18-1031
[8] Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From Word
Embeddings To Document Distances. In Proceedings of the 32nd International Con-
ference on Machine Learning (Proceedings of Machine Learning Research, Vol. 37),
Francis Bach and David Blei (Eds.). PMLR, Lille, France, 957–966.
[9] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. 2015. Learning
Transferable Features with Deep Adaptation Networks. arXiv:1502.02791 [cs.LG]
[10] Shuteng Niu, Yongxin Liu, Jian Wang, and Houbing Song. 2020. A Decade Survey
of Transfer Learning (2010–2020). IEEE Transactions on Artificial Intelligence 1, 2
(oct 2020), 151–166. https://doi.org/10.1109/tai.2021.3054609
[11] Rachid Oucheikh, Tobias Pettersson, and Tuwe Löfström. 2022. Product verifica-
tion using OCR classification and Mondrian conformal prediction. Expert Systems
with Applications 188 (2022), 115942. https://doi.org/10.1016/j.eswa.2021.115942
[12] Sinno Jialin Pan and Qiang Yang. 2010. A Survey on Transfer Learning. IEEE
Transactions on Knowledge and Data Engineering 22, 10 (oct 2010), 1345–1359.
https://doi.org/10.1109/tkde.2009.191
[13] Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove:
Global Vectors for Word Representation. EMNLP 14, 1532–1543. https://doi.org/
10.3115/v1/D14-1162
[14] Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To Tune or Not to
Tune? Adapting Pretrained Representations to Diverse Tasks. In Proceedings of the
243
ICMLT 2022, March 11–13, 2022, Rome, Italy
4th Workshop on Representation Learning for NLP (RepL4NLP-2019). Association
for Computational Linguistics. https://doi.org/10.18653/v1/w19-4302
[15] Baochen Sun, Jiashi Feng, and Kate Saenko. 2016. Correlation Alignment for
Unsupervised Domain Adaptation. arXiv:1612.01939 [cs.CV]
[16] Mamatha Thota, Stefanos Kollias, Mark Swainson, and Georgios Leontidis. 2020.
Multi-source domain adaptation for quality control in retail food packaging.
Computers in Industry 123 (2020), 103293. https://doi.org/10.1016/j.compind.
2020.103293
[17] Alessio Tonioni and Luigi Di Stefano. 2019. Domain invariant hierarchical embed-
ding for grocery products recognition. Computer Vision and Image Understanding
182 (2019), 81–92. https://doi.org/10.1016/j.cviu.2019.03.005
[18] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in Neural Information Processing Systems. 5998–6008.
[19] Mei Wang and Weihong Deng. 2018. Deep visual domain adaptation: A survey.
Neurocomputing 312 (oct 2018), 135–153. https://doi.org/10.1016/j.neucom.2018.
05.083
[20] Wenyong Wang, Yongcheng Cui, Guangshun Li, Chuntao Jiang, and Song Deng.
2020. A self-attention-based destruction and construction learning fine-grained
image classification method for retail product recognition. Neural Computing
and Applications 32, 18 (jul 2020), 14613–14622. https://doi.org/10.1007/s00521-
020-05148-3
[21] Yimu Wang, Ren-Jie Song, Xiu-Shen Wei, and Lijun Zhang. 2020. An Adversarial
Domain Adaptation Network For Cross-Domain Fine-Grained Recognition. In
2020 IEEE Winter Conference on Applications of Computer Vision (WACV). 1217–
1225. https://doi.org/10.1109/WACV45572.2020.9093306
[22] Karl Weiss, Taghi M. Khoshgoftaar, and DingDing Wang. 2016. A survey of
transfer learning. Journal of Big Data 3, 1 (may 2016). https://doi.org/10.1186/
s40537-016-0043-6
[23] Libo Zhang, Dawei Du, Lufficc Li, Yanjun Wu, and Tiejian Luo. 2020. Itera-
tive Knowledge Distillation for Automatic Check-Out. IEEE Transactions on
Multimedia PP (11 2020). https://doi.org/10.1109/TMM.2020.3037502