Proceedings of the 2012 IEEE
International Conference on Robotics and Biomimetics
December 11-14, 2012, Guangzhou, China
Mobile Robot Action Based on QR code Identification
Wenyu Li, *Feng Duan, Bo Chen, Jing Yuan, Jeffrey Too Chuan Tan, Binbin Xu, Member, IEEE
Abstract—Nowadays, with the developing of the computer
science, computer vision is widely used in many areas. It is still a
major problem about feature recognition in complex
background and uneven intensity of light environment. Some
researchers have already achieved in particular conditions.
With the robot stepping into family, it is necessary to conduct
in-depth study about feature recognition and access to
information in complex indoor environmental conditions. In this
paper, we present a fast identification approach for QR code in
the indoor environment with normal light intensity. In the
process of image feature recognition, it is very difficult for the
edge while recognizing the slope image. Therefore, we use vector
method to express the color characteristics in the image, and use
the square minimum method to determine the edge lines of the
pattern. In addition, linear scale transformation makes the
image pattern more suitable for standard grids, which can
correct the tiny sampling error. Besides, in the experiment of
emulation process, we add a colored periphery features to
positioning the image. It can be captured clearly due to the
special colored background and behave well in the distance of
0.5m to 2m. Using an 860-pixel distance camera, the QR code
pattern can be captured easily and identified effectively. A
feedback control planar mobile robot is used as a platform to
verify the identification method. After the robot heading to the
front of the QR code pattern about 0.5m, the data of the sample
image can be scanned and read accurately. The experiment
results show that the approach is feasible and effective.
I. INTRODUCTION
W ITH the developing of the modern science technology,
more and more robots will step into family in the future.
Robots can be widely used for family household, aging
people service, and so on. They need to identify, position,
move, and act, which give us a challenge.
The application of robots in family using became more and
Manuscript accepted on October 1, 2012. This work is supported by
National Science Foundation for Young Scientists of China (Grant. No.
61203339), the Doctoral Fund of Ministry of Education of China (Grant. No.
20110031120041), the Research Fellowship for International Young
Scientists (Grant. No. 61150110479), and the Natural Science Foundation of
Jiangsu Province of China (Grant. No. BK2011259).
W. L. is with the Department of Intelligence Science and Technology,
College of Information Technical Science, Nankai University, No.94, Weijin
Road, Tianjin, 300071, P.R.China (phone: +86-13752428672; e-mail:
0910692@mail.nankai.edu.cn).
F. D. is with the Department of Automation, College of Information
Technical Science, Nankai University; The Jiangnan Institute of Modern
Industry (corresponding author, e-mail: duanf@nankai.edu.cn).
B. C. and J. Y. are with the Department of Automation, College of
Information Technical Science, Nankai University, No.94, Weijin Road,
Tianjin, 300071, P.R.China ({chenbocantou, nkyuanjing}@gmail. com).
J. T. C. T. is with the National Institute of Informatics, 2-1-2 Hitosubashi,
Chiyoda-ku, 101-8430, Japan (corresponding author, i@jeffreytan.org).
B. X. is with the School of Electronic and Information Engineering South
China University of Technology, Guangzhou, 510641, P. R. China
(e-mail:xu.binbin@mail.scut.edu.cn).
‹,(((
more diversification, such as indoor cleaning, anti-theft,
especially in object grasping and goods carrying. The sensors
of the robot are full of diversity, vision, hearing, and smelling,
which can be used in objects recognition. Besides, with the
development of the wireless sensor networks, location
estimation became more and more easily.
For the most, computer vision area considers more about
the normal objects recognition and plays an increasingly
important role in objects problems solving. The more
widespread of robots in the family services, the more quickly
in the field of robot vision problems should be solved for the
identification and operation to objects. Jürgen [1] presented
an approach for detecting, tracking, and learning articulation
models for cabinet doors and drawers without using artificial
markers, which limited the using area of particular vision
robots. It also used multi cameras to recognize and analyze
the images that may cost a lot more. The accuracy of Jürgen’s
experiment needed to be considered in other complex
environment as well. Guodong [2] not only presented an
approach using the QR code to recognize round periphery
feature, pose estimation, and grasp the goods by 7-DOF robot,
but also proposed an accurate idea in household using. He
mainly talked about the visual servo system, but did not give a
clear result of the experiment. Many researchers used
binocular cameras to pose the estimation, which needed to
calculate more about the coordinate and transformation
matrix [3, 4]. Xudong [5] had summarized the importance of
the feature extraction for image recognition and computer
vision. Common method in features extraction recognition
and fast solving speed are very important.
The image background is one of the big problems for
features extraction. In different areas, the intensity of the
lighting is also different. Some researchers proposed some
approaches for removing the complex background. But in
most time, the method can only be used in special
environment [6, 7]. They can solve particular problems
effectively, but the practicality still needs to be expanded. For
the family environment, the objects fill the room here and
there, and colorful objects make the background complex.
Whereas, we are aimed to find the objects needed, we can just
recognize the feature that the other objects not containing.
Special features can be selected, like particular colors, shapes,
and patterns with large contrast.
The QR code contains a large amount of information for
taking a step forward to identify the pattern. Xiangju [8]
proposed a robust barcode reading method based on image
analysis of a hierarchical feature classification for reference.
Marks with information are needed more for computer vision
recognition in the future, and the barcode is a necessary
860
choice. 1D barcode is widely used in economics and the
technology has already been very mature [9], but it contains
less information and is difficult to encrypt. The QR code is
one kind of the 2D barcode, developed by a Japanese
company called Denso Wave. The first version of the barcode
only contains a 21×21 dot matrix. Since it came out, the code
has expanded its size to a 177×177 dot matrix, which can
even contain more than 10KB information. The 2D barcode
not only contains more information than 1D barcode, but also
can be encrypted easily. By using the mask technique, the
positioning feature can be distinguished with the other area,
as illustrated in Fig. 1. Zafar [10] proposed an approach for
real time multiple 2D barcode recognition with an accuracy of
98%, which made it possible for normal environment
application and mobile robot. Guoyu [11] showed an
application in robot indoor navigation, which proposed a
method for robot indoor navigation based on 2D barcode
landmark. Its scheme construction on the 2D barcode
principle was convenient to operate and easy to recognize
landmarks. Also the barcode had their position coordinates
and the ability to automatically correct errors.
In this paper, we propose an approach for feature extraction
using the QR code. The method is easily achieved by normal
single CCD camera. It can recognize declining pattern by
vector based expressing method as well. The slope pattern is
universal, and does not correspond to lattice arrangement
with the computer, resulting in a saw tooth bias and edge blur.
The vector expression algorithm method can solve the
problem perfectly. Moreover, the method also applies a little
longer distance pattern identification with certain accuracy.
The paper is organized as follows: Section 2 introduces the
main approach of the QR code identification; Section 3 talks
Fig. 1. The function and location of various regions of the QR barcode.
about the identification in the experiment of the mobile robot;
Section 4 briefly introduces the hand grasping process;
Conclusion and the future work are introduced in Section 5.
II. IDENTIFICATION METHOD
A. Image Pretreatment
The image pretreatment is very important for the image
recognition. We concentrate more on the image features. The
image should be dealt with due to the particular environment,
such as the special intensity of light. As for normal indoor
environment, we need to find the normal features of the
objects in order to separate it from background or other
objects. In this study, particular periphery features is selected
and designed for the indoor vision recognition, which uses a
861
pink edge as the boundary feature. After the identification of
the boundary features, the original QR code pattern image is
illustrated in Fig. 2 (a). All the vision processing is dealt by
(a) (b)
Fig. 2. (a) The original colored QR image pattern that got by separate
algorithm; (b) The binary image after pretreatment.
OpenCV. In this part, we approximately adapted common
method into a new approach.
1) Color Convert from RGB to Gray: Since the information
of the barcode is just expressed beyond color, the colored
image needs to be transformed into grayed first according to
the intensity of the indoor light environment.
2) Threshold Segmentation: Count the sum value of all the
Fig. 3. The histogram of the gray image for the QR code pattern.
pixels, and make the value to the average number as the
binary threshold. In this way, the gray image can be separated
due to the value of the whole image. The process of the binary
transformation is just to filter out the gray part except the
white and black. The two peaks of the image are just near the
side of the histogram, and the average value can be the
threshold value. The histogram is illustrated in Fig. 3.
B. Positioning Feature Identification
Although the pattern rotates, the positioning features of the
QR code do not change its texture. Its texture in color shade
still follows by 1, 1, 3, 1, 1, shown in Fig. 1. So this feature
can be used to positioning the location of the three positioning
features. In order to positioning it accurately, the following
steps are selected.
1) Scan and Record: Scan the pattern both in horizontal
and vertical direction, and record the coordinates of the
central points. They can meet the requirement.
2) Select Central Points: Take the first three most
concentrated distribution points as the three central points. In
this way, the location of the bar code has been recognized
approximately, and there is a probability of 90% to say, the
three points are located in the three central big dark areas.
3) Posture Determination: Identify the posture of the
pattern, in other words, the exact location posture of the three
positioning features needs to be determined. They are the
top-left, the bottom-left, as well as the top-right. Calculate
each of the angles consisted by the three central points, as
illustrated in Fig. 4, then find the biggest angle of the central
point, corresponding to the top-left central point, and record
the angle value of the biggest angle as α.
Fig. 4. The posture identification for the three central positioning
features.
Rotate the one of the edge like AB for α clockwise. If the
edge can be overlapped with the other edge of the biggest
edge, the point B can be determined as the top-right central
point, otherwise it will be the bottom-left one. The third
central point of the positioning feature can be recognized
either. Three different colors, red, blue, green are used to
show the last result, as illustrated in Fig. 5. The blue area
shows the top-left position area, the green shows the top-right
position area and the red shows the bottom-left position area.
Fig. 5. The result of the identification of the posture. Blue shows the
top-left, green shows the top-right and red shows the bottom-left
positioning feature.
Fig. 6. Definition of the rightward and downward vectors by center of
the positioning features.
C. Edge and Corner Identification
In this part, we concentrate more on the core tool, using the
vectors to express and calculate the coordinates of the pixel
points. The approach is developed by ourselves.
1) Define Directions: Two directions of the vector can be
determined, too. They are the rightward and the downward.
They are defined just by the lining central points, which are
862
illustrated in Fig. 6. The leftward and the upward are the same.
In the later discussion, the directions are based on these
vectors defined above.
2) Scan out Edge Points: As for the edge recognition,
scanning started from the central points of the positioning
feature pattern, and extending upward following the color
shade by dark, light, dark, and light. Grope one of the points
of the positioning feature edge A, as well as the point B
leftward, shown in Fig. 7(a). In this way, six edge points of
the pattern are found as illustrated in Fig. 7(b), and they are
numbered from 1 to 6.
3) Edge Determination: Point 1 and 6 determine the line
MC, and point 2 and 3 determine the line NC. The line MC
and NC focus at the point C, and the top-left corner of the
(a) (b)
Fig. 7. (a) Extending points to the edge of the positioning feature; (b)
Six edge points determined by the same method and determining the
top-left points by line MC and NC.
pattern can be determined as well. But the other three corners
and the other two lines cannot be determined, because they
are short for the bottom-right point data, illustrated in Fig.
7(b).
The other line shall be determined by the following method.
We use the least square method to draw up the other two lines.
The result shows that the method is feasible.
By using many points of the edge by the same scanning
method, the error of the edge line also declines. Take the
bottom edge as an example,
1) Scan Points: Scan the edge from the blank area to the
dark area upwards, from line NC until we cannot find the edge
for a long pixels-length. The length of the threshold is defined
for half the size of the dark area of the central positioning
features.
2) Determine Edges: Record all the coordinates of the
edge points in order to use the least square method. Fig. 8
illustrates the result of searching the edge of the pattern. The
four corners of the pattern are also determined by the focus of
the four lines.
Fig. 8. The result of the searching for the edge of the image pattern.
 D. Geometry Correction
The pattern of the QR code is not pictured regular most of
the time. It always has geometry error. In this part, we use the
affine transformation to correct the image. Zhang [12]
proposes a common method of the geometry correction by
affine transformation, which is used to transform the
trapezoids into square illustrated in Fig. 9 (a) to (b).
other three corners. Also, the threshold segmentation cannot
totally eliminate the influence of the uneven intensity of the
environment light. We need to revise the pattern of result by
linear transformation. It mainly solves problems about zoom
error and uniform distortion.

(a) (b)
Fig. 9. (a)Trapezoids edge of the pattern before transformation; (b)
The square edge of the pattern after affine transformation.
We put the trapezoids into plane coordinate, and define the
coordinates of the four points as (XA, YA), (XB, YB), (XC, YC)
and (XD, YD), and our destination coordinates as (XA’, YA’), (XB’,
YB’), (XC’, YC’) and (XD’, YD’). The transformation from
original coordinates to destination coordinates express as
follows:
Fig. 11. The proofreading position of the revision transformation.
As is shown in Fig. 11, according to the proofreading
position, we compare these positions of realistic pattern with
the expected pattern. We take the horizontal transformation as
an example. Define the left side of the coordinates as the P1i
and right side of the coordinates as P2i, and define the
expected coordinates as Q1i and Q2i, then it can be expressed
as:

ܲ ൌ ࡷܳଵ௜ ൅ ࡮
ܺ஺ ൌ ܽܺ஺ᇲ ൅ ܾܺ஺ᇲ ܻ஺ᇲ ൅ ܻܿ஺ᇲ ൅ ݀ ൜ ଵ௜ (4)
‫ۓ‬ ܲଶ௜ ൌ ࡷܳଶ௜ ൅ ࡮
ܺ஻ ൌ ܽܺ஻ᇲ ൅ ܾܺ஻ᇲ ܻ஻ ൅ ܻܿ஻ᇲ ൅ ݀
(1)
‫ܺ ۔‬஼ ൌ ܽܺ஼ ᇲ ൅ ܾܺ஼ ᇲ ܻ஼ ൅ ܻܿ஼ ᇲ ൅ ݀ The parameter matrix K and B also can be counted out by
‫ܺە‬஽ ൌ ܽܺ஽ᇲ ൅ ܾܺ஽ᇲ ܻ஽ᇲ ൅ ܻܿ஽ᇲ ൅ ݀ least square method. We transform the pixels by calculate
using the parameters due to:
ܻ஺ ൌ ݁ܺ஺ᇲ ൅ ݂ܺ஺ᇲ ܻ஺ᇲ ൅ ܻ݃஺ᇲ ൅ ݄
‫ۓ‬
ܻ஻ ൌ ݁ܺ஻ᇲ ൅ ݂ܺ஻ᇲ ܻ஻ ൅ ܻ݃஻ᇲ ൅ ݄
(2) ‫ܩ‬ሺܲ௜ ሻ ൌ ‫ܨ‬ሺܳ௜ ǡ ࡷǡ ࡮ሻ (5)
‫ܻ ۔‬஼ ൌ ݁ܺ஼ ᇲ ൅ ݂ܺ஼ ᇲ ܻ஼ ൅ ܻ݃஼ ᇲ ൅ ݄
‫ܻە‬஽ ൌ ݁ܺ஽ᇲ ൅ ݂ܺ஽ᇲ ܻ஽ᇲ ൅ ܻ݃஽ᇲ ൅ ݄ 2) Data Read
As is seen in Fig. 10, the pattern has been filled into the
As the four coordinates of the corners of the pattern have grid. Calculate the sum of each central square, which account
already obtained, the eight parameters a, b to h can be for 1/9 room of the small square. If the dark pixels filled
calculated first. Then we use the eight transformation above 50%, consider it as a dark color square, the other as a
parameters to transform each pixel to the destination pixel by light color square. And the result is just illustrated in Fig. 12.

‫ܩ‬ሺܺǡ ܻሻ ൌ ‫ܨ‬ሺܺ ᇱ ǡ ܻ ᇱ ǡ ܽǡ ܾǡ ܿǡ ݀ǡ ݁ǡ ݂ǡ ݃ǡ ݄ሻ (3)

the result is illustrated in Fig. 10.

Fig. 12. The result of the data reading pattern.

The data can be read easily, that we do not discuss the data
encryption here. The result is shown in Table I.
Fig. 10. The result of the affine transformation.
TABLE I THE RESULTS OF THE QR CODE IDENTIFICATION
EXPERIMENT.
E. Linear Revision and Data Reading Data Data
Experiment Image process
1) Linear Revision identification identification
times average time
success time average error
After the affine transformation, the pattern is
179 135 4.038% 0.4s
approximately suitable for the grid block. But for the
bottom-right corner, it cannot be recognized accurate as the

863
 III. IDENTIFICATION PROCESS BY PLANAR MOBILE ROBOT
As Zhongli [13] proposes, vision-based control is
becoming more and more popular in robot applications.
Pioneer Robot is a planar mobile robot, which has a single
camera. Fig. 13. illustrates the experiment mobile robot in
lab.
Fig. 13. Planar mobile robot with interference –driven hand in
different sight.
The main approach is shown in Fig.14. It is a traditional
feedback control system. The difference is just in computer
vision algorithm.
Fig. 14. Feedback control system.
In the experiment, computer vision is used to measure the
error between realistic and the expected image size. In order
to identify the QR code pattern, the mark should positioning
first. As the pink periphery is used as the mark boundary,
recognize the outward appearance first. Fig. 15 illustrates
some photos taken by the robot camera.
(a) 2s (b) 4s (c) 6s
(d) 8s (e) 10s (f) 12s
(g) 14s (h) 16s (i) 18s
Fig. 15. Photos taken by the robot camera. (size: 640pixels×480
pixels)
The pink periphery boundary is selected from all the
combination of RGB colors, they are R, G, B, RG, RB, GB.
1) Binary Image Conversion: We find that the RB
combination is the best due to the color-comparing
experiment. According to this feature, the rest of the colored
area is separated and converted into binary image.
2) Revision Binary Image: Eliminate the small area using
864
the erosion and dilate algorithm. The result is illustrated in
Fig.16.
(a) (b)
Fig. 16. (a) Original image; (b) Color threshold binary image.
At the moment, the size of the light area can be calculated
easily by scanning the light area.
3) Error Comparison: Comparing with the expected size, it
determines the action of the mobile robot.
The moving direction of the mobile robot is determined
according to the size of the realistic sample size. If the sample
size is bigger than expected, the robot will move backward,
otherwise move forward.
The rotation action is determined as well according to the
location of the sample pattern in the whole image. If it is in
the left of the whole image, the robot should rotate
counterclockwise; if it is in the right of the image, it should
rotate clockwise.
The distance and rotation angle are determined as two
small constant values. The sample time is 200 (ms). Therefore,
the moving velocity of the robot is selected for 10 (mm/s),
and the rotation velocity is 1 (deg/s).
At last, the error is limited in a small range, which can be
omitted for the hand grasping. The least error is selected
according to the size of the image, which is only 5 (pixel).
The whole moving processing of the mobile robot is shown in
Fig. 17. It can be seen that the direction of the moving robot is
changing all the time, although only a little and it can be
corrected by the rotation correction algorithm. The statement
is in 0 (s), 36 (s), and 54 (s), the robot moves towards the right,
and statements in 6 (s) and 12 (s), the robot moves towards
the left. All the direction errors are corrected at last. The
mobile robot comes to the position of grasp location at last,
then the computer reads the information in the QR code
successfully.
(a)0s (b)6s (c)12s (d)18s
(e)24s (f)30s (g)36s (h)42s
(i)48s (j)54s (k)60s (l)66s
Fig. 17. Mobile robot moving direction and expected direction.
IV. HAND GRASP
After the experiment of the moving identification of the
planar mobile robot, we added the control command to the
hand. The robot arrived to the position of the object and
grasped it. Then the robot acted following by the information
in the QR code.
As is illustrated in Fig. 18, the interference-driven hand is
expected to grasp the target depending on the particular
information in the QR code, which stores all the information
about the target, the next action, and the destination.
Fig. 18. Position of the robot, hand, the target to be grasped, and the
QR code mark.
The interference-driven hand is controlled by Chip
80C52RC, and the grasping experiment result is shown in Fig.
19. The whole process of the robot is illustrated in Fig. 20.
(a) Open statement of the hand (b) Closed statement of the hand
Fig. 19. Interference-driven hand experiment result.
(a)0s (b)4s (c)8s (d)12s
(e)16s (f)20s (g)24s (h)28s
(i)32s (j)36s (k)40s (l)44s
Fig. 20. The whole process of the mobile robot moving and grasping
the objects.
V. CONCLUSION AND FUTURE WORK
The robot will be used widely in family in the future,
because of population aging or heavier complex daily work.
Many of the common housework or objects moving can be
finished by robots in the coming near future. Computer vision
in objects identification takes more and more important
position. But QR code is still a planar mark, and the
identification in long-distance and slope posture is still
difficult. In this paper, we present a QR code fast
identification method to solve the problems above, making
865
the using area of the QR code more widely. By using the
vector based expressing method, the pattern and details of the
image boundary can be expressed clearly. Other studies can
be started depending on it more easily and conveniently.
Besides, we tested an application of the QR code in moving
and grasping particular objects. The results show that the
method is feasible.
In the future work, we consider more using area of the QR
code in the indoor environment. And the information data
reading can also be improved. We may also try to position the
barcode without periphery feature that the QR code may need
to be reformed.
ACKNOWLEDGMENT
We extend our sincere gratitude to the master students
Huan Chen and Bowen Shen for their supports. For their work,
the mobile robot can be controlled smoothly. We would also
acknowledgment Tao Ding for his preparing work in hand
controlling and technical support.
REFERENCES
[1] J. Sturm, K. Konolige, C. Stachniss, W. Burgard, “Vision-based
detection for learning articulation models of cabinet doors and draws in
household environments,” in IEEE Int. Conf. on Robotics and
Automation, 2010, pp. 362–368.
[2] G. Li, G. Tian, Y. Xue, “Research on QR code-based visual servo
handling of room service robot,” in J. of Southeast University (Natural
Science Edition), 2010, pp. 30–36.
[3] H. Vorobieva, M. Soury, P. Hède, C. Leroux, P. Morignot, “Object
recognition and ontology for manipulation with an assistant robot,” in
Proc. of the 8th Int. Conf. on Smart Homes and Health Telematics,
2010, pp. 178–185.
[4] H. Di, Q. Shang, S. Wang, “A virtual binocular vision range finding
method of remote object based on single rotating angle indexing
camera,” in ICEMI Proc. of 9th Int. Conf. on Electronic Measurement
and Instruments, pp. 2846-2849.
[5] X. Jiang, “Feature extraction for image recognition and computer
vision,” in Proc. 2nd IEEE Int. Conf. on Computer Science and
Information Technology, 2009, pp. 1-15.
[6] Eldridge Ross, Rudolph Heiko, Duckworth Jonathan, Peter H.,
“Improved background removal through polarization in vision-based
tabletop interface,” in 6th IEEE/ACIS Int. Conf. on Computer and
Information Science, 2007, pp. 326–331.
[7] B.C. Jiang, C.C. Wang, Y.N. Hsu, “Machine vision and background
remover-based approach for PCB solder joints inspection,” in Int. J. of
Production Research, vol. 45, 2007, pp. 451–464.
[8] X. Lu, G. Fan, Y. Wang, “A robust barcode reading method based on
image analysis of a hierarchical feature classification,” in Proc. 2006
IEEE/RSJ, Int. Conf. on Intelligent Robots and Systems, 2006, pp.
3358–3362.
[9] L. Fang, C. Xie, “1-D barcode localization in complex background,” in
Int. Conf. on Computational Intelligence and Software Engineering
(CiSE). Dec. 2010, pp. 1-3.
[10] I. Zafar, U. Zakir, E. A. Edirisinghe, “Real time multiple two
dimensional barcode reader,” in the 5th IEEE Conf. on Industrial
Electronics and Applications (ICIEA), 2010, pp. 427–432.
[11] G. Lin, X. Chen, “Robot indoor navigation method based on 2D
barcode landmark,” in Applied Mechanics and Materials, vol. 44 - 47,
2011, pp. 1279 - 1284.
[12] M. Zhang, J. Zheng, “Recognition algorithm of QR code based on
symbol characteristic,” in Computer Engineering, vol.37, Arp. 2004,
pp. 278-280.
[13] Z. Wang, X. Gao, B. Zhang, H. Wu, “Motion estimation for a mobile
robot based on real-time stereo vision system,” in Proc. Int. Congr.
Image Signal Process. CISP. 2009, pp. 1-5.