Professional Documents
Culture Documents
Opticsguided Robotic Systemfor Dental
Opticsguided Robotic Systemfor Dental
net/publication/360848946
CITATION READS
1 60
12 authors, including:
All content following this page was uploaded by Rongqian Yang on 25 August 2022.
Abstract
At present, dental implant surgery mainly relies on the clinical experience of the doctor and the assistance of preop-
erative medical imaging. However, there are some problems in dental implant surgery, such as narrow space, sight
obstruction, inaccurate positioning, and high requirements of doctors’ proficiency. Therefore, a dental implant robot
system (DIRS) guided by optical navigation is developed in this study, with an x-shaped tool and an irregular pentago-
nal tracer are designed for spatial registration and needle tip positioning strategy respectively. The coordinate system
of each unit in DIRS is unified through system calibration, spatial registration, and needle tip positioning strategy. Then
the surgical path is planned on the computed tomography (CT) images in the navigation software before operation.
The automatic positioning method and the auxiliary positioning method can be used in the operation to achieve
accurate positioning and assist doctors to complete the operation. The errors of spatial registration, needle tip posi-
tioning strategy, and the overall accuracy of the system were evaluated respectively, and the results showed that they
all met the needs of clinical surgery. This study preliminarily verified the feasibility of the precise positioning method
for dental surgery robots and provided certain ideas for subsequent related research.
Keywords: Dental implant surgery, Robot system, Calibration, Registration
© The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which
permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the
original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or
other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line
to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory
regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this
licence, visit http://creativecommons.org/licenses/by/4.0/.
Yan et al. Chinese Journal of Mechanical Engineering (2022) 35:55 Page 2 of 13
beyond that of a surgeon’s hand and wrist [8–10]. Up to points recorded by optical positioning can also be used
now, it has been widely used in orthopedic surgery and for spatial registration. Currently, the research about
neurosurgery. For example, the ROSA Brain designed by optical-based robot systems is divided into two catego-
Medtech is one of the most representative neurosurgical ries consisting of the “eye in hand” type (EIH) and “eye to
robots [11], and a new orthopedic surgical robot system hand” type (ETH) [19, 20]. It is necessary to change the
called Tianji was listed by Tinavi recently [12]. However, surgical instrument mounted on the end of the robot that
the systems of those types have a large operating space may bring errors into the robot system. In addition, due
and more acceptable. In the field of dental implant sur- to the oral cavity is narrow, the marker may be outside
gery [13], due to precision limitations, surgical robot of the camera’s FoV when the robot arm moves to a cer-
technology is still not mature enough. tain position [21]. In the robot system with ETH type, the
As early as 2010, Sun et al. [14] proposed the auto- marker and surgical instrument can be totally observed
mated dental implantation using image-guided robot- at the camera’s FoV, and it is more flexible, more stable.
ics, and completed the registration of surgical space In this study, an optical-based dental implant robot sys-
and image space. The final registration result was (1.42 tem (DIRS), which adopts ETH type, is designed to assist
± 0.7) mm. This work provides ideas for the following dental implant surgery. An optical tracking system (OTS)
research but does not involve final positioning results. Li provides the location information of retro-reflective
et al. [15] proposed a compact robotic system. The robot markers mounted on the surgical tool and skull model. In
design utilized tendon-sheath transmission, by which order to realize hand-eye cooperation, a multiple closed-
the actuators could be placed in robot base, resulting a loop calibration (MCLC) algorithm is used to calculate
compact size and lightweight. The first robotic surgery the rotation and translation relationship between the
system called ‘Yomi’ in the United States is approved for robot and the OTS. At the same time, in order to achieve
dental implant surgery in 2017 by U. S. Food and Drug the precise positioning of the needle tip in the robot
Administration (FDA). This first system provides soft- base coordinate system and the matching of the head
ware for planning and navigational guidance for instru- model in the image space and the actual surgical space,
mentation during implant surgery [16]. The system also an x-shaped tool and an irregular pentagonal tracer are
delivers haptic feedback and controls the position, depth, designed respectively. Then the surgical instrument and
and angulation for implant osteotomy. The operational model in image space and real space are matched by spa-
arm of the Yomi system assists in precise and automatic tial registration, spatial position information of drill nee-
positioning by connecting a coordinate measurement dle in robot coordinate system is obtained by needle tip
machine (CMM) arm to the patient’s teeth. The opera- positioning strategy. Finally, the appropriate surgical path
tions of drilling and implant placement are still per- and target can be selected according to the automatic
formed manually by the dentists using the operational positioning and auxiliary positioning methods, and the
arm. The operational arm itself does not perform any robot is controlled to reach the specified position along
automated operations, and the CMM arm occupies the the path and perform the operation.
narrow operation space in the patients’ oral cavity [12].
So it is necessary to design a simpler registration tool to
register image space and surgical space. Besides, a fully 2 System Component
automatic dental implant system was developed by Yim- In Figure 1, the DIRS is composed of the following com-
ing Zhao from the Air Force Medical University of China. ponents: a six-degree-of-freedom (6 DoF) robotic arm
This intelligent robot with a high degree of autonomy can (UR5e, Universal Robot Corporation) for actuating the
automatically adjust continuously during intraoperative surgical instrument; an OTS (AimPosition, Aimooe Cor-
procedures, and can execute surgical tasks directly on poration) for tracking the retro-reflective markers within
patients without any apparent control by a surgeon [17]. the FoV; a self-designed x-shaped tool with four retro-
However, limited confirming research is available regard- reflective markers for needle tip positioning strategy
ing the reliability and feasibility of this system in clinical and irregular pentagonal tracer with five retro-reflective
practice. So the robot must first be accurate positioning, markers for spatial registration; a navigation system inte-
to truly achieve a more safe and reliable dental implant grated into a robot control center, which is Lenovo Think-
robot clinical application. Station P720 with an Intel Xeon Silver 4210R at a 2.40
Optical positioning is one of the common position- GHz CPU and a 32 GB RAM for image segmentation,
ing methods. It has high accuracy and wide field of view algorithm analysis and robot communication control in
(FoV) [18]. At the same time, it is not easy to be dis- software; a skull model and its CT image data is used as
turbed, and can provide accurate surgical navigation for an experimental subject and a dental implant machine
the robot. At the same time, the coordinates of marker for drilling holes in the jaw. The robot control center can
Yan et al. Chinese Journal of Mechanical Engineering (2022) 35:55 Page 3 of 13
pi = (xi , yi , zi , rxi , ryi , rzi ),(xi , yi , zi ) and (rxi , ryi , rzi ) stand
for the position and orientation respectively. The robot
communicate with the robot and OTS through TCP/IP starts from each initial position and moves to the offset
and USB respectively. position. The offset pose is calculated according to the
initial pose and the axial direction of the base coordi-
3 Methods nate system Cb and the end effector coordinate system
Robot movement can only be carried out in the robot Ce is the extension direction.
base coordinate system, while the visual information is In the process of the robot moving from the initial
captured by OTS. Therefore, the coordinate system of pose to all the offset positions, the reflective mark-
each unit in the system should be unified through the fol- ers on the surgical tool as shown in Figure 3 should be
lowing system calibration, space registration and needle located in the FoV of the OTS to ensure that the OTS
tip positioning strategy. can capture the markers in real time. In order to reduce
the inherent error, this algorithm takes a total of 25 ini-
tial poses. After the robot reaches each initial pose, it is
3.1 System Calibration offset 4 times along each coordinate axis of Cb , and the
System calibration of the robot and OTS is the founda- offset distance is δ each time in Eq. (1). When the robot
tion to surgical robot application, coordinate transforma- moves along the Cb axis to each target pose, the spatial
tion relation between robot base to OTS and robot end position of the three non-collinear reflective markers
effector to the surgical tool could be calculated through on the surgical tool is recorded as Eq. (2):
calibration. In this study, on the basis of the MCLC algo-
rithm proposed previously [22], the coordinate value
jx � �
pi = � xi + jδ, yi , zi , rxi , ryi , rzi� ,
matrix of the marker points recorded by OTS can be jy
pi = xi , yi + jδ, zi , rxi , ryi , rzi , (1)
used as an input to realize the system calibration. The jz �
�
pi = xi , yi , zi + jδ, rxi , ryi , rzi ,
calculation process of MCLC is divided into three loops,
as shown in Figure 2, the first loop consists of robot
base, robot end-effector, and OTS, the second loop con-
Table 1 The lowercase meaning
sists of robot end-effector, surgical tool, dental implant
machine needle tip, and OTS, and the first two loops Symbol Meaning
are connected to get the third loop. The following pro-
b Robot base
cedure obtains the input matrix required for the MCLC
e Robot end effector
algorithm.
t Surgical tool
Cx is the coordinate system of x , Rxy and T xy denote
p Dental implant
the rotation matrix and translation matrix from Cx to Cy o machine needle
respectively. The lowercase meaning of each coordinate tip
system is shown in Table 1. Optical tracking
system
To begin with, a series of initial pose pi of
i CT image
robot end effector in Cb should be selected, and
Yan et al. Chinese Journal of Mechanical Engineering (2022) 35:55 Page 4 of 13
Figure 5 a Skull model in actual surgical space, b Skull model in image space
of resin materials by 3D-printing, and the function of the the surgical space and image space. And then the ele-
tracer is to fix 5 retro-reflective markers pj. ments in the matrix s D are subtracted from the corre-
Figure 5a shows a rigid structure can be formed by sponding elements of the matrix i D to obtain the error
sticking tracer onto the teeth of a skull model. After CT matrix E.
scanning and 3D visualization reconstruction, a 3D skull
model image can be obtained in the navigation software E = e1 e2 e3 e4 , (10)
system of the robot control center as shown in Figure 5b.
The Euclidean distance between every two markers T
ej = ej1 ej2 ej3 ej4 . (11)
on the 3D skull model can be calculated by image seg-
ment. After each column element of the Euclidean dis- If the minimum element of the first column vector e1
tance matrix is arranged in descending order, it can be in matrix E is e1k , then the marker p1 in the surgical
expressed as: space corresponds to the marker qk in the image space.
Then the quaternion method can be used to obtain the
i D = i d1 i d2 i d3 i d4 . (6) coordinate transformation relationship between the
surgical space and the image space.
When the skull model is placed in the FoV of OTS,
markers on the tracer can be identified and tracked. And
Euclidean distance of every two markers can be calcu- 3.3 Needle Tip Positioning Strategy
lated and arranged in descending order similarly as: As shown in Figure 3, three non-collinear retro-reflec-
tive markers construct the surgical tool coordinate
system Ct . To implement precise positioning, it is nec-
sD = sd1 sd2 sd3 sd4 , (7)
essary to calculate the coordinate value of the drill nee-
(8)
T
sdj = pj − p1 , · · · , pj − pj−1 , pj − pj+1 , · · · , pj − p5 ,
(9)
T
idj = qj − q1 , · · · , qj − qj−1 , qj − qj+1 , · · · , qj − q5 ,
where, matrix i D matrix s D are both 4×4 order matrices; dle tip in Ct . OTS can track and locate retro-reflective
s d j and i d j respectively represent the Euclidean distance markers in the FoV to establish surgical tool coordinate
between the j-th marker and the other four markers in system Ct , while the dental implant machine’s needle
Yan et al. Chinese Journal of Mechanical Engineering (2022) 35:55 Page 6 of 13
Figure 6 The x-shaped tool: a Vertical view, b Establish the coordinate system of Cp , c Use the x-shaped tool to determine the position of the
needle tip and keep it vertical to determine the direction of the needle body, d Some drill needles of different sizes
tip coordinate system Cp can be established by markers to determine the direction and tip position of the needle.
similarly. The rotation matrix Rbo , Ret , Rbe and transla- As shown in Figure 6c, when using it to locate the tip of
tion matrix T bo , T et , T be have been calculated in sys- the needle, the drill needle needs to be vertical to the center
tem calibration, but the relationship between Ct and Cp hole of the x-shaped tool. Then let the center at the bot-
remains unknown. In clinical practice, it is necessary to tom of the hole be the origin of Cp , the V x pointing hori-
replace drill needles in Figure 6(d) with different diam- zontally to Pm be the X axis, V y be the Y axis. So the Z axis
eters and lengths during operation. Hence, a separate V z shown in Figure 6b, can be expressed as
x-shaped tool is designed to establish Cp and determine
the position of the drill needle tip in that coordinate
V z = V x × V y. (12)
system. After theoretical measurement and calculation, the
Figure 6a shows a hole of 3 mm in diameter and 3 mm in rough spatial coordinates of the four markers’ center
depth at the center of the x-shaped tool. This hole is used can be obtained. At the same time, the origin and a
Yan et al. Chinese Journal of Mechanical Engineering (2022) 35:55 Page 7 of 13
Figure 8 Surgical path planning: a Automatic positioning method, set the target on the alveolar bone, b Auxiliary positioning method, set the
target point outside the alveolar bone
Pbt Pit
Rob T ob Rio T io
1
=
0 1 0 1 1
. (16)
Figure 11 The results of automatic positioning method: a Tip positioning result display, b Most of the eight final positioning results are within
1mm, which can meet the actual needs of dental implant surgery
technology [29]. As a representative product of dental to the patient’s teeth, and it must be closely embedded
implant robot, YOMI system adopts the assisted arm on the teeth during the actual registration process. If it
positioning method, and its positioning deviation value becomes loose, it may cause positioning failure. Likewise,
is 0.8 mm [16]. Therefore, compared with the YOMI the needle point positioning strategy also has require-
system, DIRS has similar positioning accuracy, but the ments. It is necessary to ensure that the drill needle is
oral space occupied by the tracer used for spatial regis- perpendicular to the x-shaped tool so that the robot can
tration is smaller and lighter, which will not hinder the adjust the angle of the drill needle to the angle required
robot. Both automatic positioning and auxiliary position- by the implant before reaching the target point.
ing methods have high accuracy, and the path is care- Overall, there are certain limitations in this study.
fully designed by professional dentists before operation, Firstly, we did not carry out animal experiment verifica-
and the robot will not exceed the preset path. In fact, the tion, which is also the work we need to do in the future.
manipulator itself also has the function of emergency Secondly, if the automatic positioning method is adopted,
braking. In case of emergencies, doctors can directly the patient should remain static in the operation, and
intervene in braking to ensure safety. the auxiliary positioning method can only guide the
The spatial registration and needle tip positioning strat- doctor to control the drill needle to move along a fixed
egy based on optical markers has achieved good results. path, which requires more intervention from the doctor.
Compared with the autonomous oral and maxillofa- Finally, although the experimental results show that the
cial surgery robot proposed by Ma et al. [30], which can positioning accuracy is less than 1mm, since the system
markerlessly detect the pose of the skull model, its final itself is complex and has many sources of error, the accu-
positioning error is larger than 1mm. The spatial registra- racy cannot be guaranteed after the error of each unit is
tion and needle-point positioning strategy we designed accumulated. This system still needs to reduce the source
not only occupies a small space, but also has a higher of system error and a large number of experimental veri-
accuracy of identification and tracking, and the accumu- fication to ensure the safety and feasibility of the system.
lated error is smaller. Meanwhile, the time consuming In summary, this paper developed a DIRS to assist den-
each time is very short. It only takes 0.5 milliseconds for a tal implant surgery. The spatial position relationships of
single mark recognition calculation, shortening the time each unit in DIRS are unified by system calibration, spa-
of robot-assisted surgery. The pentagonal tracer used for tial registration, and needle tip positioning strategy. The
spatial registration needs to be customized according system can achieve high accuracy either by automatic
Yan et al. Chinese Journal of Mechanical Engineering (2022) 35:55 Page 11 of 13
Figure 12 The results of auxiliary positioning method: a After the needle tip reaches the target point, manually push the robotic arm to move
along the fixed path in the UR5e force mode, b The trajectory of the needle tip, c Real-time angle of each joint of the robot, d Real-time angular
velocity of each joint of the robot, the time period when the speed is zero represents the program interval stage
Yan et al. Chinese Journal of Mechanical Engineering (2022) 35:55 Page 12 of 13
positioning method or by auxiliary positioning method. Peifeng Guan, received the B.S. degree in biomedical engineering from the
South China University of Technology, China, in 2013, and the M.S. degree in bio-
With the deepening of automation and intelligence of medical engineering from the South China University of Technology, China, in
medical devices, more intelligent medical instruments 2016. He is currently an engineer from Guangzhou Aimooe Technology Co., Ltd.
will appear in future clinical surgery. The purpose of His main research interests include surgical navigation systems and robotics.
Wei Yang, received the B.Sc. degree in automation from the Wuhan University
this system is to establish a more standardized surgical of Science and Technology, China, in 2001, the M.Sc. degree in control theory
procedure, design a better surgical plan in the surgical and control engineering from Xiamen University, China, in 2005, and the Ph.D.
navigation software in pre-operation, and assist doctors degree in biomedical engineering from Shanghai Jiao Tong University, China,
in 2009. He is currently a Professor with the School of Biomedical Engineer-
in accurately positioning the target area for drilling. The ing, Southern Medical University, China. His main research interests include
advantage is that the surgical robot has good stability, medical image analysis, machine learning, and computerized-aid diagnosis.
and the preoperative scheme is consistent with the intra- Tel:+86-13433964132.
Jiang Li, received his Bachelor and Master degree in dentistry from Norman
operative positioning. This work may be able to provide Bethune University of Medical Science, China, in 1993 and 1999, and the Doc-
new ideas and solutions for the development of robots in tor degree in dentistry from Jilin University, China, in 2002. He is cuttently a
the field of dental implants. Professor and Chief Physician in Dept. Prosthodontics, School of Stomatology,
Guangzhou Medical University, China. His research focus on screening of nature
Acknowledgements products in sjogren’s syndrome, molecular signal transduction, and AI applica-
Not applicable. tion in dentistry.
Rongqian Yang, received the B.S. degree in electronic instrumentation and
Author contributions measurement from Nanchang Hangkong University, China, in 2001, the M.S.
BY: Conceptualization, methodology, writing-original draft; WZ, LC, LZ and KB: degree in communication and information systems from Jinan University,
Resources, data curation, software; YR, LY, WY, PG, WY, JL: Supervision, funding Guangzhou, China, in 2005, and the Ph.D. degree in biomedical engineering
acquisition and validation; RY: writing-reviewing and editing. All authors read from Shanghai Jiao Tong University, China, in 2009. He is currently an associate
and approved the final manuscript. professor with the Department of Biomedical Engineering, South China University
of Technology, China.. His main research areas include medical robotics and
Authors’ Information biomedical instruments.
Biao Yan, born in 1997, is currently pursuing the M.S. degree at South China
University of Technology, China. He received the B.S. degree from South Funding
China University of Technology, China, in 2020. His main research interests Supported by Natural Science Foundation of Guangdong Province (Grant No.
include computer vision, medical image visualization and robotics. Tel: +86 2021A1515011208), National Natural Scientific Foundation of China (Grant No.
13132318724. 81671788), National Science Foundation for Young Scientists of China (Grant
Wenlong Zhang, born in 1997, is currently pursuing the M.S. degree at South No. 81701662) and The Joint Found of National Science Foundation of China
China University of Technology, China. He received the B.S. degree from Hohai and GuangDong Provincial Government (Grant No. U1301258).
University, China, in 2020. His main research interests include force feedback,
light force fusion and robotics. Tel: +86 19129212125/ Competing interests
Lijing Cai, rceived her B.S. degree in biomedical engineering from South China The authors declare no competing financial interests.
University of Technology, China, in 2019, where she is currently pursuing the
M.S. degree in biomedical engineering. Her main research interests include Author Details
1
medical image visualization and medical robotics. Tel: +86- 18073725995. School of Materials Science and Engineering, South China University
Lingxiang Zheng, born in 1995. He received the B.S. degree in biomedical of Technology, Guangzhou, China. 2 RD Department, Shenzhen SONTU Medi-
engineering in 2018 from the South China University of Technology, China, cal Imaging Equipment Co., Ltd, Shenzhen, China. 3 Department of Radiology,
where he is currently working toward the M.S. degree in biomedical engineer- Guangdong Provincial People’s Hospital, Guangdong Academy of Medical
ing with the South China University of Technology, China. His main research Sciences, Guangzhou, China. 4 Aimooe Technology Co., Ltd, Guangzhou, China.
5
interests include computer vision, medical image visualization, and robotics. School of Biomedical Engineering, Southern Medical University, Guangzhou,
Tel:+86-7620959105. China. 6 Department Prosthodontics, School of Stomatology, Guangzhou
Kaiyang Bao, received the B.S. degree in biomedical engineering from the Medical University, Guangzhou, China.
Changzhi Medical College, China, in 2017. He is currently pursuing the M.S.
degree in biomedical engineering with the South China University of Technol- Received: 24 May 2021 Revised: 15 April 2022 Accepted: 17 April 2022
ogy, China. His main research interests include computer vision, medical image
visualization, and robotics. Tel:+86 18026416849.
Yuming Rao, received the degree of Bachelor of Engineering in biomedical
engineering from the Tianjin Medical University, China, in 2003, the degree of
Master of Engineering in biomedical engineering from Shanghai Jiao Tong References
University, China, in 2007. He is currently senior engineer at the RD Department, [1] M H Kim, E J Cho, J W Lee, et al. A study on setting of the fatigue limit of
Shenzhen SONTU Medical Imaging Equipment Co., Ltd, China. His main research temporary dental implants. Proceedings of the 39th Annual International
interests include medical image analysis, machine learning, and medical Conference of the IEEE Engineering in Medicine and Biology Society (EMBC),
devices. Jeju Island, Korea, F, 2017: 1962-1965.
Lin Yang, born in 1982. is currently an Attending Physician with the Depart- [2] R J Fonseca. Oral and Maxillofacial Surgery-E-Book: 3-Volume Set. Elsevier
ment of Radiology, Guangdong Provincial People’s Hospital, Guangdong Academy Health Sciences, 2017.
of Medical Sciences, China. He received the Ph.D. degrees from Southern Medi- [3] M Temiz, E Dayi, N Saruhan. Evaluation of blood titanium levels and total
cal University, China, in 2017.His research interests include the application of bone contact area of dental implants. BioMed research international,
optical navigation clinic and cardiovascular imaging. Tel: +86-020-83870125. 2018(1): 1-7.
Weitao Ye, born in 1989. is currently a Resident Doctor of the Department [4] P Ahmad, M K Alam, A Aldajani, et al. Dental robotics: a disruptive tech-
of Radiology, Guangdong Provincial People’s Hospital, Guangdong Academy of nology. Sensors, 2021, 21(10): 3308-3323.
Medical Sciences, China. He received the M.D. degree from Southern Medical [5] N Panchal, L Mahmood, A Retana, et al. Dynamic navigation for dental
University, China in 2016. His research interests include the abdominal imaging implant surgery. Oral and maxillofacial surgery clinics of North America,
and interventional radiology. Tel: +86-020-83870125. 2019, 31(4): 539-47.
Yan et al. Chinese Journal of Mechanical Engineering (2022) 35:55 Page 13 of 13