07/12/2020

超声机器人的目的主要是采集清晰的超声图片

https://xueshu.baidu.com/usercenter/paper/show?paperid=1s4g0aa077470ep0sq790rf0qb261248&site=xueshu_se

http://mp.ofweek.com/medical/a045693226316

Toward Real-time 3D Ultrasound Registration-based Visual Servoing for Interventional Navigation

Abstract：While intraoperative imaging is commonly used to guide surgical interventions, automatic robotic support for image-guided

navigation has not yet been established in clinical routine. In this paper, we propose a novel visual servoing framework that combines,

for the first time, full image-based 3D ultrasound registration with a real-time servo-control scheme. Paired with multi-modal fusion to a

pre-interventional plan such as an annotated needle insertion path, it thus allows tracking a target anatomy, continuously updating the

plan as the target moves, and keeping a needle guide aligned for accurate manual insertion. The presented system includes a motorized

3D ultrasound transducer mounted on a force-controlled robot and a GPU-based image processing toolkit. The tracking accuracy of our

framework is validated on a geometric agar/gelatin phantom using a second robot, achieving positioning errors of on average 0.420.44

mm. With compounding and registration runtimes of up to total around 550 ms, real-time performance comes into reach. We also

present initial results on a spine phantom, demonstrating the feasibility of our system for lumbar spine injections.

摘要—虽然术中成像通常用于指导外科手术干预，但在临床常规操作中尚未为图像导航提供自动机器人支持。在本文中，我们

提出了一种新颖的视觉伺服框架，该框架首次将基于全图像的 3D 超声配准与实时伺服控制方案相结合。与多模式融合与介入

前计划（例如带注释的针头插入路径）配对使用，因此可以跟踪目标解剖体，随着目标的移动不断更新计划，并保持导针器对

齐以进行准确的手动插入。提出的系统包括安装在受力控制的机器人上的电动 3D 超声换能器和基于 GPU 的图像处理工具包。

使用第二台机器人在几何琼脂/明胶体模上验证了我们框架的跟踪精度，平均定位误差为 0.42 0.44 mm。混合和注册运行时间总

计约 550 ms，实时性能可达到。我们还在脊柱体模上展示了初步结果，证明了我们的系统用于腰椎注射的可行性。

I. INTRODUCTION

Ultrasound (US) has become a standard diagnostic and guidance modality in many clinical felds, especially because of its low cost, lack of

ionizing radiation and ease of use in interventional settings. In particular, US is real-time and therefore allows for movement-

independent guidance. However, the exclusive use of US imaging has not become standard of care for many interventions. In this work,

we focus on needle injections in the lumbar spine [1], for which guidance with X-ray fluoroscopy is currently recommended [2]. But also

in other felds, such as electrode placement for deep brain stimulation [3], or cardiac catheter placement [4], US guidance has not yet

reached maturity. In these scenarios, lower image quality, limited view of the anatomy and higher difficulty in interpreting the images

are regularly identified as challenges in guaranteeing accurate placement [2]. In addition, manually navigating the transducer to the

correct site and maintaining a suitable acoustic window has been reported cumbersome and linked to a steep learning curve [5].

一，引言

超声（US）已成为许多临床领域的标准诊断和指导手段，尤其是由于其成本低，缺乏电离辐射和易于在干预环境中使用。特别

是，US 是实时的，因此允许独立于运动的导航。但是，在许多干预措施中，仅使用 US 成像技术尚未成为护理的标准。在这项

工作中，我们专注于腰椎的针头注射[1]，目前推荐使用 X 射线透视检查指导[2]。但在其他方面，例如用于深部脑刺激的电极放

置[3]或心脏导管放置[4]，US 指南尚未成熟。在这些情况下，较低的图像质量，有限的解剖结构视图和较高的解释图像难度被

定期确定为保证准确放置的挑战[2]。另外，据报道，手动将换能器导航到正确的位置并保持合适的声窗很麻烦，并且与陡峭的

学习曲线有关[5]。

In recent years, several techniques have been proposed to tackle a subset of these limitations. Real-time guidance can be achieved by

optical or electromagnetic tracking of a freehand US transducer and initial registration to a preinterventional CT or MR image (as in [6],

[7], [8] and [9]), or an atlas of the spine [10]. Once the registration is established, and as long as the patient does not move, the contours

of the vertebrae can be highlighted, and the insertion path of a tool, which is also tracked, visualized. Brudfors et al. [11] utilized such an

atlas and presented again a tracker-less guidance system for spine anaesthesia. In their study, a statistical spine model is continuously
registered to and overlaid on live 3D US volumes together with a calibrated needle guide line.

近年来，已经提出了几种技术来解决这些局限性的一部分。 可以通过徒手操作的 US 换能器的光学或电磁跟踪并初步注册到介

入前的 CT 或 MR 图像（如[6]，[7]，[8]和[9]）或图集来实现实时指导。 脊柱[10]。 一旦建立配准，并且只要患者不动，就可以

突出显示椎骨的轮廓，并且还可以跟踪，可视化工具的插入路径。 Brudfors 等[11]利用这样的地图集，并再次提出了一种无追

踪器的脊柱麻醉引导系统。 在他们的研究中，将统计脊柱模型与校准后的针头引导线连续注册并覆盖在实时 3D US 体积上。

Aforementioned works share the common disadvantage that the physician is required to manually maintain sufficient image quality

using a handheld US transducer. Hence, several robot-assisted systems incorporating a visual servocontrol scheme based on features

derived from the live 2D or 3D US images have been proposed in the last decade. Exemplary applications include in-plane [12] and out-

ofplane [13] carotid artery tracking, compensation of organ motion [14], and maintenance of visibility in tele-operated settings [15]. In a

variety of works, graphics hardware is exploited for real-time processing of US images, including motion detection with strain imaging

[16].

上述工作具有共同的缺点，即需要医师使用手持式 US 换能器手动保持足够的图像质量。 因此，在过去的十年中，已经提出了

几种结合了基于实时 2D 或 3D US 图像特征的视觉伺服控制方案的机器人辅助系统。 示例性应用包括颈内动脉的平面内 [12]和

平面外[13]，器官运动的补偿[14]以及在远距操作环境中保持可见度[15]。 在各种各样的作品中，图形硬件被用于 US 图像的实

时处理，包括带有应变成像的运动检测[16]。

The vast majority of prior art on ultrasound servoing tackles the issue of needle tracking and steering, regularly relying on a static US

probe, see [17] and [18]. Nevertheless, as necessary control schemes translate seamlessly to systems where the US transducer itself is

steered by the robot, progress has been made toward full automation of specific parts of certain surgeries. Nadeau et al. [19] have

shown that real-time visual servoing for both instrument and anatomy tracking for reasonably small regions of interest is feasible.

超声伺服的绝大多数现有技术解决了针跟踪和操纵的问题，通常依靠静态的 US 探头，请参见[17]和[18]。 然而，由于必要的控

制方案可以无缝转换为由机器人操纵 US 传感器本身的系统，因此在某些手术的特定部分实现完全自动化方面已经取得了进展 。

Nadeau 等。 文献[19]表明，在适当的小范围内对器械和解剖结构进行实时视觉伺服是可行的。

To the best of our knowledge, fully employing complete image-based 3D-to-3D volume registration in the scope of US transducer visual

servoing has not been performed yet. In this work, we propose a novel robotic visual servoing framework based on 3D ultrasound

images that continuously re-registers the live image stream with an interventional plan. Derived using multi-modal image registration,

such a plan is based on annotions on pre-interventional CT or MR images. The rigid transformation found by the registration algorithm

directly serves as signal for servo-control. Any movement of the target anatomy therefore leads to an adjustment of the pose of the US

transducer. Thus, our framework does not only release the physician from holding the US probe, but at the same time provides

continuous guidance even if the target moves.

据我们所知，在 US 换能器视觉伺服范围内，尚未完全采用完全基于图像的 3D 到 3D 体积配准。 在这项工作中，我们提出了一

种基于 3D 超声图像的新颖的机器人视觉伺服框架，该框架可通过干预计划连续重新配准实时图像流。 使用多模式图像配准派

生的这种计划基于干预前 CT 或 MR 图像上的注释。 配准算法发现的刚性变换直接用作伺服控制的信号。 因此，目标解剖结构

的任何移动都会导致 US 换能器的姿势调整。 因此，我们的框架不仅使医师不再持有 US 探针，而且即使目标移动，也能提供

连续的指导。

II. METHODS

Our interventional navigation system utilizes a robot with seven degrees of freedom, equipped with torque sensors at each joint, and an

US transducer rigidly attached to its end-effector (see Fig. 1a). The robot is controlled following the system architecture illustrated in Fig.

2. The two-layer concept reflects the proposed combination of intensity-based image registration and visual servoing. In an outer control

loop, each incoming 3D US image is registered to an intial scan. The resulting transformation to align these two images is utilized to

update the desired pose of the US transducer, thus following a moving target anatomy. An inner control loop implements an indirect,

compliant force control scheme, maintaining a constant contact force onto the patient. This does not only ensure sufcient image quality

but also tackles patient safety concerns. After describing the control laws for both loops in Sec. II-A/B, the alignment to a planned needle

path is explained in Sec. II-C.

二。方法
我们的介入式导航系统利用了一个具有七个自由度的机器人，每个关节处均配备了扭矩传感器，并在其末端执行器上固定了一

个 US 传感器（见图 1a）。机器人按照图 2 所示的系统架构进行控制。两层概念反映了基于强度的图像配准和视觉伺服的建议

组合。在外部控制循环中，每个传入的 3D US 图像都会配准到初始扫描。用来对齐这两个图像的结果转换可用于更新 US 换能器

的所需位姿，从而遵循移动的目标解剖结构。内部控制回路实现了一种间接的，顺应性的力控制方案，可保持对患者的恒定接

触力。这不仅可以确保足够的图像质量，而且可以解决患者的安全隐患。在描述了两个循环中的控制规律之后。在第 II-A / B 节

中，与计划中的针头路径的对齐方式在第二节中说明。 II-C。

A. Impedance Control Scheme

Impedance control 的目的: EE 的位置和外力 F ext 之间的关系，使用机器人配置和关节扭矩传感器测量得到。

Impedance control 的预期行为：平衡外力 F ext 和预期的 EE 上的力 F d ，即 F ex t −F d =0

根据参考文献 20：τ cmd=J ¿

τ cmd：关节扭矩指令
J ( q ) ：Jacobian Matrix
K∧D ：positive definite damping and stiffness matrices
∆ x ：笛卡尔坐标系下的实际位置和预期位置的误差
：不受外部影响的逆动力学系统的解决方案
f idyn [ q , q̇ , q̈ ]
B. Visual Control Scheme

visual servoing controller 的目的：补偿通过 3D UltraSound 成像可见的目标解剖运动。

C. Needle Guide Control Scheme

a needle target point t 0

a suitable insertion path ⃗

s0
an intended US plane for image guidance are defined in a pre-interventional(CT/MR) image Ip

III. SYSTEM SETUP

A. Hardware

B. 3D Image Acquisition

IV. EXPERIMENTS

A. Tracking Accuracy Validation

B. Spine Phantom Validation

V. RESULTS

A. Tracking Accuracy Validation

B. Spine Phantom Validation

C. Computational Performance

VI. DISCUSSION

The relative tracking errors obtained in our sphere phantom experiments illustrate that accurate tracking relative to an initial 3D image is

possible, and that there is no drift of the tracking performance over time. This is particularly important in clinical scenarios requiring

image guidance over longer durations. However, the time until complete convergence, i.e. motionlessness after target movement, is

relatively high, even though compounding and registration are computed in parallel on GPU. In this regard, several factors can be

considered to increase the rapidity of the system. First, the low 3D volume update rate clearly forms an obstacle. The utilization of 4D

matrix US transducers or high frame rate technology [31] might solve this problem. Second, the registration itself could be potentially
sped up by reducing the image size (downsampling) or by registering only parts of the involved images. A more extensive study is

required to determine the inuence of registration accuracy on the servoing behavior. Reducing the overall update rate of the system

might also limit the oscillations (overshoots) observed after rapid movements, which are induced by the fact that while the robot is

compensating, new (distorted) volumes are acquired. In the future, this effect could also be mitigated by using the current robot

position for each individual US frame during compounding. Nevertheless, current limitations in the imaging rate do not restrict the

usability of the proposed framework as the controller regularly recovers after few iterations, and the expected target movements in

clinical routine are smaller than those evaluated in our experiments [30].

在我们的球体模型实验中获得的相对跟踪误差表明，相对于初始 3D 图像的精确跟踪是可能的，并且跟踪性能不会随时间发生

漂移。这在需要较长时间图像指导的临床场景中尤其重要。但是，即使复合和配准是在 GPU 上并行计算的，直到完全收敛（即

目标运动后静止不动）的时间也相对较长。在这方面，可以考虑多种因素来提高系统的速度。首先，低 3D 体积更新率显然构

成障碍。利用 4D 矩阵 US 换能器或高帧率技术[31]可以解决这个问题。其次，可以通过减小图像大小（下采样）或仅对所涉及

图像的一部分进行注册来加快注册本身。需要更广泛的研究来确定套准精度对伺服行为的影响。降低系统的整体更新速率也可

能会限制快速运动后观察到的振荡（过冲），这是由于在机器人进行补偿时会获取新的（失真的）体积而引起的。将来，还可

以通过在合成期间为每个单独的 US 框架使用当前机器人位置来减轻这种影响。尽管如此，当前的成像速率限制并没有限制所

提出框架的可用性，因为控制器经过几次迭代后便会定期恢复，并且临床常规操作中的预期目标运动要小于我们实验中评估的

目标运动[30]。

VII. CONCLUSION

In this work, we presented an overall solution for interventional navigation based on multi-modal fusion to preinterventional data, and a

novel visual servoing framework that combines real-time image-based 3D ultrasound registration with real-time servo-control. In

phantom experiments including a lumbar spine model, we have demonstrated that the proposed robotic system can accurately follow

moving target anatomies and thus greatly support physicians in performing needle insertions. Our framework is generic and can

potentially be employed in the future also to other interventions that require ultrasound guidance.

在这项工作中，我们提出了一种基于多模式融合到介入前数据的介入导航的整体解决方案，以及将基于实时图像的 3D 超声配

准与实时伺服控制相结合的新型视觉伺服框架。 在包括腰椎模型的幻像实验中，我们已经证明了所提出的机器人系统可以准

确地跟踪运动目标解剖结构，从而极大地支持医师进行针头插入。 我们的框架是通用的，将来可能还会用于需要超声引导的

其他干预措施。