Ocean Engineering 260 (2022) 111968

Contents lists available at ScienceDirect

Ocean Engineering
journal homepage: www.elsevier.com/locate/oceaneng

Deployment of a deep-learning based multi-view stereo approach for

measurement of ship shell plates
Pengpeng He a, b, Delin Hu a, b, Yong Hu a, b, *
a
Key Laboratory of High Performance Ship Technology (Wuhan University of Technology), Ministry of Education, Wuhan, 430063, China
b
School of Naval Architecture, Ocean and Energy Power Engineering, Wuhan University of Technology, Wuhan, 430063, China

A R T I C L E I N F O A B S T R A C T

Keywords: In this work, a measurement system integrated with a deep-learning based multi-view stereo (MVS) approach is
Ship shell plates developed to measure ship shell plates. Specifically, a deep learning architecture of CasMVSNet is deployed for
Vision measurement depth map inference from multi-view images, which can remarkably decrease the dense reconstruction time and
Deep learning
GPU memory consumption. This is the first report that a learned MVS architecture is deployed to three-
Multi-view stereo
3D reconstruction
dimensional (3D) reconstruction of ship shell plates. Measurement experiments are then performed for three
typical hull plates to evaluate the accuracy, efficiency and completeness of the proposed measurement method.
The results suggest that the complete point cloud data of the curved hull plates can be reconstructed in about 3
min with the average of errors less than 1 mm, which fulfills the requirements of precision and efficiency in
shipbuilding production. Compared with traditional wooden templates, the proposed measurement method is
more accurate, efficient and inexpensive. The developed measurement system with quantitative data can also be
readily integrated with the 3D computer numerical control (CNC) plate bending machine. Moreover, the
robustness and flexibility of the proposed measurement method have been verified by comparison with the
measurement method based on active binocular stereovision.

1. Introduction technologies, a large-size plate bending machine SKWB-2500 with 3D

Ship shell plates usually have complicated three-dimensional (3D)
curved surfaces, especially at the bow and stern (Mitsuyuki et al., 2020).
Manufacturing of ship shell plates has been carried out for many years
by line heating (Yoshihiko et al., 2011; Park et al., 2016) combined with
the rolling forming process, which is heavily dependent on the experi­
ence of the workers. As a result, the production efficiency would be
restricted. An alternative method is the mechanical cold-bending tech­
nique (Munro and Daniel, 2007; Hwang and Lee, 2018), which utilizes a
reconfigurable pin-type tool for forming. Curved hull plates with
different geometries can be flexibly manufactured by using a set of
punches, which is able to save time and reduce cost. Wang et al. (2010)
and Wang et al. (2015) invented a curved surface forming device for
adjustable segmented mold board with square rams, as shown in Fig. 1
(a), to eliminate dimples and wrinkles during stamping process of ship
shell plates. Subsequently, a step by step approximation method was
introduced to effectively solve the spring-back problem of hull plate cold
molding forming work with multi-press head (Yuan et al., 2012; Su
et al., 2015; Neven et al., 2019). Along with a breakthrough of key
computer numerical control (CNC) was developed for shipbuilding, as
illustrated in Fig. 1 (b). The effectiveness and reliability of the
cold-bending process have been verified through enormous pressing
practices of ship shell plates (Xiong et al., 2014; Yuan et al., 2014).
Presently, 3D CNC plate bending machines have been put into use in a
couple of shipyards, as indicated in Fig. 2. In order to determine whether
the formed shape satisfies the design requirements as well as to calculate
the springback of a plate, the digital detection of curved hull plates is
extremely essential in the process of cold-bending forming. However,
the existing measuring tools adopted by 3D CNC plate bending ma­
chines, as illustrated in Fig. 3, have greatly restricted the forming quality
and efficiency of ship shell plates. Therefore, development of an accu­
rate, efficient and flexible digital inspection system is urgently required
for the cold-bending forming of ship shell plates.
Since the automated and intelligentized fabrication process of ship
shell plates is hampered by traditional inspection methods with wooden
templates (Shen et al., 2017; Son et al., 2020), development of a digital
inspection system has received increasing attention. Park et al. (2007)
used a portable coordinate measuring machine (CMM) device to

* Corresponding author. Key Laboratory of High Performance Ship Technology (Wuhan University of Technology), Ministry of Education, Wuhan, 430063, China.
E-mail address: y.hu@163.com (Y. Hu).

https://doi.org/10.1016/j.oceaneng.2022.111968
Received 10 December 2021; Received in revised form 1 June 2022; Accepted 8 July 2022
Available online 18 July 2022
0029-8018/© 2022 Elsevier Ltd. All rights reserved.
P. He et al.
Fig. 1. (a) adjustable segmented mold board with square rams, (b) 3D CNC
plate bending machines SKWB-2500.
Fig. 2. Examples of 3D CNC plate bending machines for different models.
Fig. 3. Existing measuring tools: (a) wooden templates, (b) the magnetostric­
tive displacement transducer, (c) the 3D point-by-point laser scanner.
measure curved hull plates manually. The CMM device can cover a range
of 2.8 × 2.8 m, and its measurement error is within ±0.068 mm.
However, such point-by-point and contact measurement is quite ineffi­
cient. Shin et al. (2004) constructed a three-axis line-scanning system to
collect the measured data of large-scale hull plates. However, only a set
of 3D points in a line can be acquired at one time and repetitious work of
data processing must be done during the scanning process. Inspired by
the wooden templates, Heo et al. (2008) proposed a measurement
method by using a fixed four-line structured light on the target surface
for checking the shape of curved hull plates. Hiekata et al. (2011)
applied a laser scanner to measure a curved shell plate, but this process
requires a very high level of manual efforts. Paoli and Razionale (2012)
combined a robotic system with an optical scanner to measure an
as-built whole hull surface. But the adjacent scans require the over­
lapping areas at least 20%, which significantly reduces the measurement
efficiency. Lee et al. (2013) developed a measurement system for curved
hull plates with multi-slit structured light. Although the efficiency has
been increased, the convenience and flexibility of the measurement
system are restricted since the camera calibration has to be performed
regularly. An active binocular stereovision sensor was developed by
Wang et al. (2016) for onsite inspection of the formed hull plates. The
sensor can cover a range of 2.8 × 2.3 m, and the total time of 3D
reconstruction is less than 5 s. However, the projection of circular spot
array is highly susceptible to illumination conditions, and the extrinsic
parameters of cameras must be calibrated repeatedly due to the negative
impact of vibrations in the working environment. In order to reduce the
systematic errors of prototype ship model measurement, Mohd et al.
(2017) investigated the suitability of an on-site calibration method for
application in high-accuracy shipbuilding. The maximum errors are 6
mm and 4 mm for the tsunami fisherman boat and for catamaran hull
2
Ocean Engineering 260 (2022) 111968
vessel, respectively. As a result, the accuracy requirement for the hull
plate measurement cannot be fulfilled. In addition, the multi-node ro­
tary laser positioning system can be applied to measure ship shell plates.
The system is a large-scale coordinate measuring system based on the
rotary-laser scanning technique. Xue et al. (2017) explored the rela­
tionship between the measuring space and the accuracy of the
multi-node rotary laser positioning system by applying ill-posed mea­
surement theory. The analysis provides a guidance to optimum appli­
cation of the multi-node rotary laser positioning system in shipbuilding.
Wei et al. (2019) applied an image-based 3D reconstruction technology
to measure ship blocks. It is proved by the shipyard experiments that the
proposed method satisfies the requirements of accuracy and efficiency in
shipbuilding industry.
Nowadays, the digital inspection technologies have been continu­
ously developed and applied to inspection of vessels (Bonnin-Pascual
and Ortiz, 2019). In particular, multi-view stereo (MVS) methods have a
great potential in measurement of ship shell plates due to their flexi­
bility, high accuracy, large measurement range and low cost. MVS aims
to recover the dense representation of the scene given multi-view images
and calibrated cameras (Furukawa and Hernandez, 2015). Although
traditional MVS methods have shown acceptable results under ideal
Lambertian scenarios, they suffer from some common limitations. For
example, low-textured, specular and reflective regions of the scene make
dense matching intractable and thus lead to incomplete 3D re­
constructions. Fortunately, the success of Convolutional Neural Net­
works (CNN) in the field of computer vision provides the possibility that
data driven models can solve some of these issues that classical MVS
models struggle with. Yao et al. (2018) first presented an end-to-end
deep learning architecture for 3D reconstruction, denoted as MVSNet.
MVSNet flexibly takes unstructured N-view images as input, and infers
the depth map for the reference image in an end-to-end fashion. It has
been demonstrated on DTU dataset (Aanæs et al., 2016) that MVSNet
produces the complete point clouds especially in those textureless and
reflected areas, which are commonly considered as the most difficult
parts to recover in MVS reconstruction. However, one major limitation
of current MVSNet is the scalability: the memory-consuming cost vol­
ume regularization makes the MVSNet hard to be applied to high reso­
lution scenes (Yao et al., 2019). Gu et al. (2020) proposed a memory and
time efficient cost volume formulation complementary to the existing
MVS approach, named as CasMVSNet. CasMVSNet obtains a 35.6%
improvement on DTU benchmark, with 50.6% and 59.3% reduction in
GPU memory and run-time. Therefore, CasMVSNet will be adopted to
3D reconstruction of curved hull plates in this work.
Although the learning-based MVS methods have made great progress
in academia for the past few years, their industrial applications are still
relatively few. Especially, their applications in the digital inspection of
shipbuilding industry have not been reported in public literatures so far.
This work aims at developing a robust and flexible MVS measure­
ment system for ship shell plates by applying a state-of-the-art deep
learning network architecture, CasMVSNet proposed by Gu et al. (2020).
The measurement experiments of three real hull plates with different
sizes and shapes are performed to evaluate the measuring accuracy,
efficiency and completeness. The results show the potential that the
learning-based MVS reconstruction can be applied for the automation of
the cold bending process at the shipyard. It is worth noting that this
work first applies a deep-learning based MVS approach for measurement
of ship shell plates, which laid a solid foundation for the application of
this technology in the cold-forming process.
2. Methodology
The architecture of the cascade cost volume on MVSNet is proposed
by Gu et al. (2020), denoted as CasMVSNet. The full architecture of
CasMVSNet is visualized in Fig. 4. The flow chart of CasMVSNet algo­
rithm mainly includes four steps as follows.
P. He et al. Ocean Engineering 260 (2022) 111968

Fig. 4. Network architecture of CasMVSNet proposed by Gu et al. (2020).

(1) Firstly, taking a single-scale image of an arbitrary size as the
input, Feature Pyramid Network (Lin et al., 2017) outputs pro­
portionally sized feature maps at three levels.
(2) Secondly, N feature maps at the first level are warped into
different fronto-parallel planes of the reference camera in order
to form N feature volumes via the differentiable homography
warping. The variance cost metric is then used to aggregate N
feature volumes to one cost volume. After that, the cost volume
regularization is designed for refining the above cost volume to
generate a probability volume for depth inference. In the end, the
initial depth map at the first stage is regressed from the regular­
ized probability volume. This part is migrated from MVSNet (Yao
et al., 2018).
(3) Thirdly, the hypothesis depth range is narrowed based on the
predicted output from the previous stage, and finer hypothesis
plane intervals are applied to recover more detailed outputs. The
next step is same as step (2) in order to output a higher resolution
depth map.
(4) Finally, step (3) is repeated at the third stage in order to narrow
the hypothesis depth range and hypothesis plane interval again,
and the final depth map is acquired.
2.1. Feature extraction
The first step of CasMVSNet is to extract the deep features of the N
input images at multiple levels. Here, Feature Pyramid Network (FPN)
(Lin et al., 2017) is applied to obtain feature maps at three levels. The
construction of FPN involves a bottom-up pathway, a top-down pathway
and lateral connections, which are designed for building feature maps at
multiple scales. The bottom-up pathway computes the feedforward of
conv0, conv1 and conv2 in Table 1, which generates feature maps at
three scales. The conv0, conv1 and conv2 are the basic blocks consisting
of two or three 2D CNNs for learning the feature extraction. The
top-down pathway adopts the nearest neighbor interpolation with a
scale factor of 2 in order to produce higher-resolution feature maps from
spatially coarser, but semantically stronger, feature maps at higher
pyramid levels. These upsampling feature maps are then merged with
the corresponding bottom-up feature maps by element-wise addition.
Lateral connections utilize a 1 × 1 convolutional layer to match the
channel dimensions of feature maps from the bottom-up and top-down
pathways.
The total thirteen-layer 2D CNNs are applied, where the strides of
Conv2d_3 and Conv2d_6 in Table 1 are set to two to divide the feature
maps to three scales. Within each scale, two or three convolutional
layers are applied to extract the higher-level image representation. The
outputs of FPN are the feature maps at three levels, viz. output1, output2
and output3 in Table 1. Their correspondent spatial resolutions at three
levels are 1/16, 1/4, 1 of the input image size, respectively. Detailed
parameters of FPN can be found in Table 1. The conv2d, upsampling and
addition refer to a layer 2D CNN operation, the interpolation operation
and the element-wise addition operation respectively, while conv2d_x,
upsampling_x and addition_x represent their middle outputs after the xth
according operations. The in_chs, out_chs, kernel_size, stride and padding
are parameters of each layer 2D CNN. H and W denote the height and
width of the input image, respectively.
2.2. Cost volume
After the deep feature maps {Fi }Ni=1 at three scales are extracted from

Table 1
Parameters of feature pyramid network.
Output Middle output Input in_chs out_chs kernel_size stride padding Output dimension

conv0 conv2d_1 image 3 8 3×3 1 1 H×W×8

conv2d_2 8 8 3×3 1 1 H×W×8
conv1 conv2d_3 conv0 8 16 5×5 2 2 1/2H × 1/2W × 16
conv2d_4 16 16 3×3 1 1 1/2H × 1/2W × 16
conv2d_5 16 16 3×3 1 1 1/2H × 1/2W × 16
conv2 conv2d_6 conv1 16 32 5×5 2 2 1/4H × 1/4W × 32
conv2d_7 32 32 3×3 1 1 1/4H × 1/4W × 32
conv2d_8 32 32 3×3 1 1 1/4H × 1/4W × 32
output1 conv2d_9 conv2 32 32 1×1 1 0 1/4H × 1/4W × 32
output2 upsampling_1 conv2 nearest neighbor interpolation with a scale factor of 2 1/2H × 1/2W × 32
conv2d_10 conv1 16 32 1×1 1 0 1/2H × 1/2W × 32
addition_1 upsampling_1 conv2d_10 element-wise addition 1/2H × 1/2W × 32
conv2d_11 addition_1 32 16 3×3 1 1 1/2H × 1/2W × 16
output3 upsampling_2 addition_1 nearest neighbor interpolation with a scale factor of 2 H × W × 32
conv2d_12 conv0 8 32 1×1 1 0 H × W × 32
addition_2 upsampling_1 conv2d_10 element-wise addition H × W × 32
conv2d_13 addition_2 32 8 3×3 1 1 H×W×8

3
P. He et al. Ocean Engineering 260 (2022) 111968

N input images through FPN, the next step of CasMVSNet is to warp all
feature maps {Fi }Ni=1 into different fronto-parallel planes of the reference
camera frustum by differentiable homographies to build N feature vol­
umes {Vi }Ni=1 in 3D space. CasMVSNet adopts a cascade differentiable
homography warping, whose homography at the (s + 1)th stage is
( (
) )
( ) t1 − ti ⋅nT1
Hi dsk + Δks+1 = Ki ⋅ Ri ⋅ I − ⋅ RT1 ⋅Κ−1 T (1)
dsk + Δks+1
where Hi (dks + Δks+1 ) refers to the homography between the ith feature
map and the reference feature map at depth d = dks + Δks+1 . dks denotes the
predicted depth of the kth pixel at the sth stage, and Δks+1 is the residual
depth of the kth pixel to be learned at the (s + 1)th stage. Moreover, Ki,
Ri, ti are the camera intrinsics, rotations and translations of the ith
source view respectively. K1, R1, t1 are the camera intrinsics, rotations
and translations of the reference view respectively. n1 denotes the
principle axis of the reference camera. I is the identity matrix.
Next, CasMVSNet aggregates multiple feature volumes {Vi }Ni=1 to one
cost volume C via a variance cost metric. The cost volume C is defined as:
∑
N the MVS measurement system of a curved hull plate, as demonstrated in
(Vi − Vi )2
C = i=1 (2)
N
where Vi is the average volume among all feature volumes, and all op­
erations above are element-wise.
Subsequently, CasMVSNet regularizes the cost volume C using the
multi-scale 3D CNNs, and regresses a coarse depth map. The later stages
use the estimated depth maps from the earlier stages to adaptively
generate hypothesis planes and construct new cost volumes with higher-
res feature maps. This adaptive depth sampling and adjustment of
feature resolution ensures the computation and memory resources are
spent on more meaningful regions (Gu et al., 2020). Thus, the compu­
tation time and GPU memory consumption can be decreased remarkably
via the coarse-to-fine pattern.
2.3. Loss function
The cascade cost volume with N stages produces N − 1 intermediate
outputs and a final prediction. The supervision is applied to all of the
outputs, and the total loss (Gu et al., 2020) is defined as:
∑
N remotely adjusted in order to make hull plates of different sizes kept in
Loss = λs ⋅Ls (3)
s=1
source code and the pre-trained model of CasMVSNet provided by Gu
et al. (2020) is available at https://github.
com/alibaba/cascade-stereo/tree/master/CasMVSNet. Although the
model is trained on a DTU indoor dataset, CasMVSNet without any
fine-tuning is still able to produce the excellent reconstructions on the
more complex outdoor scenes, demonstrating the strong generalization
ability of the network. Thus, it could be feasible to use the pre-trained
model of CasMVSNet for 3D reconstruction of curved hull plates in the
absence of the curved hull plates dataset. It is worth noting that each
image from different views takes turns to be the reference image to es­
timate depth maps. Moreover, CasMVSNet adopts a three-stage cascade
cost volume and two neighboring images as source images during
training.
3. Measurement system
Theoretically, multi-view images of a hull plate can be directly taken
manually. However, this mode of image acquisition is quite inefficient,
and the quality of the collecting image is easily affected by human fac­
tors. In order to improve the efficiency and quality of images acquisition,
Fig. 5, is established to realize automatic acquisition and synchronous
transmission of images. It mainly consists of a digital single lens reflex
(DSLR) camera to acquire high-resolution images and larger field of
view (FOV), a motorized camera slider to improve efficiency of image
acquisition, a metal test frame to simulate the working condition and a
laptop to process data. The main details of hardware will be presented as
follows.
3.1. The DSLR camera
Canon EOS 90D DSLR camera matched with a Canon EF-S 18–135
mm f/3.5–5.6 IS USM lens is selected, as shown in Fig. 6 (a) and 6 (b).
The ordinary consumer-grade DSLR camera is served as the vision sensor
in this study, mainly because it has the advantages of high resolution,
low cost and flexibility compared with industrial cameras. The camera
body adopts a complementary metal-oxide semiconductor (CMOS) with
about 6960 × 4640 pixels, and each pixel size is around 3.20 × 3.20 μm.
The communication function supports Wi-Fi, Bluetooth and USB 2.0.
The camera lens has a focal length of 18–135 mm. Canon Power Zoom
Adapter PZ-E1, as indicated in Fig. 5 (c), is used to achieve a stable and
smooth power zoom drive. By this device, the camera focal length can be
the appropriate proportion in the image.

where Ls refers to the loss at the sth stage and λs refers to its corre­
sponding loss weight. The smooth L1 loss (Girshick, 2015) is adopted for
Ls at each stage.

2.4. Implementations

DTU (Aanæs et al., 2016) is a large-scale MVS dataset consisting of

124 different scenes scanned in 7 different lighting conditions at 49 or
64 positions. The DTU training set is used to train the model, and the
trained model is tested on DTU evaluation set. During the training
process, Adam optimizer with β1 = 0.9 and β2 = 0.999 is adopted. The
training is done for 16 epochs with an initial learning rate of 0.001,
which is downscaled by a factor of 2 after 10, 12, and 14 epochs.
CasMVSNet is implemented by using PyTorch (Paszke et al., 2019), and
the network is trained with 8 Nvidia GTX 1080Ti GPUs with 2 training
samples on each GPU (Gu et al., 2020). More details and implementa­
tions of CasMVSNet can be found in literatures (Lin et al., 2017; Yao
et al., 2018; Gu et al., 2020).
In the present work, the pre-trained model provided by Gu et al.
(2020) is adopted for the 3D reconstruction of curved hull plates. The Fig. 5. MVS measurement system for a curved hull plate.

4
P. He et al.
Fig. 6. The camera body, lens and power zoom adapter PZ-E1.
3.2. Motorized camera slider
As shown in Fig. 7, the adopted camera slider is ASHANKS C300S
Smooth ONE, which is mainly composed of two main sliding rails, a
central guide sliding rail, an electronic control unit, a cable release, a
sliding block, a belt and four corner brackets. The main slide rail is made
of carbon fiber material with a total length of 1200 mm and a maximum
load of 8 kg. In fact, with the limitation from the size of the sliding block,
the movable length of the sliding block on the camera slider is approx­
imately 1100 mm. The total weight of the camera body, lens, and power
zoom adapter is much smaller than the load-bearing capacity of the
camera slider. The guide sliding rail is located between the two main
sliding rails. Different shooting modes are obtained by adjusting its left
and right ends. When collecting the images of a curved hull plate, the
follow focus mode should be adopted so that a target hull plate is always
kept in the center of the FOV. Follow-focus mode means that the camera
rotates while translating along two main sliding rails to achieve the
function of shooting around a fixed object. In this mode, the positions of
both ends of the guide sliding rail are “left front and right rear”, as
illustrated in Fig. 8. The electronic control unit is connected to the
application program (APP) in the mobile phone or iPad via Bluetooth,
which can remotely control the sliding block through APP.
Fig. 7. The camera slider.
5
Ocean Engineering 260 (2022) 111968
Fig. 8. Schematic diagrams of follow-focus mode.
3.3. Metal test frame
In order to simulate the real measuring scene of curved hull plates, a
cubic metal test frame with a size (length × width × height) of 3000 ×
3000 × 2500 mm is set up in the plate bending machine workshop, as
shown in Fig. 5. The sliders are mounted on the top beams of the metal
test frame.
3.4. Effective FOV
In order to acquire a larger measurement range for a single view
under the present hardware conditions, it is necessary to design the
downward tilt angle of the camera lens. On the basis of the ideal pinhole
model, it can be estimated that when the focal length is 18 mm and the
angle between the optical axis of the camera and the horizontal line is
65◦ , a single view of the camera can cover a largest measurement area.
The effective FOV of MVS measurement system is composed of
overlapping FOV in different views. Since the camera takes pictures in
the different views, the FOV of the camera is different in various posi­
tions of slider. CAD software is used to simulate the overlapping FOV of
the camera in different views, and the result is shown in Fig. 9. When the
camera moves from left to right, the range of the camera rotation angle is
[11◦ , − 11◦ ]. The rotation angle of the camera at the center of the slider
is defined as 0◦ , and turning to the right is positive. The final effective
FOV is about 2140 × 2140 mm. This effective FOV can measure all hull
plates formed by SKWB-800, SKWB-1800 and SKWB-2000 and most of
hull plates formed by SKWB-2500.
Nevertheless, for the SKWB-2500, the effective FOV of the mea­
surement system should be larger than its processing area of 2500 ×
2500 mm. Larger effective FOV could be obtained by increasing the
working distance or by reducing the focal length of the optical devices.
The first choice would be limited by the maximum working determined
Fig. 9. Effective FOV of the MVS measurement system.
P. He et al.
by the shipyard workshop. Alternatively, the other two lenses with a
shorter focal length, EF-S 10–22 mm f/3.5–4.5 USM or EF-S 10–18 mm
f/4.5–5.6 IS STM, can be used for the SKWB-2500, which can acquire a
larger effective FOV at the same working condition.
4. Measurement objects
Three typical hull plates with different sizes and shapes are selected
as measuring objects in this study. They are formed by 3D CNC plate
bending machine SKWB-2500 shown in Fig. 1. The dimensions (l1, l2, l3,
l4) of the small-size hull plate, as shown in Fig. 10 (a), are about 837 mm,
844 mm, 838 mm and 841 mm, respectively. Its thickness is about 5 mm.
This kind of curved hull plate is called the saddle-shaped plate. Fig. 10
(b) shows that the dimensions (l1, l2, l3, l4) of the middle-size hull plate,
named as the sail-shape plate, are about 635 mm, 1394 mm, 1846 mm
and 1501 mm with the thickness of around 20 mm. Its maximum
bending camber (hmax) is around 333 mm. As illustrated in Fig. 10 (c),
the dimensions (l1, l2, l3, l4) of the large-size hull plate are about 1873
mm, 1850 mm, 2355 mm and 1855 mm, whose thickness is about 20
mm. All of three hull plates belong to the doubly curved surface, and
their molded hull surfaces, viz. the upper surfaces as indicated in Fig. 10,
will be measured.
5. Measurement process
Fig. 11 shows an overview of measuring process for a hull plate,
mainly including 3D reconstruction, measurement of approximate
theoretical values and point clouds registration.
5.1. The 3D reconstruction of a curved hull plate
The flow of 3D reconstruction for a curved hull plate is illustrated in
Part I of Fig. 11. Firstly, arbitrary multi-view images of a curved shell
plate are automatically taken by the DSLR camera fixed on a slider and
synchronously transmitted to a computer by USB cable. Secondly, the
camera parameters of each image and all undistorting images are ob­
tained from COLMAP’s Structure-from-Motion (SfM) (Schonberger and
Frahm, 2016) outputs, which generates the CasMVSNet inputs together
with sparse point cloud data. Thirdly, the depth maps and dense point
cloud data are acquired by CasMVSNet. Finally, the noise points of the
dense point cloud are removed. In the following contents, the 3D
reconstruction process is elaborated by taking the sail-shaped plate in
Fig. 10 (b) as an example.
Fig. 10. The in-kind images of three curved hull plates.
6
Ocean Engineering 260 (2022) 111968
5.1.1. Images acquisition and transfer
The time lapse mode is applied for images acquisition of a hull plate.
This mode means that the sliding block runs when the shutter is closed,
whereas the sliding block stops when the shutter is exposed, viz.
moving-stopping-shooting. At this mode, the positions of the starting
point (POINT A) and the end point (POINT B), the shooting time
(Tshooting), stabilizing time (Tstabilizing), and the number of the shooting
image (N) should be set in the APP. Tshooting refers to the time taking one
image, including the camera shutter exposure time and automatic focus
time. Generally, the shutter exposure time should be set in the camera
according to the shooting scene. The shutter exposure time is set to 0.01
s when shooting images of the hull plate. Tstabilizing refers to the time
waiting for the camera and lens to dead stop in order to prevent them
from shaking during shooting. The stabilizing time is generally set to be
0.5–2 s. N refers to the number of images taken in a single journey from
POINT A to POINT B. Therefore, the total time (TIAT) taking all images is
TIAT = N × (Tshooting + T stabilizing)+Ttotal moving (4)
where Ttotal moving is total moving time of the sliding block from POINT A
to POINT B.
The specific process of automatic collection of images for a curved
hull plate is as follows. First, a hull plate is placed within the measure­
ment area of the metal frame, using a crane. Second, the PZ-E1 is used to
remotely adjust the focal length of the lens in order to obtain the
appropriate proportion of the different hull plates in the images. It is also
necessary to ensure that the hull plate is entirely covered by the FOV of
the camera in the different positions of the slider. Next, the sharpness of
the image should be adjusted to the optimum by focusing. Finally, the
start and end positions as well as the moving speed of the sliding block
are set, and the shooting time, stabilizing time and the number of images
acquisition are specified in the time lapse mode.
Generally, the transfer of images stored in the secure digital memory
card is a simple manual file transfer from the camera to the computer.
However, this step requires user interaction and it is generally time-
consuming. Therefore, further improvement is required. Ideally, the
images taken should be synchronously transferred to the computer
without any user interaction and extra transfer time. Fortunately, EOS
90D can be connected to the EOS Utility software provided free of charge
by Canon through a USB cable or WiFi, which can make the images
taken by the camera synchronously transferred to a designated folder on
a computer. Images of the sail-shape plate acquired by the sensor in four
different directions are presented in Fig. 12.
5.1.2. Sparse 3D reconstruction
The input folder of CasMVSNet should contain the image files,
camera files and view selection file. All undistorted image files are
stored in the images folder. The camera parameter of one image is stored
in a cam.txt file. The text file contains the camera extrinsic E = [R|t],
intrinsic K and the depth range. The depth range and depth resolution
are determined by the minimum depth, the interval between two depth
samples, and the depth sample number. The view selection result is
stored in the pair.txt. For each reference image, its view selection scores
are calculated by each of the other views, and the 10 best views are
stored in the pair.txt file.
In order to apply CasMVSNet to reconstruct the curved hull pate, the
sparse 3D reconstruction should be conducted for obtaining undistorted
images, camera parameters and sparse 3D point cloud data. Here, the
open source software COLMAP’s incremental SfM (Schonberger and
Frahm, 2016) is adopted to recover a sparse representation of the scene
and the camera poses of the input images. It commonly starts with
feature extraction and matching, followed by geometric verification.
The resulting scene graph serves as the foundation for the reconstruction
stage. The sparse reconstruction stage seeds the model with a carefully
selected two-view reconstruction, before incrementally registering new
images, triangulating scene points, filtering outliers, and refining the
P. He et al.
Fig. 11. Overview of measuring process for a hull plate.
reconstruction using bundle adjustment.
The specific four steps to acquire the CasMVSNet input are given as
follows:
(1) In the first step, sparse feature points in the image are extracted
by using the scale-invariant feature transform (SIFT) (Lowe,
2004) and their appearance is described by a numerical
descriptor. In this process, an intrinsic camera model of OPENCV
is selected. The camera model includes focal distances, co­
ordinates of the principal points, radial and tangential
distortions.
(2) In the second step, the feature matching and geometric verifica­
tion are adopted to find correspondences between the feature
points in different images. The exhaustive matching mode is
adopted. Since the number of images for the hull plate is rela­
tively low (less than one hundred), this matching mode can be
fast enough and leads to the best reconstruction results. Here,
each image is matched against other images.
(3) After producing the scene graph in the previous two steps, the
incremental reconstruction is conducted. COLMAP first loads all
extracted data from the database into memory and seeds the
reconstruction from an initial image pair. Then, the scene is
incrementally extended by registering new images and triangu­
lating new points. The sparse reconstruction results of the sail-
shape plate are finally obtained in Fig. 13.
(4) After recovering SfM result and undistorting all images, COLMAP
exports the following three text files for the reconstructed model:
cameras.txt, images.txt, and points3D.txt.
The cameras.txt file contains the intrinsic parameters of all recon­
structed cameras in the dataset using one line per camera. The images.txt
7
Ocean Engineering 260 (2022) 111968
file contains the pose and keypoints of all reconstructed images in the
dataset using two lines per image. The reconstructed pose of an image is
specified as the projection from world to the camera coordinate system
of an image by using a quaternion and a translation vector. The points3D.
txt file contains the information of all reconstructed 3D points in the
dataset using one line per point. Eventually, a script colmap2mvsnet.py
provided by MVSNet (Yao et al., 2018) is used to convert COLMAP SfM
result as the input of CasMVSNet.
5.1.3. Generation of depth maps and 3D dense point cloud
CasMVSNet is used to estimate a depth value for every pixel. First,
the cascade cost volume is built upon a feature pyramid encoding ge­
ometry and context at gradually finer scales. The depth range of each
stage is then narrowed by the prediction from the previous stage. With
gradually higher cost volume resolution and adaptive adjustment of
depth intervals, the output is recovered in a coarser to fine manner (Gu
et al., 2020).
Before converting the result to the dense point clouds, it is necessary
to filter out the outliers at the background and occluded areas. Photo­
metric and geometric consistencies are applied for the robust depth map
filtering. The photometric consistency and the geometric constraint
measure the matching quality and the depth consistency among multiple
views, respectively. This simple two-step filtering strategy shows strong
robustness for filtering different kinds of outliers.
CasMVSNet itself only produces per-view depth maps. In order to
generate 3D point cloud, a depth map fusion should be carried out to
integrate depth maps from different views. The fused depth maps are
then directly reprojected to space in order to generate 3D point cloud.
The depth map fusion is a general step in MVS reconstruction, and there
are commendable implementations in the open-source MVS algorithms.
Here, a script depthfusion.py provided by MVSNet (Yao et al., 2018) is
P. He et al. Ocean Engineering 260 (2022) 111968

Fig. 12. Images of the sail-shape plate acquired by the sensor in four different directions.

Fig. 13. The sparse 3D reconstruction result of the sail-shape plate

with COLMAP.

Fig. 14. The dense reconstruction result of the sail-shape plate.

8
P. He et al.
adopted to generate 3D dense point cloud with RGB color and normal
vector information, as shown in Fig. 14.
5.1.4. Point cloud data processing
In the stage of current feasibility study, in order to acquire the dense
representation of a molded surface as shown in Fig. 15, the open source
software CloudCompare is used to manually remove the noise points of
the background far away from the hull plate, the noise points at the four
edges, as well as the outliers near the hull plate. In the future study, the
noise points is expected to be removed automatically via the Point Cloud
Library, whose main difficulty is to remove the noise points at the edges
of the hull plate.
5.2. Measurement of hull plates by Metronor One
In this work, the sparse point cloud data of a hull plate are obtained
by Metronor One, whose process of measurement can be seen in Part II of
Fig. 11. Metronor One, as shown in Fig. 16, is a single camera electro-
optical portable coordinate measuring system. It uses electro-optics
and photogrammetry for accurately calculating where the handheld
probe, called Lightpen, is located both in terms of position and orien­
tation. The unique highlight of Metronor One is that it uses an optical
technology based on a special mathematical optical modelling method.
The measuring system mainly includes a camera with tripod, a Lightpen,
a Laptop, the software and transportation case. It has a 3D length ac­
curacy of ±0.08 mm as well as parallelism and planarity accuracies of
±0.025◦ within a 2.5 × 2.5 × 2.5 m3 volume (Metronor, 2019). More­
over, it can measure a whole hull plate formed by the bending machine
SKWB-2500 with high precision. With an integrated battery and WiFi
pack, the Lightpen is a true cable-less, joint-less design that gives users
full flexibility to move and measure in the working area. However, the
measuring efficiency is not high due to such point-by-point and contact
measurement mode. Nevertheless, it still serves as a powerful device for
verifying the accuracy of the MVS reconstruction in this study. Fig. 17
presents 3D points of three curved hull plates measured by Metronor
One. They are considered accurate enough because the current
measuring error with wooden templates is 1–2 mm according to
workers’ experience. Therefore, these 3D points can be regarded as the
theoretical values of the molded surface of a hull plate with negligible
errors.
In the present study, 3D points measured by Metronor One are served
as approximate theoretical values to compare with the results of MVS
reconstruction. But it should be noted that in practical applications, the
Fig. 15. The dense representation of a molded surface of the sail-shape plate
after denoising.
9
Ocean Engineering 260 (2022) 111968
Fig. 16. In-kind photo of Metronor One.
design values of a curved hull plate are compared with the measurement
result to determine whether the shape of a curved hull plate meets the
design requirement. The four red points marked in Fig. 17 (b) corre­
spond to the four markers shown in Fig. 15, which are subsequently used
for point cloud registration.
5.3. Registration and error computation
In this work, the accuracy of measurement is assessed by computing
the proximity between the two aligned point clouds. The accuracy
evaluation process of 3D reconstruction for a curved hull plate is
sketched out in Part III of Fig. 11. First, the dense point cloud and sparse
point cloud are preliminarily registered by four corner points or markers
of a curved hull plate. Then, they are finely registered by an iterative
closest point (ICP) algorithm proposed by Besl and McKay (1992).
Finally, the gaps between two point cloud data are estimated by calcu­
lating the absolute displacement error, and the results are presented by
the error histogram and distribution. 3D Point cloud registration and
accuracy evaluation are performed in the source software CloudCom­
pare, whose key processes will be expounded below.
5.3.1. Scale match
Since there is a mismatch between scales of sparse and dense point
clouds, the scales of two selected clouds should be matched firstly. Full-
scale sparse point cloud measured by Metronor One is selected as the
reference entity. The scale is then computed via principal dimensions
deduced by principal component analysis. On completion, the dense
point cloud will be scaled to full size.
5.3.2. Centers of gravity match
If two point clouds are far apart, select both of them and use the
centers of gravity match method to make their centers of gravity coin­
cide. The first selected point cloud will be used as reference. The other
will be translated so that their centers of gravity come at the same place
as that of the first one.
5.3.3. Alignment and registration
The sparse and dense point clouds are aligned by picking four
equivalent point pairs in both clouds. This method is useful to align
clouds with corners or markers. The reference cloud (will not move) and
the aligned cloud (the one actually moving at the end) should be
designated. Here, the sparse point cloud is still selected as the reference.
P. He et al.
Fig. 17. 3D points measured by Metronor One.
This method can also allow the optimization of the scale parameter if the
two clouds have different scales.
After both clouds are already roughly registered, the ICP algorithm is
adopted to automatically and finely register sparse and dense clouds
representing the same hull plate. The sparse point cloud and the dense
point cloud are specified as the registered cloud (will eventually move)
and the reference cloud (will not move), respectively. Fig. 18 presents
the alignment and registration for the sail-shape plate.
5.3.4. Error computation
The distances between each point of the compared cloud relatively to
the reference cloud are computed to assess the measuring accuracy. The
reference cloud should have the widest extents and the highest density.
Therefore, the dense point cloud is selected as the reference cloud, while
the sparse point cloud is selected as the compared cloud. The octree
structure, corresponding to the recursive partition of a cubical volume of
space, is applied to increase efficiency for the nearest neighbor
extraction.
One way to compute the distances between two point clouds (i.e., the
cloud-cloud distances) is the nearest neighbor distance: for each point of
the compared cloud, the nearest point in the reference cloud is searched
via the octree structure, and the Euclidean distance is computed. If the
reference point cloud is dense enough, it is acceptable to approximate
the distance from the compared cloud to the underlying surface repre­
sented by the reference cloud. In fact, the nearest neighbor is not
necessarily the actual nearest point on the surface represented by the
cloud. This is especially true if the reference cloud has a low density or
10
Ocean Engineering 260 (2022) 111968
has big holes. Therefore, it can be a good idea to locally model the
reference cloud underlying surface by fitting a local model on the
‘nearest’ point and several of its neighbors in order to approximate the
real surface as well as get a better estimation of the ‘real’ distance, as
shown in Fig. 19. The distance to this local model is statistically more
precise. However, some distance values can be potentially worse than
the nearest neighbor distance. In order to cope with this effect, the
smaller distance between the nearest neighbor distance and the distance
to the local model is kept for each point in this work. Eventually, these
smaller distances are defined as the displacement error of cloud-cloud
distances.
6. Results
All reconstructed tasks of curved hull plates in this study are
executed on a linux operating system of an ubuntu 18.04 release using
one consumer level NVIDIA GeForce GTX 1660 SUPER graphics card
with a video memory capacity of 6 GB in combination with AMD Ryzen
9 3950X 16-core processors with random access memory (RAM) of 32
GB.
6.1. Accuracy, efficiency and completeness
Table 2 presents the influence of the image number and image res­
olution on the efficiency and accuracy of 3D reconstruction for the sail-
shaped plate, where IN, IR and NPC denote the image number, the image
resolution and the number of the point cloud after denoising,
P. He et al. Ocean Engineering 260 (2022) 111968

respectively. TIAT, TSR, TDR and TTR refer to time of image acquisition and
transfer, time of sparse 3D reconstruction, time of dense 3D recon­
struction and total time of 3D reconstruction, respectively. Average of
errors, standard deviation (SD) of errors and the maximum displacement
error are indicated by μ, σ and Dmax, respectively. They are computed
from a grouped frequency distribution:
/
∑
m
μ = fi xi n,
i=1
√̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅̅
√ m
√∑
)2 / (5)
σ=√ fi (xi − μ (n − 1),
i=1
Dmax = max(D1 , D2 , ..., Di , ..., Dn )

Fig. 18. Alignment and registration for the sail-shape plate: (a) alignment with
four markers, (b) fine registration with the ICP algorithm.
Fig. 19. Illustration of cloud-cloud distances computation.
where xi and fi are the midpoint value and frequency of each interval,
respectively. m and n refer to the total number of intervals and samples,
respectively. Di denotes the displacement error of the ith sample. For
simplicity, four image resolutions of 6960 × 4640 pixels, 4800 × 3200
pixels, 3472 × 2320 pixels and 2400 × 1600 pixels are represented by
high resolution, medium resolution, low resolution and minimum res­
olution, respectively.
The time of image acquisition and transfer (TIAT) is computed by
equation (4) with Tshooting = 2.5 s, T stabilizing = 1.5 s and Ttotal moving =
30 s for all of the cases. Four cameras are mounted on the four sides of
the top of the metal test frame to take pictures simultaneously in prac­
tical applications. Therefore, the time to collect and transmit images for
four cameras is equal to the time taken by any one of them. The time of
sparse 3D reconstruction (TSR) mainly includes the time for feature
extraction, feature matching and SfM computation. The time of dense
3D reconstruction (TDR) comprises the time for undistorting images,
transforming data format and generating depth maps as well as 3D dense
point cloud. The total time of 3D reconstruction (TTR) is the sum of TIAT,
TSR, and TDR. In order to reduce the influence of the noise points around
the edges of a curved hull plate on the accuracy evaluation, only the
approximate theoretical points within the molded surface are used for
the error statistical analysis. As illustrated in Fig. 17, the small-size
plate, the middle-size plate and the large-size plate have 169, 81 and
88 approximate theoretical points within the molded surface,
respectively.
Table 2 indicates that as the number of images decreases from 52 to
20 at the high resolution, the efficiency of 3D reconstruction is improved
by more than 2.5 times, while its accuracy presents a slight decline.
However, when the number of images decreases from 20 to 12, μ and
Dmax increase from 0.422 mm to 0.864 mm and from 1.061 mm to 2.555
mm, respectively. The loss of accuracy is attributed to the incomplete­
ness of 3D reconstruction, as shown in Fig. 20 (d). In addition, as the
image resolution increases gradually by using 20 images, the accuracy of
3D reconstruction is steadily improved, whereas its efficiency is slightly
reduced. Besides, when using 20 high-resolution images, TTR of the sail-
shaped plate is 169 s, and its μ, σ and Dmax are 0.422 mm, 0.271 mm and
1.061 mm, respectively. TTR is within 3 min, and μ as well as Dmax are
both within 1.061 mm. It can also be found that the number of the dense
point cloud is mainly related to the number of images, and the current
four image resolutions have little effect on its number since they are all
larger than specified maximum resolution (1152 × 864 pixels) in

Table 2
3D reconstruction of the sail-shaped plate with various image numbers and resolutions.
IN IR/pixel NPC TIAT/s TSR/s TDR/s TTR/s μ/mm σ/mm Dmax/mm

52 6960 × 4640 10373421 82 92 255 429 0.388 0.272 1.019

32 6960 × 4640 6488650 62 44 164 270 0.419 0.294 1.030
20 6960 × 4640 3872393 50 23 96 169 0.422 0.271 1.061
12 6960 × 4640 1500068 42 11 58 111 0.864 0.550 2.555
20 4800 × 3200 3870123 50 16 80 146 0.457 0.351 1.798
20 3472 × 2320 3870391 50 13 77 140 0.513 0.413 1.848
20 2400 × 1600 3871670 50 10 68 128 0.564 0.395 2.386

11
P. He et al. Ocean Engineering 260 (2022) 111968

Fig. 20. The completeness of 3D reconstruction with different image numbers and resolutions for the sail-shaped plate.

CasMVSNet. Based on the maximum capacity of GPU memory available using 12 high-resolution images. Therefore, according to the accuracy,
in present device, the maximum resolution is specified to save GPU efficiency and completeness of the 3D reconstruction for the sail-shaped
memory consumption of dense reconstruction. Although four image plate, it is suggested that about 20 high-resolution images are used for
resolutions are all downsized to the same maximum resolution to the 3D reconstruction of a hull plate in the present measurement system.
reconstruct a curved hull plate, images with higher resolution still pre­ Table 3 compares the NPC, efficiency and accuracy of 3D recon­
sent higher precision, as shown in Table 2. struction for three curved hull plates with various sizes. It can be found
It is shown in Fig. 20 that when the number of images exceeds 20, the that since the middle-size plate occupies least total pixels with its re­
four resolutions can reconstruct a complete surface of the sail-shaped striction of shape, the NPC of the middle-size plate is far less than that of
plate, which is represented by 3D dense point cloud with RGB color the small-size plate and the large-size plate. Therefore, the TTR of
and normal vector information. However, the four corner areas and the middle-size plate is slightly shorter than the other two plates. But TTR of
complete surface of the sail-shaped plate fails to be reconstructed by the three plates are all about 3 min. Moreover, by comparing the mea­
surement accuracy i.e., μ, σ and Dmax, of three hull plates, the mea­
Table 3 surement accuracy of the small plate is highest among the three hull
3D reconstruction of three curved hull plates with various sizes. plates, since the small plate occupies more pixels per unit area than the
other two plates. In addition, the averages of errors of the small-size
Items Unit Small-size plate Middle-size plate Large-size plate
plate, middle-size plate and large-size plate are 0.215 mm, 0.422 mm
IN 20 20 20
–
and 0.541 mm, respectively. The averages of errors of three hull plates
IR pixel 6960 × 4640 6960 × 4640 6960 × 4640
NPC – 4758094 3872393 5978518 are all within 1 mm.
TIAT s 50 50 50 The histograms of the displacement errors for three curved hull
TSR s 32 23 29 plates are shown in Fig. 21, which assists in understanding the distri­
TDR s 103 96 103 bution of errors. Four colors of blue, green, yellow and red are applied to
TTR s 185 169 182
differentiate various error values from small to large. Fig. 21 shows the
μ mm 0.215 0.422 0.541
σ mm 0.177 0.271 0.405 errors of all points are within 0.871 mm for the small-size plate and
Dmax mm 0.871 1.061 2.008 within 1.061 mm for the middle-size plate. For the large-size plate,

12
P. He et al. Ocean Engineering 260 (2022) 111968

96.59% of all points are within 1.410 mm, but only three points are in
1.410 mm–2.008 mm. In addition, the SD of three hull plates are 0.177
mm, 0.271 mm and 0.405 mm, respectively. The data indicate that the
deviation between measured data of the small-size plate and its mean
value is smallest, followed by that of the middle-size plate, and the de­
viation between measured data of the large-size plate and its mean value
is largest. Since 1–2 mm interspaces between hull plates and wooden
templates is currently tolerable in the evaluation, the adopted
measuring method fulfills the requirement of measurement accuracy for
various hull plates within 2500 × 2500 mm according to the measure­
ment accuracy (μ, σ and Dmax) and error distribution of the three hull
plates.

6.2. Resampling and surface reconstruction

In order to decrease the number of points dramatically, space the

sample points on the whole hull plate evenly as well as to generate a
smooth surface finally, an uniform resampling process should be carried
out for 3D dense point cloud. Uniform resample of 3D dense point cloud
is performed by picking points from the original cloud so that no point in
the output cloud is closer to another point than a specified minimal
distance (termed subsampling space). The bigger subsampling space
leads to the less points kept.
As indicated in Table 4, after the resampling algorithm is carried out,
the total number of 3D points acquired by the 20 high-resolution images
is reduced to about 22,000 points with the subsampling space of 5 mm
for the small-size plate, around 16,000 points with the subsampling
space of 10 mm for the middle-size plate and about 9,000 points with the
subsampling space of 20 mm for the large-size plate, respectively. In
addition, after resampling, the averages of errors of the small-size plate,
the middle-size plate and the large-size plate are 0.324 mm, 0.550 mm
and 0.734 mm, whose losses of accuracy are about 50.70%, 30.33% and
35.67%, respectively. Although the resampling of the dense point cloud
has a little loss of accuracy, the accuracy of resampling points is still
enough for practical application for the small-size plate and middle-size
plate. However, for the large-size plate, the displacement error of a very
small number of points is greater than 2 mm but less than 3 mm. Thus,
the resampling accuracy of the large-size plate requires to be improved
in the future. In order to improve measuring accuracy of the large-size
plate, one effective way is to increase specified maximum resolution
in CasMVSNet, which requires more GPU memory.
In order to visualize the location of measuring errors, the approxi­
mate theoretical points measured by Metronor One are colored with the
values of the displacement error relatively to the resampling point cloud.
The size of the colouring points is then enlarged to 3 times than that of
resampling points. The RGB colors in Figs. 23, 25 and 27 are in line with
Figs. 22, 24 and 26, respectively. Therefore, by combination of Figs. 22
and 23, Figs. 24 and 25 or Figs. 26 and 27, the distribution of errors can
be observed clearly. For example, as indicated in Figs. 22 and 24, the
displacement errors of only 5 points and 4 points, respectively, are
greater than 1 mm and 1.173 mm, respectively, and their positions can

Table 4
Resampling results for three curved hull plates with different sizes and shapes.
Items Unit Small-size Middle-size Large-size
plate plate plate

IN – 20 20 20
IR pixel 6960 × 4640 6960 × 4640 6960 × 4640
NPC – 4758094 3872393 5978518
Comparing points – 169 81 88
Subsampling mm 5 10 20
space
Subsampling – 22037 16435 9229
points
μ mm 0.324 0.550 0.734
Fig. 21. Histograms of the displacement error: (a) the small-size plate, (b)the
σ mm 0.255 0.376 0.584
middle-size plate, (c)the large-size plate. Dmax mm 1.272 1.508 2.751

13
P. He et al. Ocean Engineering 260 (2022) 111968

Fig. 25. The colouring diagram of the displacement error for the middle-
size plate.

Fig. 22. Histogram of the displacement error for the small-size plate.

Fig. 26. Histogram of the displacement error for the large-size plate.

Fig. 23. The colouring diagram of the displacement error for the small-
size plate.

Fig. 27. The colouring diagram of the displacement error for the large-
size plate.

Fig. 24. Histogram of the displacement error for the middle-size plate.

14
P. He et al.
Fig. 28. The results of the surface reconstruction: (a) the small-size plate, (b)
the middle-size plate, (c)the large-size plate.
be easily found in Figs. 23 and 25.
As illustrated in Fig. 28, the surface reconstruction for three curved
hull plates are performed by the surface reconstruction algorithm
(Kazhdan et al., 2006; Kazhdan and Hoppe, 2013). It is found that the
complete and smooth surfaces of three curved hull plates can be
reconstructed with the resampling point clouds.
7. Discussions
In this section, the measurement method proposed in this work is
compared with three typical methods: the inspection method with
wooden templates, the measuring method based on active binocular
stereovision (Wang et al., 2016) and the classical MVS approaches.
Traditionally, wooden templates are manually used to diagnose the
formed quality of a hull pate in shipbuilding industries, as seen in Fig. 3
(a). However, the shortages are obvious for the wooden templates
(Wang et al., 2016). Firstly, several templates are required for each hull
plate, and these templates are not reusable. Therefore, the inspection
method by using wooden templates is costly and wasteful. Secondly, the
shape accuracy of a wooden template is low due to restriction of tradi­
tional, hand-finished craftsmanship. Moreover, the measurement accu­
racy is evaluated only by workers’ experience. Therefore, the inspection
accuracy of wooden templates cannot be guaranteed. Thirdly, during the
forming process of a hull plate, the workers have to put wooden tem­
plates on the formed surface and take them off the formed surface
repeatly. As a result, the inspection process of wooden templates is
time-consuming and labor-intensive. Finally, the inspection method
based on wooden templates cannot provide quantitative values and the
formed accuracy depends heavily on the implicit knowledge of workers.
Therefore, compared with the traditional wooden templates, the mea­
surement method proposed in this work is more accurate, efficient and
inexpensive. Furthermore, the measuring result is quantitative, which
can be seamlessly integrated with 3D CNC plate bending machine.
Alternatively, the measurement method using active binocular ster­
eovision, which adopts a composite structured light of circular spot
array and Gray code, is proposed to reconstruct the 3D model of a hull
plate (Wang et al., 2016). A pattern with circular spot array is used to
generate these feature points on the surface of a hull plate. Additionally,
in order to make the stereo correspondence between the left and right
views more robust, each spot is encoded by the Gray code after pro­
jecting a sequence of Gray code patterns onto the surface. However, this
measurement method has some disadvantages by comparison with the
proposed measurement method in this study. First, this measurement
method is extremely susceptible to illumination intensity because it
requires to project a composite structured light of circular spot array and
Gray code. In contrast, CasMVSNet is trained in 7 different lighting
conditions, which can highly adaptable for different light conditions.
Second, when the measurement method based on active binocular
stereovision is utilized, the left and right cameras must be calibrated
accurately via a calibration board before using. Besides that, the cam­
eras’ external parameters require to be calibrated regularly due to vi­
brations at the shipyard, which is a cumbersome process. Once the
camera pose is changed accidently during working process, the
15
Ocean Engineering 260 (2022) 111968
measurement results will be seriously incorrect. Instead, the measure­
ment method proposed in this work is flexible and convenient because it
does not require to use a calibration board or to project structured lights.
Finally, when the method based on active binocular stereovision is
adopted, a large-size plate has to be divided into several small parts to be
measured, and then splice all the parts of the measurement data ac­
cording to some reference points. However, the point cloud stitching can
be avoided in the proposed measurement method in this work. Because
even if a large-size hull plate is incompletely covered in several views, a
complete point cloud of the large plate can still be acquired by depth
map fusion.
Finally, the traditional MVS algorithms, which are implemented in
the open source software COLMAP and OpenMVS, respectively, are
adopted to reconstruct the sail-shaped plate by using 20 high-resolution
images. The time of dense reconstruction for COLMAP and OpenMVS is
about 17.86 min and 7.42 min, respectively. It can be found that the
efficiency of the learning-based MVS dense reconstruction has been
improved by about 11 times than that of COLMAP and around 4.5 times
than that of OpenMVS. Therefore, the adopted measurement method in
this work is more efficient than the traditional MVS algorithms.
8. Conclusions
A measurement system in combination with a deep-learning based
MVS approach is developed for measurement of ship shell plates. The
measurement system and whole measuring process are firstly eluci­
dated. The accuracy, efficiency and completeness of 3D reconstruction
are then evaluated for three curved hull plates. Finally, the resampling
and the surface reconstruction are carried out. The following conclu­
sions can be drawn:
(a) The effective FOV of the measurement system is obtained by
reasonable selection of parts and optimization of the structure. 20
high-resolution images are sufficiently suitable for the current
measurement system to balance the measurement accuracy and
efficiency.
(b) The measured point cloud data of three hull plates with different
sizes and shapes are compared with their corresponding theo­
retical point cloud data, respectively. It is indicated that the av­
erages of errors of three curved hull plates are all within 1 mm
with 20 high-resolution images. Additionally, the total time of 3D
reconstruction of these three curved hull plates is all about 3 min.
These three cases demonstrate that the developed measurement
system combined with CasMVSNet can reconstruct complete
point cloud data of a curved hull plate with average of errors less
than 1 mm and time consumption around 3 min, which fulfills
precision and efficiency requirements in shipbuilding production.
(c) Although the resampling of the dense point cloud has a little loss
of accuracy, it can remarkably reduce the number of the point
cloud and make the point cloud evenly distributed, which con­
tributes to acquiring to acquire a complete and smooth surface of
the curved hull plate.
(d) Compared with traditional wooden templates, the developed
measurement system integrated with a deep-learning based MVS
approach is more accurate, efficient and inexpensive. Its mea­
surement result is quantitative, which can be seamlessly inte­
grated with 3D CNC plate bending machine. The measurement
system is also robust, flexible and convenient by comparison with
the method based on active binocular stereovision. Moreover, the
deep-learning based MVS approach is more efficient than tradi­
tional MVS algorithms.
In future work, better measurement results could be expected by
training and fine-tuning the deep learning model on a dataset that
contains curved hull plates objects. Thus, our next task is to build a
dataset for ship shell plates. Besides, aiming at the present application,
P. He et al.
GPU memory consumption should be further reduced for high-
resolution images by improving the network architecture. A promising
improvement is to devise a novel cost volume regularization scheme to
substitute 3D CNNs.
CRediT authorship contribution statement
Pengpeng He: Conceptualization, Methodology, Formal analysis,
Writing – original draft. Delin Hu: Validation, Investigation. Yong Hu:
Project administration, Supervision, Funding acquisition.
Declaration of competing interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influence
the work reported in this paper.
Acknowledgements
This research was supported by the Fundamental Research Funds for
the Central Universities, grant number 215202001; and the National
Natural Science Foundation of China, grant number 51779200. The first
author wishes to thanks for the valuable comments provided by Pro­
fessor Wei Chai, at Wuhan University of Technology.
References
Aanæs, H., Jensen, R.R., Vogiatzis, G., et al., 2016. Large-scale data for multiple-view
stereopsis. Int. J. Comput. Vis. 120, 153–168.
Besl, P.J., McKay, N.D., 1992. A method for registration of 3-D shapes. IEEE Trans.
Pattern Anal. Mach. Intell. 14 (2), 239–256.
Bonnin-Pascual, F., Ortiz, A., 2019. On the use of robots and vision technologies for the
inspection of vessels: a survey on recent advances. Ocean Eng. 190, 106420.
Furukawa, Y., Hernandez, C., 2015. Multi-view stereo: a tutorial. Found. Trends®
Comput. Graph. Vis. 9, 1–148.
Girshick, R., 2015. Fast R-CNN. In: Proceedings of the IEEE International Conference on
Computer Vision. ICCV), pp. 1440–1448.
Gu, X.D., Fan, Z.W., Zhu, S.Y., et al., 2020. Cascade cost volume for high-resolution
multi-view stereo and stereo matching. In: Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition. CVPR), pp. 2495–2504.
Heo, E.C., Kwon, M.C., Kim, B.C., 2008. Large free form measurement using slit beam.
Int. Conf. Control Autom. Syst. 1224–1227.
Hiekata, K., Yamato, H., Enomoto, M., et al., 2011. Development and case studies of
accuracy evaluation system for curved shell plates by laser scanner. J. Ship. Product.
Des. 27 (2), 84–90.
Hwang, S.Y., Lee, J.H., 2018. Feasibility of multipoint press with continuously divisional
forming for double curvature plates in shipbuilding. J. Ship. Product. Des. 34 (2),
94–110.
Kazhdan, M., Bolitho, M., Hoppe, H., 2006. Poisson surface reconstruction. In:
Proceedings of the Fourth Eurographics Symposium on Geometry Processing,
pp. 61–70.
Kazhdan, M., Hoppe, H., 2013. Screened Poisson surface reconstruction. ACM Trans.
Graph. 32 (3), 1–13.
Lee, H., Lee, D.J., Huh, M.J., 2013. Development of a measurement system for curved
ship hull plates with multi-slit structured light. J. Kore. Soc. Precis. Eng. 30 (30),
292–299.
Lin, T.Y., Dollar, P., Girshick, R., et al., 2017. Feature pyramid networks for object
detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition. CVPR), pp. 2117–2125.
16
Ocean Engineering 260 (2022) 111968
Lowe, D.G., 2004. Distinctive image features from scale-invariant keypoints. Int. J.
Comput. Vis. 60, 91–110.
Metronor, 2019. Metronor one: single camera electro-optical portable coordinate
measuring system. http://www.metronor.com.cn/wp-content/uploads/2019/05/
Metronor-One-Datasheet-19.1.pdf. (Accessed 9 October 2021).
Mitsuyuki, T., Hiekata, K., Kasahara, T., 2020. Development of manufacturing support
system for ship curved shell plate using laser scanner. Res. Eng. 7, 100157.
Mohd, A.A., Derek, B.D., Albert, K.C., et al., 2017. Improvements to the accuracy of
prototype ship models measurement method using terrestrial laser scanner.
Measurement 100, 301–310.
Munro, C., Daniel, W., 2007. Reconfigurable pin-type tooling: a survey of prior art and
reduction to practice. J. Manuf. Sci. Eng. 129, 551–565.
Neven, H., Zdenka, K., Marko, H., et al., 2019. Analysis of elastic-plastic steel plates
forming based on typical shipyard’s roller bending machine. Ocean Eng. 190,
106438.
Paoli, A., Razionale, A.V., 2012. Large yacht hull measurement by integrating optical
scanning with mechanical tracking-based methodologies. Robot. Comput. Integrated
Manuf. 28 (5), 592–601.
Park, J., Kim, D., Hyun, C., et al., 2016. Thermal forming automation system for curved
hull plates in shipbuilding: analysis and design. Int. J. Comput. Integrated Manuf. 29
(3), 287–297.
Park, J.S., Shin, J.G., Ko, K.H., 2007. Geometric assessment for fabrication of large hull
pieces in shipbuilding. Comput. Aided Des. 39 (10), 870–881.
Paszke, A., Gross, S., Massa, F., et al., 2019. PyTorch: an imperative style, high-
performance deep learning library. Adv. Neural Inf. Process. Syst. 32, 8026–8037.
Schonberger, J.L., Frahm, J., 2016. Structure-from-Motion revisited. In: Proceedings of
the IEEE Conference on Computer Vision and Pattern Recognition. CVPR),
pp. 4104–4113.
Shen, H.Q., Son, S.H., Kim, Y.M., et al., 2017. Forming information calculation algorithm
of 3-D template for evaluation of curved hull plates. Proc. CIRP 62, 411–416.
Shin, J.G., Ryu, C.H., Nam, J.H., 2004. A comprehensive line-heating algorithm for
automatic formation of curved shell plates. J. Ship Prod. 20 (2), 69–78.
Son, S., Kim, B., Ryu, C., et al., 2020. Production automation system for three-
dimensional template pieces used to evaluate shell plate completeness. Int. J. Nav.
Archit. Ocean Eng. 12, 116–128.
Su, S.J., Wang, C.F., Hu, Y., 2015. The key technology research about 3D CNC bending
machine and experimental verification. J. Coast Res. 73, 584–588.
Wang, C.F., Hu, Y., Li, J.X., et al., 2010. A novel forming method for 3D ship hull
forming. J. Wuhan Univ. Technol. (Transp. Sci. Eng.) 34, 431–434.
Wang, C.F., Yuan, P., Li, J.X., et al., 2015. Curved Surface Forming Device for Adjustable
Segmented Mold Board with Square Rams. United States Patent, 8939754.
Wang, Z.X., Wu, Z., Zhen, X., et al., 2016. An onsite inspection sensor for the formation of
hull plates based on active binocular stereovision. Proc. IME B J. Eng. Manufact. 230
(2), 279–292.
Wei, Y., Ding, Z.R., Huang, H.C., et al., 2019. A non-contact measurement method of ship
block using image-based 3D reconstruction technology. Ocean Eng. 178, 463–475.
Xiong, J., Hu, Y., Xia, Z.L., 2014. Design of hyperbolic curved hull plates pressing control
system based on CAN bus[J], 2003–2009 Open Autom. Control Syst. J. 6.
Xue, B., Zhao, T., Yang, X.X., et al., 2017. Relation study on the measuring space and
accuracy level of the multi-node rotary laser positioning system. Ocean Eng. 130,
429–436.
Yao, Y., Luo, Z., Li, S., et al., 2018. MVSNet: depth inference for unstructured multi-view
stereo. Proc. Eur. Conf. Comput. Vis. (ECCV) 767–783.
Yao, Y., Luo, Z., Li, S., et al., 2019. Recurrent mvsnet for high-resolution multi-view
stereo depth inference. In: Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition. CVPR), pp. 5525–5534.
Yoshihiko, T., Morinobu, I., Hiroyuki, S., 2011. “IHIMU-α” A fully automated steel plate
bending system for shipbuilding. IHI Eng. Rev. 44 (1), 6–11.
Yuan, P., Li, J.X., Li, S.Y., et al., 2012. Studies on step by step approximation method of
springback control of curved plate forming by 3D CNC hull plate forming machine.
Ship. Eng. 34 (3), 65–68.
Yuan, P., Wang, C.F., Hu, Y., et al., 2014. Development of large plate bending machine
for shipbuilding with three dimensional numerical control. Shipbuild. China 55 (2),
122–131.