AISTech 2019 — Proceedings of the Iron & Steel Technology Conference

6–9 May 2019, Pittsburgh, Pa., USA

DOI 10.1000.377.151

Efficient Determination of Misalignment of Ladle Shroud Using Machine Vision

Arijit Das1, Saikat Chatterjee1, 2, Atanu Mukherjee1

1
M. N. Dastur & Co. (P) Ltd.
P-17 Mission Row Extension
Kolkata 700013, India
Email: Arijit.Das@dastur.com
2
Dastur Innovation Labs
250 Yonge Street, Suite 2201
Toronto, ON M5B 2L7, Canada
Phone: + (1) - 647-6-DASTUR, + (1) - 647-632-7887
Email: Chatterjee.Saikat@dastur.com

Keywords: Continuous casting, tundish, Shroud, SEN, inclusion, erosion, misalignment, slag entrainment, machine vision,
stereo vision, morphological operation, disparity, segmentation, extraction.

INTRODUCTION
Continuous casting of steel includes multiple transfer operations of liquid steel from ladle to mold via tundish, followed by its
solidification to semi-finished products such as blooms, billets or slabs. Since the transfer operations can easily result into
reoxidation of liquid steel, it presents significant challenges in terms of retaining the steel quality obtained through secondary
metallurgical operations such as ladle refining, vacuum degassing or RH degassing. Over the years, the transfer operations
have seen significant improvement due to increasing usages of slide gates, ladle shrouds, ladle nozzles, flow modifiers such
as impact pads, weirs dams and so on [1]. Injection of inert gases into the ladle shrouds has also turned out to be a common
practice as it creates a protective gas blanket which prevents air ingress and subsequent reoxidation.
Although a lot has been done, one aspect of transfer operation which has largely gone unnoticed is the misalignment of ladle
shroud. On careful observation, it can be noticed that ladle shrouds become tilted with respect to the vertical direction.
Recently, Chattopadhyay et. al.[2] and Chatterjee et. al.[3] have pointed out continuous casting operations in both billet caster
and slab caster with misaligned ladle shrouds, as depicted in Figure1. The luminous circular regions and dark cylindrical
regions depict tundish open eyes (TOEs)[3] and ladle shrouds respectively. Ladle shroud misalignment affects steel quality
and operation as it undermines the performance of impact pad and dams causing adverse flow conditions, increasing the
chance of open eye formation, decreasing residence time for inclusion flotation along with increasing refractory wear in
continuous casting.[3]

(a) (b)
Figure 1: Photographs showing misaligned ladle shrouds during plant operations resulting in the formation of eccentric
tundish open eyes (TOE) in a (a) billet caster and (b) slab caster tundish. [3]

© 2019 by the Association for Iron & Steel Technology. 1479

 PHYSICAL DESCRIPTION OF THE PROBLEM

Possible causes of shroud misalignment

During the continuous casting operations, the ladle turret swivels and the empty ladle is replaced by a filled ladle. The ladle
shroud is then placed on the collector plate nozzle, at the ladle bottom. The sheer inertial force generated due to flow of liquid
steel within the ladle, or within the shroud, can easily displace or incline the shroud from its intended position. A schematic
diagram of the ladle bottom-collector plate-ladle shroud assembly is depicted in Figure 2(a).
Steel plant operators often clean the inclusions/solid residue sticking on the inner walls of ladle shroud by blowing a
supersonic oxygen jet, just before attaching the shroud to the collector nozzle. Many a times, it so happens that all the
residues are not removed due to improper blowing of oxygen jet, and as a result, the inner surface of the shroud becomes
uneven which can result in improper setup of the whole arrangement. The ladle shroud can easily get misaligned when it is
being fixed to the ladle bottom.

Effect of shroud misalignment on steel quality

A schematic representation of misaligned shrouded transfer of molten steel from ladle to tundish along with other associated
events is depicted in Figure 2(b).

Figure 2 (a): Mounting of an actual shroud Figure 2 (b): Schematic representation of shroud misalignment
on a collector plate nozzle[1] (exaggerated view) and its effect on steel quality.

Flow control devices (FCDs) such as turbo-stop, weirs, and dams are usually placed inside the tundish in order to obtain
clean steel by aiding inclusion flotation to the top surface and get collected by the overlying slag layer. The recirculatory
flows within the tundish, along with the buoyancy effect, help in inclusion flotation. The main objective of these FCDs is to
restrain the flow of highly turbulent liquid steel within itself, which prevents the spread of incoming inclusions throughout
the whole tundish. The entrainment of inclusions into the mold via the SENs is thus, prevented. However, confining the
highly turbulent flow within the turbo-stop region becomes more and more difficult if the misalignment of the ladle shroud
increases. The inclination of ladle shroud on a particular side directs the fluid flow in a path away from the intended one. The
residence time, which is defined as the time a single fluid element spends in tundish, decreases due to much reduced
recirculatory flows and hence, results into reduced inclusion flotation. The negative effects of shroud misalignment can be
stated as follows:
a. Tundish bottom refractory life reduction: The flow directed outside the turbo-stop results in increased wear/erosion
of the tundish bottom refractory. The maintenance cost thus increases due to increased gunning of refractory mass at
tundish bottom.
b. SEN life reduction: Biased flow of inclusions towards SEN may result in its increased clogging, thus necessitating
its premature replacement.
c. Eccentric open eye: A misaligned ladle shroud in a particular direction can lead to formation of an eccentric open
eye in the corresponding direction[3], giving rise to reoxidaion and quality issues. Reoxidaion of steel occurs when it

1480 © 2019 by the Association for Iron & Steel Technology.

 comes into direct contact with ambient air. Harmful inclusions such as Al2O3[4], Al2O3-TiOx[5] or MnAl2O4[6] can
form when oxygen and nitrogen present in air react with dissolved elements in steel such as Al, Mn and Ti. These
inclusions have tendency to form clusters and attach to the walls of the nozzles, thereby clogging them.
d. Product quality deterioration: Inclusions which pass on to the final product can be very detrimental to the product
quality as it severely affects its mechanical properties such as fatigue and fracture toughness[7,8].
A couple of recent studies[3] show the effect of ladle shroud misalignment on fluid flow within tundish. From the calculated
turbulent kinetic energy contours within a slab caster tundish for different cases of misalignment, viz. 0, 2 and 5 degrees,
Chatterjee and Chattopadhyay[3] observed that the turbulent flow from the shroud shifts away from its intended target with
increasing misalignment angle. As can be observed from Figure 3, although the flow for the case of 2 degree misalignment
can still be confined within the turbo-stop region, the same is not true for the case of 5 degree misalignment.

Figure 3: Turbulent kinetic energy contours inside tundish at a Figure 4: Preferential inclusion entrainment in
misalignment angle of: (a) 0°, (b) 2°, and (c) 5°[3]. the direction of misalignment[2].

Since the inclusion trajectories follow the path of fluid flow in tundish, they can get easily dispersed into the bulk liquid steel
or flow preferentially towards the direction of bias, instead of floating up to the overlying slag layer. Chattopadhyay et al. [2]
performed physical modelling experiments with 5 degree misalignment in billet caster tundish to see its effect on inclusion
dispersion within the bulk fluid. It was clearly observed that more slag droplets (polyethylene beads) were entrained in the
direction of misalignment, as depicted in Figure4. These experimental observations strongly substantiate the claims presented
above.

MACHINE VISION APPROACH

The artificial intelligence (AI) and industrial internet of thing (IIOT) integrate production technology and information
technology to support smart factories across the globe. Over the years, various approaches have been used to provide
customized solution for steel productivity improvement. Among them machine vision approach, instead of manual
inspection, improves the quality enhancement by image based automatic inspection, component monitoring and analysis from
raw materials to final product. Machine vision comprises several stages such as image acquisition, processing, segmentation
and recognition. Several successful practical applications of machine vision approach in steel industry have been reported in
the past. A few of them are listed below:
• Slag detection system [12,13]
• Coil surface defect detection [14,15]
• BF tuyere monitoring system [16,17]
• Refractory maintenance [18,19]

© 2019 by the Association for Iron & Steel Technology. 1481

Rationale for Machine Vision
Machine Vision (MV) is the technology to provide automatic recognition and evaluation of images for automatic inspection,
process control, and robot path guidance in industry. A MV system consists of (a) vision camera for object detection and (b)
computer vision algorithms for feature extraction, image analysis and quantitative measurement. The relative advantages of a
MV system as compared to other possible sensor systems (such as infrared, laser, UV) for shroud misalignment detection are
as follows:
• Lower capital cost compared to a system consisting of IR/laser sensor array along with their logic controllers.
• Multiple feature extraction including object shape, pattern, color and geometry recognition.
• Different type of foreground separation techniques can be applied depending upon the environmental hazards.
• Several kind of electromagnetic radiation like radio waves, infrared, ultraviolet, gamma radiation which lead to error
of the measured variable can be avoided.
• Higher resolution cameras provide pixel spreading over a large surface area lead to higher degree of precision.
• Better speed of response, accuracy, reliability and repeatability.
• Two connection points are sufficient between camera and central processor unit without any signal conditioning.
• Flexibility of the object 3D rendering lead to proper investigation and visualization of the system.

STEREO VISION
Stereo Vision is an area of study in the field of Machine Vision, which recovers 3D information of an object using two or
more images from camera. The depth output of each pixel is determined by computing the difference of two pixel positions
in two different image planes. Finally the disparity map is obtained by filling the spaces between two consecutive edge
pixels. In short, it emulates the human vision by using two or more 2D views of the same view to derive its 3D depth
information.
An efficient stereo vision algorithm was developed to reliably detect shroud misalignment by deriving its 3D depth
information from captured 2D images. Extensive experiments were conducted to demonstrate that this approach can provide
consistent results in misalignment detection. In the present work, a simplified stereo vision technique is depicted in Figure 6
and Figure 7. Stereo vision fundamentally works on the principle of triangulation[9], which is described below.

Fundamental principle of stereo vision

Triangulation is a robust technique that facilitates verification of data through cross verification from two or multiple sources.
In computer stereo vision technique refers the process of determining a 3D space given its projection onto two or more
images.
Using a pair of stereo images the apparent pixel difference of a particular pixel or cell or patch can be calculated which is
called the disparity depicted in Figure 5. The values in this disparity map are inversely proportional to the scene depth at the
corresponding pixel location. By disparity mapping we are computing depth map information.

Figure 5: Schematic representation of the triangulation technique

Points CL, CR and P (and PL and PR) lie on a plane. Since two image planes lie on the same plane (distance f from each
camera), the lines CLCR and PLPR are parallel. The distance between the centers of the two lenses is called the baseline width
(B), focal length (f) and disparity (D). The projection of the world point on the two image planes is (xl, yl) and (xr, yr). So
from the method of similarity of triangles we found the equations are as follows [9].

1482 © 2019 by the Association for Iron & Steel Technology.

 Depth (Z) = f * B/ D (1)

PROPOSED METHODOLOGY
In our proposed model two web cameras have been used with their optical axes parallel and aligned horizontally. The
disparities of 3D points mapped to pixels in two images are computed by using the principle of triangulation. Its essence is to
schematically reconstruct 3D object (pipe) by analyzing the 2D information extracted. The deviation of angle with reference
to a vertical line in 3D geometry can be acquired using these disparities.

Proposed hardware design and implementation

The proposed system consists of a couple of cameras (30 fps @640X480, 60° FOV) where their optical axes are parallel.
This is used to get two different images (Left & Right) in a manner similar to human binocular vision placed at a distance
from a sphere-shaped pipe. A hollow PVC pipe with proper high intensity LED radiation generates a function simulating the
shroud which can be aligned at different angles in different quadrants through a mechanical ball joint arrangement shown in
Figure 6 and Figure 7 below.

Figure 6: Front view of the proposed 3D model Figure 7: Top view of the proposed 3D model

Software design
By enhancing the image processing algorithm and disparity computation method, we have estimated the misalignment angle
of the shroud and also relative 3D schematic reconstruction from the relative disparity point calculation. A simple block
diagram of the system is shown in Figure 8.
In this application, a setup of OpenCV (Open Source Computer Vision Library) in Ubuntu operating system has been
employed to obtain the flexibility of capture image directly from camera and using its library programming functions for
image analysis. Code blocks is a cross platform IDE to supports compiling and running multiple languages. The complete
algorithm is developed in the C++ environment.

© 2019 by the Association for Iron & Steel Technology. 1483

 Figure 8: System flow chart diagram
For the epipolar rectified image pair, each point in the left image lies on the same horizontal line (epipolar line) as in the right
image. This approach is used to reduce a search space for depth map computation algorithms. The depth of an image pixel is
the distance of the corresponding space point from the camera center. To estimate the depth map and detect 3D objects, the
corresponding pixels in the left and right images have to be matched. To eliminate all the significant distortion camera
calibration is mandatory shown in Figure 9.

Left camera Right camera

Figure 9: Original image pairs of a chess board captured for camera calibration

Stereo rectification and Camera Calibration

In order to apply stereo ranging techniques with a higher level of accuracy it is important to eliminate radial and tangential
lens distortion, which gives extrinsic and intrinsic parameter by using chess board (9X7 crossed) pattern. The intrinsic
parameters of individual camera include distortion coefficients and camera matrix (focal lengths and optical center) [10,11].
Extrinsic coefficients contain information about rotation and translation matrices between the stereo pairs. Intrinsic
parameters are used to remove the distortion in the image pairs and extrinsic parameters are then estimated with the help of
corrected pairs. Few original image pairs of a chess board, considered for camera calibration, are shown in Figure 9. The
following data are obtained from the camera calibration algorithm[10,11]:

1484 © 2019 by the Association for Iron & Steel Technology.

 • the camera matrix (intrinsic and extrinsic parameter)
• the distortion matrix

Algorithm of the proposed system

Online Frame Capturing

Image capturing is a fundamental step in machine vision to prepare raw data. By creating a video capture Application
Programming Interface (API), the left and right images are simultaneously captured frame-by-frame.

Distortion Rectification/Remapping
Raw image frames can be rectified by several linear and nonlinear techniques for 2D images. The image content is not
changed but the pixel grid is deformed and mapped to the destination image. The computed response is stored in the
destination image at the same location (x, y). It means that the output image is the same size as the input image. Prior to
image rectification it is essential to choose a valid region of interest (ROI) by upper and lower threshold limit with variable
track bar method.

Foreground Extraction
Prior to the rectification of the valid image the foreground (i.e. pipe), are extracted from the background object which may
vary depending upon the steel structure. There are different types of background foreground segmentation methods like,
GrabCut based segmentation, superpixel based segmentation, watershed based segmentation etc. The method of
implementation of segmentation algorithm can be chosen depending upon the type of actual environment. In this
experimental model, HSV (hue, saturation, value) colorspace represents the colorspace similar to the RGB color model. By
choosing the correct upper (HU) and lower (HL) boundaries variation of the different Hue value the red hot pipe filtered out
or separated from the background. For industrial noisy environments the process of the segmentation may differ but can be
used for any type of background foreground separation.

Morphological Operation
A morphological approach is used to decompose the object into simple components. After foreground extraction, multiple
foreground or background noise body is removed by despeckling or the denoising filtration method. To precede this
operation a morphological opening and closing operation can be applied. Subsequently a morphological technique is used
here to preserve the structure of the shape but removes the entire identical pixel in order to achieve faster processing speed
and a smaller memory footprint.

Feature Computation
Our aim is to determine the relevant feature points in both left and right image view of the pipe. In this particular case the
feature of the pipe is well defined and statistically sound rather than being statistically fuzzy. So the feature computation
technique of the particular system is fairly straightforward. We choose various appropriate points, mainly top and bottom
pixel points, to perform pixel matching between left image views corresponding to the right image view. Ultimately by
matching each pixel in the left image with its corresponding right image, the top disparity (xltop –xrtop) and bottom disparity
(xlbot –xrbot) is computed as shown in Figure 10.

Angle Estimation from 3D Geometry

The proposed methodology is used for detection of misalignment angle of the pipe with respect to its X and Z coordinates as
the pipe lies in Y axis. The characteristic of the pipe is statistically sound and linear in nature so we do not require the dense
disparity mapping rather and use a spark disparity method of two points. From the camera calibration method we have
already found the focal length (f). So, the final depth information (Z) is arrived at by using the camera intrinsic and extrinsic
parameter and calculating the top depth (Ztop) and bottom depth (Zbot) by equation (2) & (3)

Ztop = f*b / (xltop –xrtop) =f*b/d, (2)

Zbot = f*b / (xlbot –xrbot) =f*b/d, (3)

© 2019 by the Association for Iron & Steel Technology. 1485

Once we get the depth information, we compute the conical X angle with respect to Y coordinates and conical Z angle with
respect to Y coordinates which are the final measurement values for misalignment detection by the Pythagorean theorem, as
shown in equation (4) and (5)

XY angle(θ xy) = tan-1 y/x (4)

ZY angle(θ zy) = tan-1 z/x (5)

Figure 10: Disparity matching and depth calculation

The measurement accuracy can be further improved by using higher resolution camera or increasing the camera base length.

RESULTS AND DISCUSSION

In the experimental setup, a real angle measuring arrangement was placed on both the backside of the plane and in the work
plane (concentric circle) so that the measurement of the true angle of the misalignment can be visualized, as shown in Figure
6 & Figure 7. The shroud was placed at different angles in different quadrants and simultaneously actual and online angles
xy zy
were recorded as shown in Table 2. A single data record set of θ angle as well θ angle for quadrant - I are shown in Table
1. The errors of true value and measured value were also calculated. The proposed system was tested and verified by
capturing multiple data record set shown in Figure 11 and 12. Based on the experiments we found that the final deviation of
the value was within ±5%.

Table 1: Actual angle (°) Vs online value

Actual XY Online XY Deviation Actual ZY Online ZY Deviation
Angle (°) Angle (°) Angle (°) Angle(°) Angle(°) Angle (°)

0.0 0.13 0.13 0.0 0.15 0.15

5.0 5.14 0.14 5.0 5.19 0.19
10.0 10.23 0.23 10.0 10.31 0.31
15.0 15.36 0.36 15.0 15.44 0.44
20.0 20.47 0.47 20.0 20.68 0.68

1486 © 2019 by the Association for Iron & Steel Technology.

Figure 11: Deviation of actual XY angle with online value Figure 12: Deviation of actual ZY angle with online value

Table 2: Online monitoring value for different angle with different quadrant.
Quadrant Image Online Value

Vertical
   

QI
   

QII

   

QIII

   

QIV
   

© 2019 by the Association for Iron & Steel Technology. 1487

3D rendering for online monitoring
To further monitor 3D widgets of the hollow pipe we have used VTK (visualization tool kit), by using its volume rendering
technique and obtained real time online views of the pipe with reference to the 3D Widget. The actual front view and the 3D
reconstructed view of the misaligned pipe are shown in Figure 13 and Figure 14 respectively.

Figure 14: Schematic reconstruction of the pipe in 3D

Figure 13: Front view of the pipe at particular angle
Geometry

SHROUD MISALIGNMENT DETECTION AND RECTIFICATION SYSTEM SETUP IN PLANT

ENVIRONMENT

Existing set-up and operations

The existing plant operations employ a device called ‘shroud manipulator’, as shown in Figure15, to handle the ladle shroud
in between transfer operations. Shroud manipulator is necessary as manual handling of red hot ladle shrouds would be a
major safety concern. However, the control of the shroud manipulator is in the hands of an operator, who regulates its
movement using a joystick. It is very difficult for an operator to check the alignment of the ladle shroud with respect to the
vertical axis after placing the ladle shroud in position and thus can result in ladle shroud misalignment.

Corrective action and recommendations

The ‘integrated stereo vision system’ developed in the present work is able to identify the alignment of the ladle shroud with
an error of ±5%. An air-cooled water and dustproof housing (IP65/67) is required for it to sustain the hazardous industrial
environment.
The raw image data from the stereo camera is first sent to the processing unit for interpretation and analysis using the
custom-made image processing algorithm. The resulting misalignment angle is (a) revealed on a display monitor and (b) fed
into a modular I/O controller which can generate alarms when a pre-defined threshold value is attained. The shop floor
personnel can take corrective actions when the threshold limit is crossed. This type of feedback control system has shown
promising results in the plant environments and thus, is well-accepted in the steelmaking fraternity.

1488 © 2019 by the Association for Iron & Steel Technology.

 Figure 15: Schematic view of the plant setup for shroud misalignment detection

CONCLUSION
Since ladle shroud misalignment results in various problems such as reduction in tundish bottom refractory life and SEN life
along with product quality deterioration on account of increased inclusion formation, its on-line rectification is necessary. A
machine vision approach has been developed to investigate the precise position of a ladle shroud and its angle of
misalignment from the vertical axis. The ‘integrated stereo vision system’ developed in the present work can be used during
the continuous casting operation. The major advantages are listed below:
• Automatic and precise inspection system instead of manual visualization of the misalignment by the operator thus
increasing productivity and quality
• More cost-effective solution for caster operations compared to other sensors
• Algorithm based machine vision detection obviates the need for a GPU (graphics processing unit)
• Efficient denoising/despeckling image filtering algorithm increases precision in noisy environments
The proposed method is a robust and cost effective solution for detection of misalignment angle. Compared to other sensors,
camera based detection is a simpler solution and a simple computer vision algorithm technique can be used to get 3D
correspondence of the shroud in continuous caster. The computer vision algorithm used here also allows us to work in super
resolution. The algorithm used in the proposed system can also be embedded in single board computer system like Raspberry
pi.
In the future, we plan to perform experiments and also tests for higher accuracy with a higher resolution industrial camera
and enhanced algorithms on greater number of real (data) images with an aim to compare the presented approach with other
existing algorithms. We also plan to investigate the reliability of the system by implementing it in very noisy operating
environments.

© 2019 by the Association for Iron & Steel Technology. 1489

 REFERENCES
1. D. Mazumdar , P.K Singh, R.K.Tiwari: ISIJ International, Vol58 (2018),p.1545-1547.
2. Chattopadhyay, M. Isac, R. I. L. Guthrie: ISIJ International, Vol. 51 (2011), No.5, p. 759 - 768.
3. S. Chatterjee, D. Li, and K. Chattopadhyay: ISIJ Int., 2016, Vol. 56, pp. 1889 – 92.
4. S. Chatterjee, PhD thesis, University of Toronto, Toronto, 2017.
5. C. Lyons and P. Kaushik: Steel Res. Int., 2011, 82[12], pp. 1394-03.
6. S. Chatterjee and I.-H. Jung: J. Eur. Cer. Soc., 2014, 34[6], pp. 1611–21.
7. K. Peaslee, V. Singh, and S. Lekakh, Proceedings of the Richard J. Fruehan Symposium (2011).
8. P. A. Thornton, J. Mat. Sci., 1971, 6, pp 347-356.
9. https://docs.opencv.org/3.4.1/d0/dbd/group_triangulation.html.
10. http://www.cs.cmu.edu/~16385/s17/Slides/11.1_Camera_matrix.pdf.
11. https://tigerprints.clemson.edu/cgi/viewcontent.cgi?article=2216&context=all_theses.
12. https://www.vesuvius.com/en/our-solutions/sv-se/iron-and-steel/continuous-casting/ladle-to-tundish/slag-detection-
systems.html
13. https://www.ametek-land.com/products/fixedthermalimagersandlinescanners/sds-e
14. http://automationtatasteel.com/html/SQIS.html
15. http://www.winspection.com/surface-inspection.php
16. http://automationtatasteel.com/html/tuyereye.html
17. http://www.paulwurth.com/News-Media/News-and-Archives/Market-Breakthrough-for-Tuyere-Phenomena- Detection-
System
18. http://www.tempsens.com/blog/thermal-imager/ladle-refractory-monitoring-system
19. https://automationtechnology.de/cms/wp-content/uploads/2015/06/Ladle_Refractory_Monitoring_web.pdf

1490 © 2019 by the Association for Iron & Steel Technology.