Acta Astronautica 195 (2022) 234–242

Contents lists available at ScienceDirect

Acta Astronautica
journal homepage: www.elsevier.com/locate/actaastro

Research paper

Model-based supervised sensor placement optimization to detect propellant

leak in a liquid rocket engine
Noriyasu Omata a ,∗, Daiwa Satoh a ,1 , Seiji Tsutsumi a , Kaname Kawatsu b , Masaharu Abe c
a
Japan Aerospace Exploration Agency, Sagamihara, Kanagawa, 252-5210, Japan
b
Japan Aerospace Exploration Agency, Tsukuba, Ibaraki, 305-8505, Japan
c
Ryoyu Systems Co., Ltd., Nagoya, Aichi, 445-0024, Japan

ARTICLE INFO ABSTRACT

Keywords: The sensor selection for leak detection and diagnosis in a reusable liquid rocket engine was studied.
Sensor placement optimization Sufficient amounts of normal and leaked training data were generated by Monte Carlo simulations considering
Supervised fault detection variations in system conditions based on past firing test results. A multivariate supervised analysis, in which
Model-based fault detection
measurements are linearly projected onto a vector that characterizes the difference between distributions of
Liquid rocket engine
normal and leaked cases, successfully detected the simulated leaks that could not be detected by conventional
univariate red-line judgment and unsupervised principal component analysis. Significant sensors for leak
detection were selected using the greedy approach based on the detection performance score. It was found
that only 11 of the 27 sensors were sufficient to maintain the detection performance.

1. Introduction Data-driven health management methods utilizing machine learning

Sensor placement optimization is indispensable for performance
prediction, control of the system, prognostics and health management
(PHM), especially in aerospace systems due to the limitation of the in-
stallation, cost, and downlink capacity to obtain measurement data [1].
A reduction in the number of sensors is also preferable for applying
machine learning to PHM in order to reduce the effect of the curse
of dimensionality [2]. Therefore, sensor placement optimization is
essential for PHM of aerospace systems. In this study, sensor placement
optimization in a reusable liquid rocket engine developed by the Japan
Aerospace Exploration Agency (JAXA) [3,4], named RSR engine, was
investigated.
According to Maul et al. [5], there are generally four performance
requirements in sensor placement optimization: observability, relia-
bility, fault detectability, and cost. This paper focuses on the third
requirement, fault detectability, and selection of sensors crucial for fuel
leak detection in the rocket engine, which uses liquid hydrogen and
oxygen for propellants, is studied. The leakage of cryogenic propellants
from pipe connections is one of the major failure modes of rocket
engines, which may lead to fatal accidents; however, the differences
in sensor values caused by leaks are so subtle that detection is hardly
possible with conventional red-line judgment. Therefore, a high cost is
paid for inspection and maintenance between flights and static firing
tests based on our experience.
techniques have been studied and demonstrated in the RSR engine
to achieve fast and accurate failure detection and diagnosis between
flights [6,7]. Sato [6] applied a method called system invariant analysis
technology (SIAT) to detect failures based on the collapse of relation-
ships between sensor values. Tsutsumi [7] studied data-driven fault
detection using a bivariate time-series analysis.
In general, there are roughly two approaches for data-driven health
management: a supervised approach and an unsupervised approach; the
former refers to the data of the fault condition, while the latter does not.
Unsupervised approach detects faults by measuring deviations from
normal data. Thus, unsupervised learning is not restricted to a specific
fault contained in the training data. Several unsupervised methods,
including the nearest neighbor and one-class support vector machine
(SVM), have been considered in terms of their application to rocket
engines [8]. The supervised approach uses both normal and fault data
and performs supervised learning techniques to detect faults. There are
few recent studies on fault detection using supervised approaches for
rocket engines, such as [9], although supervised fault detection has
been considered since the 1990s [10].
Data-driven PHM requires a sufficient amount of training data
to achieve satisfactory performance in fault detection and diagnosis,
which is one of the major disadvantages. This is crucial for supervised
approaches in which training data for fault conditions are required,

∗ Corresponding author.
E-mail address: omata.noriyasu@jaxa.jp (N. Omata).
1
Currently, Hitachi, Ltd., Hitachi, Ibaraki, 319-1292, Japan.

https://doi.org/10.1016/j.actaastro.2022.02.009
Received 8 October 2021; Received in revised form 19 January 2022; Accepted 15 February 2022
Available online 26 February 2022
0094-5765/© 2022 The Authors. Published by Elsevier Ltd on behalf of IAA. This is an open access article under the CC BY license
(http://creativecommons.org/licenses/by/4.0/).
N. Omata et al. Acta Astronautica 195 (2022) 234–242

Table 1 Table 2
Specifications of the RSR engine. Sensors modeled in the SLS model of RSR engine, and the order of the sensors
Item Value extracted by the proposed method. The gray hatch highlights an effective set of
11 sensors.
Propellant LH2 & LOX
ID Item Order
Engine cycle Expander bleed
Throttling range 40–100 % P1 Pressure of FTP pump discharge 17
Thrust (sea level) 40 kN P2 Pressure of OTP pump discharge 7
Specific impulse 320 s P3 Pressure at inlet of regenerative cooling channel 19
Combustion pressure 3.4 MPa P4 Pressure at outlet of regenerative cooling channel 2
Mixture ratio 6.0 P5 Pressure at downstream of main fuel valve 5
P6 Pressure at low temperature side of mixer inlet 6
P7 Pressure at high temperature side of mixer inlet 13
P8 Pressure of fuel side injector 11
whereas faults rarely occur in real systems. Owing to the lack of fault P9 Pressure of oxidizer side injector 16
P10 Pressure of FTP turbine inlet 8
data, unsupervised approaches have been employed in the RSR en-
P11 Pressure of FTP turbine outlet 26
gine [6,7]. Obtaining sufficient data from experiments is costly in terms P12 Pressure of OTP turbine inlet 14
of both financial cost and time. The use of high-fidelity simulations such P13 Pressure of OTP turbine outlet 25
as computational fluid dynamics (CFD) to simulate the global behavior P14 Pressure of exhaust gas in turbine vent line 10
P15 Pressure in combustion chamber 4
of a liquid rocket engine for generating training data is also unrealistic
T1 Temperature difference between inlet and outlet of FTP pump 21
because of the computational cost. T2 Temperature of OTP pump discharge 9
As a substitute for CFD, a system-level simulation (SLS) is employed T3 Temperature of fuel side injector 22
to simulate the global behavior of liquid rocket engines [11–14]. The T4 Temperature of FTP turbine inlet 24
T5 Temperature of OTP turbine inlet 3
major components of liquid rocket engines are turbopumps, combus- T6 Temperature of OTP turbine outlet 20
tion chambers, nozzles, regenerative cooling channels, pipes, valves, T7 Temperature of exhaust gas in turbine vent line 15
etc. SLS consists of reduced-order models for these components, and QF Volumetric flow rate of FTP pump 27
computation of the entire operational sequence is possible within an af- QO Volumetric flow rate of OTP pump 18
NF Rotation speed of FTP 23
fordable time. Both normal and abnormal conditions can be computed NO Rotation speed of OTP 12
by SLS as far as the modeling of faults is possible in the framework of Feng Thrust 1
the SLS.
The model-based PHM approach using the SLS has recently be-
come a promising candidate for the health management of rocket
A schematic diagram of the RSR engine is shown in Fig. 1, and
engines [15–18]. Iannetti et al. [15] and Cha et al. [16] developed an
the engine specifications are listed in Table 1. The propellants for
analytical rocket engine model as a state-space model, and real-time
the RSR engine are liquid hydrogen (LH2) and liquid oxygen (LOX).
fault detection was conducted by sequentially evaluating the residuals The yellow, blue, and red lines in Fig. 1 represent the flow paths of
of the extended Kalman filter or unscented Kalman filter. Kawatsu the liquid hydrogen, liquid oxygen, and heated hydrogen through the
et al. [17] developed an SLS model of an electromechanical actuator in regenerative cooling channels. This engine employs an expander bleed
a reusable liquid rocket engine to generate reference data containing cycle, and a fuel turbopump (FTP) and oxidizer turbopump (OTP) are
faults. Then, a model-based diagnosis was conducted by comparing driven by heated hydrogen. As many as 60 sensors were installed to
the simulated measurement data with the reference data. Park and monitor the engine conditions.
Ahn [18] recently employed supervised fault detection and diagnosis Various operation sequences were tested in the static firing test of
based on data obtained from model-based simulations. the RSR engine [4]. Fig. 2 shows the focus of this study. A thrust of 40%
The health management method developed in this study employs was maintained for 8 s after ignition of the engine at 𝑡 = 0 s, and then
both model-based and data-driven approaches. Monte Carlo simulations the engine reached 100% thrust. The results of the static firing test are
were performed with the SLS, and the leak model required for the shown by dashed red lines in Fig. 2 for three typical sensors (Feng, P4,
fault simulation was presented. The variation of the system conditions and T5). It was found that the steady-state condition at 100% thrust
is introduced in the Monte Carlo simulation by referring to the past was reached approximately 20 s after ignition.
test data, which enables the application of a data-driven approach. A
supervised leak detection is performed using a linear projection that 3. Normal and fault simulation of the RSR engine
characterizes the effect of the leaks, although the detection was unable
through red-line judgment and principal component analysis. Efficient Satoh et al. [14] have developed the SLS for the RSR engine. The SLS
sensors are finally selected depending on the performance of the leak model consists of a reduced-order model of the FTP, OTP, combustor,
nozzle, regenerative cooling channels, valves, etc. In this SLS, 𝐾 = 27
detection.
sensors listed in Table 2 were modeled, where 15 were pressure sensors,
7 were temperature sensors, and the remaining sensors measured the
2. RSR engine flow rate, rotation speed, and thrust. In the SLS model, pump per-
formance is computed using the Suter curve [19]. The combustion of
The RSR engine employed in this study was developed in the fuel and oxidizer in the combustor was calculated using a chemical
reusable sounding rocket (RSR) program [3,4]. The aim of this program equilibrium program [20]. The amount of heat exchanged between the
is to realize a reusable rocket that can launch a payload to an altitude hydrogen and regenerative cooling channel can be obtained from the
of 100 km, return to the launch site, and then be ready to fly again heat transfer coefficient of the combustion gas, calculated using the
within 24 h. To meet these requirements, the RSR engine was designed Bartz equation [21]. More details of the SLS model are summarized
to realize the following features: in the literature [14].
Because the SLS is composed of the reduced-order models of each
1. capability of wide-range throttling; component, there are model parameters to be identified to agree with
2. quick and accurate engine control; the actual test results. Let 𝜽 represent the set of all the model pa-
3. long-lasting durability; and rameters included in the SLS. Based on the static firing test results,
4. health monitoring. the best model parameter 𝜽̂ was identified using the ensemble Kalman

235
N. Omata et al.
Fig. 1. Locations of sensors and leaks in the RSR engine. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this
article.)
Fig. 2. Time-series of sensor values in the continuous combustion sequence of the RSR
engine.
filter [14]. The result obtained using 𝜽̂ is shown as a solid black line in
Fig. 2. Although disagreement is observed up to 𝑡 = 80 s, especially for
the temperature measured by sensor T5, the steady-state results at the
end of the operation sequence are in reasonable agreement.
The fuel leak model was newly added to the SLS model to generate
the fault training data required for supervised fault detection. Leak
was assumed to occur from a leak hole in a pipe. The leak hole was
simulated using an orifice model:
√
2𝛥𝑃
𝑚̇ = 𝑘𝑑𝑝 𝑅 𝜌𝐴 , (1)
𝜌 simulation is performed to achieve this.
where 𝑚̇ is the mass flow rate, 𝑅 is the resistance coefficient, 𝜌 is the
density, 𝐴 is the cross-sectional area of the leak hole, 𝛥𝑃 is the pressure
difference, and 𝑘𝑑𝑝 is the correction coefficient. Assuming a leak into
the atmosphere, 𝛥𝑃 is the pressure difference from the atmospheric
pressure. The magnitude of the leak is adjusted by the area of the
leak hole 𝐴, ranging from 0.25𝜋 to 1.0𝜋 mm2 . The locations of the
leaks were selected from high-pressure locations with higher risk of
leakage as shown in Fig. 1. Hydrogen leak was assumed to occur where
hydrogen was in a gas-like supercritical state: the inlet of the FTP
turbine (Leak 1), outlet of the regenerative cooling channels (Leak 2),
and fuel feed line of the igniter (Leak 3). This is because 𝑘𝑑𝑝 and 𝑅
in the leak model for hydrogen were determined only for the gas-like
supercritical state based on experimental results [22]. The outlet of the
OTP pump (Leak 4) was selected as the location of oxygen leakage,
because the oxygen there is in a liquid state and 𝑘𝑑𝑝 and 𝑅 were
determined based on the literature. Generating fault training data by
236
Acta Astronautica 195 (2022) 234–242
such cryogenic physics-based modeling is a new attempt, to the best of
the authors’ knowledge.
There are two types of fuel leak. One occurs before engine ignition,
and the other occurs during engine operation. This study only focused
on the former case because the latter case can be detected easily by
an apparent change in the time-series results. The dotted blue line in
Fig. 2 shows the SLS result for the former leak case of 𝐴 = 1.0𝜋 mm2
at 𝑙 = Leak 1. There was almost no difference between the normal and
leaked cases. Leaks from other points showed similar results. This result
shows that it is almost impossible to detect the fuel leak if a fuel leak
occurs before engine ignition.
This study focuses on the steady-state condition at 100% thrust,
where SLS agrees reasonably with the static firing test. The steady
sensor values represented as a 𝐾-dimensional vector are used in the
following discussion. The results of the SLS are uniquely determined
when the parameters of the model 𝜽 are determined. Let the SLS results
without a leak be represented as 𝒖𝑁 = SLS𝑁 (𝜽) ∈ R𝐾 . When there is
a leak, there are two additional parameters: the cross-sectional area of
the leak hole 𝐴 and the location of the leak 𝑙. Let the leaked result be
written as 𝒖𝐴 = SLS𝐴 (𝜽, 𝐴, 𝑙) ∈ R𝐾 .
4. Monte Carlo simulation for realistic system variation
Sensor values vary even if the same engine is operated under the
same operating conditions because of the variation in ambient con-
ditions, precooling temperature, timing of valve opening/closing, and
so on. Sensor noise also appears as shown in Fig. 2. It is necessary to
obtain a sufficient amount of data to represent the realistic system vari-
ations mentioned above for data-driven fault detection. Monte Carlo
In the Monte Carlo simulation, it is important to determine how
to vary the model parameters to represent the system variation. In
this study, random variations 𝛿𝜽 are added to the estimated model
parameter 𝜽̂ as 𝜽 = 𝜽̂ + 𝛿𝜽, and the problem is reduced to the selection
of 𝛿𝜽. The variation 𝛿𝜽 was chosen so that the variability of the sensor
values obtained by the Monte Carlo simulation would be similar to the
variability in the past static firing tests.
The procedure to obtain the set of 𝛿𝜽 for Monte Carlo simulations
is as follows:
Step. 1 Perform SLS for several temporary parameters 𝜽 = 𝜽̂ + 𝛥𝜽
that are randomly distributed around 𝜽. ̂ Obtain pairs of 𝛥𝜽 and 𝛥𝒖 as
𝛥𝒖 = SLS𝑁 (𝜽̂ + 𝛥𝜽) − SLS𝑁 (𝜽).
̂ (2)
Step. 2 From the pairs of 𝛥𝜽 and 𝛥𝒖, approximate the relationship
between 𝛥𝜽 and 𝛥𝒖 locally linearly as 𝛥𝒖 ≃ 𝐹 𝛥𝜽 using a linear
regression model.
N. Omata et al. Acta Astronautica 195 (2022) 234–242

Step. 3 Conduct singular value decomposition onto the matrix 𝐹 as

𝐹 = 𝑈 𝑆𝑉 ⊤ , and get low rank approximation as 𝐹 ♯ = 𝑈 ♯ 𝑆 ♯ 𝑉 ♯⊤ to obtain
the inverse matrix of 𝐹 .
Step. 4 Calculate the system variation in the past static firing tests,
𝛴𝑇 , and specify the 𝛿𝒖.
Step. 5 Find the parameter 𝛿𝜽 corresponding to the target value 𝛿𝒖
by the following equation:

𝛿𝜽 = 𝐹 ♯−1 𝛿𝒖 = 𝑉 ♯ 𝑆 ♯−1 𝑈 ♯⊤ 𝛿𝒖. (3)

The novel procedure to obtain the variation of the model parameters

is different from that of Park and Ahn, in which the difference is given
in the test sequence itself [18], and from the authors’ work, the range
of 𝛿𝜽 was determined manually [23].
Measurement noise was also considered in this study. The approxi-
mate magnitude of the measurement noise 𝜎𝑖 for sensor 𝑖 was estimated
based on the specifications of each sensor and the past static firing test
results; the noise 𝜀𝑖 following a normal distribution with zero as the
mean and the estimated magnitude 𝜎𝑖 as the standard deviation, was
added as follows:

𝑥 𝑖 = 𝑢𝑖 + 𝜀 𝑖 , 𝜀𝑖 ∼  (0, 𝜎𝑖2 ). (4)

Fig. 3 shows the histograms of simulated sensor values 𝒙 for both

normal and leak cases. The leak hole area 𝐴 for the leaked cases in
this figure is 0.5𝜋 mm2 . There is an overlap between the normal and
leaked sensor values for all sensors, although the size of the overlap
varies among sensors and among leak positions 𝑙. There are sensors
such as T5, where there is almost no difference in the distributions.
As long as this overlap exists, false alarms and/or missed alarms will
occur regardless of where the threshold is set as the redline. Leaks
of propellants cannot be detected by conventional red-line judgment
because the normal variation is larger than the effect of the leaks.

5. Methodology for supervised sensor placement optimization

In this section, the methodology for supervised sensor placement

optimization is proposed. First, the method of supervised fault detection
is described because the sensor placement optimization is based on the
performance of the fault detection.

5.1. Method of supervised fault detection

As observed in Fig. 3, it is not possible to detect leaks using Fig. 3. Histograms of simulated steady sensor values 𝒙 for the normal and leaked cases
conventional red-line judgment based on a sensor value alone. Another of 𝐴 = 0.5𝜋mm2 .
candidate for leak detection is principal component analysis (PCA),
a widely used unsupervised feature extraction method in various ap-
plications, including anomaly detection. PCA achieves the projection
of the maximum variance. Fig. 4 shows histograms of the first PCA
score for the same data as Fig. 3. There is a large overlap between the
distributions for normal and fault cases, which indicates the difficulty
of leak detection by PCA. Note that, in this figure, PCA was trained only
on the normal training data, and the normal and fault test data were
projected onto the same principal component vector, but the result was
almost the same even if the PCA was conducted on both normal and
fault training data.
The problems observed in Figs. 3 and 4 can be understood as
follows: Fig. 5 shows schematic of the multidimensional space of sensor
values for 𝐾 = 2 case. The normal cases lie in the normal subspace
Fig. 4. Histogram of the 1st PCA score for the normal and leaked cases of 𝐴 = 0.5𝜋mm2 .
𝑆𝑁 , and the fault cases are in the anomaly subspace 𝑆𝐴 . In Fig. 5(a),
which focuses on a single sensor, the ranges where the sensor values
normally exist are indicated by the dashed black line, and the range for
anomalous cases is indicated by the dashed red line. For both sensors, components of normal variation. One may consider projection onto
the two ranges overlap, and a red-line judgment is impossible, as is the anomaly vector 𝒂,
̄ which is the direction between the normal and
the case in Fig. 3. Similarly, the result of applying PCA to the same
anomaly subspaces, to resolve this problem. However, as shown in
situation is shown in Fig. 5(b). The projected regions for 𝑆𝑁 and 𝑆𝐴
overlap with each other, and the fault cannot be detected. The problem Fig. 5(b), this is not a sufficient solution. The anomaly vector 𝒂̄ contains
in these cases is that each sensor value and PCA score contain the not only the direction of the vector 𝒑 peculiar to the fault but also the

237
N. Omata et al.
Fig. 5. Discrimination of faults by linear projection in the multidimensional space of
sensor values. (For interpretation of the references to color in this figure legend, the
reader is referred to the web version of this article.)
component of the variations under normal conditions, and the overlap
between the normal and anomalous cases still exists.
In the scenario of Fig. 5(c), however, the direction represented
by the blue line isolates 𝑆𝑁 and 𝑆𝐴 , which enables fault detection.
This is achieved by projective transformation of the multiple sensor
values onto the projection vector 𝒑, which characterizes the difference
between the distributions of the normal and anomalous data. The
projection result
∑
𝐾
𝑠 = 𝒑⊤ 𝒙 = 𝑝𝑖 𝑥𝑖 ,
𝑖=1
represents the axis drawn by the blue line and indicates how anomalous
the sensor values are. Let the value 𝑠 be called the anomaly score. The
anomaly score characterizes the difference between normal and fault
cases, and this enables fault detection.
The projection vector 𝒑 can be calculated as follows, assuming
that the anomaly subspace 𝑆𝐴 is a space shifted in the direction of
the anomaly vector 𝒂̄ from the normal subspace 𝑆𝑁 . There are two
Acta Astronautica 195 (2022) 234–242
objectives for the optimization of 𝒑. First, the projection with the
anomaly vector
∑
𝐾
𝜙𝒂̄ = 𝒑⊤ 𝒂̄ = 𝑝𝑖 𝑎̄𝑖 (6)
𝑖=1
is maximized such that the anomaly score under anomalous conditions
increases. The anomaly vector 𝒂̄ is obtained as the difference between
the mean of the simulated normal and fault sensor values:
𝒂̄ = 𝒖̄ 𝐴 − 𝒖̄ 𝑁 (7)
Second, in contrast, the projection of the normal subspace on the vec-
tor, 𝜙𝑁 , is minimized to decrease the effect of variation under normal
conditions. The projection 𝜙𝑁 is defined by the standard deviation of
the normal sensor values projected onto 𝒑:
√
√
√
√ 1 ∑ ⊤
𝑀
𝜙𝑁 = √ (𝒑 𝒙𝑁𝑗 − 𝒑⊤ 𝒙̄ 𝑁 )2 (8)
𝑀 𝑗=1
The projection vector 𝒑 was obtained to minimize the ratio of the two
projective results.
( )
𝜙𝑁
𝒑 = arg min . (9)
𝒑 𝜙𝒂̄
This optimization problem can be solved analytically as
−1
𝒑 = 𝛴𝑁 𝒂,
̄ (10)
where 𝛴𝑁 is the covariance matrix of the normal sensor values:
1 ∑
𝑀
𝜮𝑁 = (𝒙 − 𝒙̄ 𝑁 )(𝒙𝑁𝑗 − 𝒙̄ 𝑁 )⊤ . (11)
𝑀 𝑗=1 𝑁𝑗
This formulation is almost equivalent to Fisher’s linear discriminant
analysis (LDA) [24] and is also applicable when there are more than
two sensors.
5.2. Sensor placement optimization based on fault detection performance
Efficient sensors for leak detection are selected by optimizing the
combination of sensors to maximize the performance of leak detec-
tion and minimize the number of sensors. There are 2𝐾 available
combinations of sensors, and the brute force approach to determine
the optimum combinations is time-consuming. Therefore, a greedy
approach was employed, and the sensors were removed one by one.
First, one sensor was removed from the list of sensors used for the
detection. There are 𝐾 candidate sensors to be removed, and 𝐾 tem-
porary detectors using 𝐾 − 1 sensors can be obtained. The performance
score of detection was calculated for each detector, and the detector
with the highest score was adopted as the detector using 𝐾 − 1 sensors.
Next, another sensor was removed from the obtained 𝐾 − 1 sensor
detector. All of the 𝐾 − 1 candidates were tested, and the one with
the highest score was adopted as the detector using 𝐾 − 2 sensors. By
repeating the same procedure, the number of sensors can be reduced
individually.
In this study, the area under the curve (AUC) score of the receiver
operating characteristic (ROC) curve (ROC-AUC score) was used to
measure the detection performance. The number of sensors can be re-
duced while maintaining the ROC-AUC score. The ROC curve represents
(5)
the characteristics of a detection model when the threshold for the
alarm is changed, with the false positive rate (FPR) on the horizontal
axis and the true positive rate (TPR) on the vertical axis. In general,
the model with the curve on the upper left side, that is, the model that
achieves high TPR with low FPR, has better performance. The ROC-
AUC score is the area under the ROC curve, where larger values indicate
better performance. The maximum value of 1 indicates that there is a
threshold for perfect detection, while 0.5 indicates that the model is
meaningless and no better than random.
238
N. Omata et al. Acta Astronautica 195 (2022) 234–242

Fig. 6. Histogram of the proposed anomaly score 𝑠 for the normal and leaked cases Fig. 7. ROC curves for different leak location 𝑙.
of 𝐴 = 0.5𝜋mm2 .

6. Demonstration of the supervised sensor placement optimiza-

tion

In this section, the supervised leak detection and effective sensor

placement optimization are demonstrated using the results of Monte
Carlo simulations. As described in Section 4, leakage due to the four
leak hole areas 𝐴 ∈ {0.25𝜋, 0.5𝜋, 0.75𝜋, 1.0𝜋} [mm2 ] at four different
leak locations 𝑙 ∈ {Leak 1, 2, 3, 4} were simulated in addition to the
normal condition. Monte Carlo simulations were conducted 1,000 times
for each case using the same 𝛿𝜃. Data corresponding to 𝑀 = 800 of 𝛿𝜃
were used as training data, and the remaining 200 were used as test
data.
The implementation details for obtaining 𝛿𝜃 for the Monte Carlo
Fig. 8. ROC curves for different leak magnitude 𝐴.
simulation are as follows: In Step. 1, 𝛥𝜽 required to estimate the linear
relationship 𝐹 was sampled independently from a uniform distribution
whose mean was zero for each sensor. The range of the uniform
distribution was set to ±10% for most parameters and ±1% for some
highly sensitive parameters that were determined based on the expert
knowledge of the RSR engine and the simulation model. The number
of samples was set to 1,000, which is equal to the number of simulated
cases 𝑀0 . In Step. 2, ridge regression was employed to approximate the
linear relationship between 𝛥𝜽 and 𝛥𝒖. In Step. 3, the rank of 𝐹 ♯ was
set to 15 so that its minimum singular value would be approximately
±1% of the maximum singular value. In Step. 4, the results of the static
firing test of the RSR engine conducted in 2014 and 2015 [4] were used
to calculate the covariance matrix 𝛴𝑇 of the system variance. As part of
this firing test, there were 19 tests in which a 100% thrust of 10 s was
repeated five times at intervals. The covariance of the system variance
was evaluated using the results of the last combustion of 100% thrust
for the 19 tests. Fig. 9. ROC curves for different methods of the fault detection. (For interpretation of
The effectiveness of the supervised leak detection described in the references to color in this figure legend, the reader is referred to the web version
Section 5.1 is firstly demonstrated. According to Eqs. (5)–(11), the of this article.)
projection vector 𝒑 was obtained by referring to the training data for
the normal and leaked conditions, and the anomaly score 𝑠 of the
simulated test case was obtained by projecting the sensor data 𝒙. Note leaks are detectable with a false alarm rate of approximately 10%. The
that the sensor data were standardized before the calculation of the ROC curves for the different leak sizes are shown in Fig. 8. Almost all
projection vector 𝒑. cases of leaks can be correctly detected when the hole size 𝐴 is no
The histograms of the anomaly score 𝑠 for the normal and leaked smaller than 0.75𝜋 mm2 . In the case of 𝐴 = 0.25𝜋 mm2 , which is the
cases are shown in Fig. 6. The area of the leak hole 𝐴 is fixed at most difficult case to detect, the false alarm rate increases to 20% in
0.5𝜋 mm2 , as shown in Figs. 3 and 4. In contrast to Figs. 3 and 4, order to detect 80% of the leaked cases, and a leak of 𝐴 = 0.25𝜋 mm2
the distributions of the anomaly scores for Leaks 1 and 2 are clearly would be the approximate detection limit of this method. Nevertheless,
divided from the normal cases. Leaks 3 and 4 also have a distribution the results obtained here indicate that a leak can be detected by the
different from that of the normal case, although there is a small overlap supervised linear transformation in the multidimensional space of the
with the normal case. The ROC curves corresponding to the four leak sensor values.
locations are shown in Fig. 7. These ROC curves include all four leak A comparison with the existing method is shown in Fig. 9. The solid
hole sizes, 𝐴 = 0.25𝜋–1.0𝜋 mm2 . Almost all leaks were detectable for black line is the ROC curve for the proposed supervised method for all
Leak 1, and the detection performance decreased in the order of Leak 2, the test cases, including all leak hole sizes and locations. The result of
4, and 3. Even for the worst case, Leak 3, approximately 80% of the the red-line judgment currently used in rocket engine PHM is shown by

239
N. Omata et al.
Fig. 10. ROC-AUC score relative to the number of sensors.
Fig. 11. Histograms of anomaly score for selected sensors. 𝐴 = 0.5𝜋mm2 .
the dashed red line. A sensor to show the best detection performance
based on the red-line judgment was selected for this comparison. As
shown in Fig. 9, the red-line judgment always shows a lower detection
performance than the proposed supervised method. The dotted blue
line represents the results of the first PCA. The FPR and TPR in the
ROC curve were almost the same, indicating that leak detection could
not be realized by PCA.
Sensor placement optimization is performed based on the leak
detection performance, as described in Section 5.2. Identified order of
effectiveness is shown in Table 2, and Fig. 10 summarizes the relation-
ship between the number of sensors and the ROC-AUC score. It was
found that the detection performance did not change significantly, even
if the number of utilized sensors 𝐾𝑢 is reduced to 11. Further reduction
of the sensor results in degradation of the detection performance. For
models with 11 ≤ 𝐾𝑢 < 27, the detection performance is slightly better
than that using all sensors (𝐾𝑢 = 27), even though the number of
sensors is reduced. The comparison shown here indicates the effect of
the curse of dimensionality. Fig. 11 shows a histogram of the anomaly
score using the selected 11 sensors. The overlap between distributions
in Fig. 11 is comparably small to that in Fig. 6, where all sensors are
utilized. Fig. 12 shows the ROC curves for leak detection using different
numbers of sensors. It can be seen that the ROC curves show almost the
same performance. The advantage of the supervised sensor placement
optimization is evident.
The selected sensors may also be useful in identifying the location of
the leak, which reduces maintenance costs significantly. The difference
between the normal and fault cases under the ideal stationary condition
is evaluated as follows:
𝒂 = SLS𝐴 (𝜽̂ + 𝛿𝜽, 𝐴, 𝑙) − SLS𝑁 (𝜽̂ + 𝛿𝜽).
A scatter plot of 𝒂 for two of the selected sensors, Feng and T5, is shown
in Fig. 13. It is found that 𝒂 is classified into four groups depending on
Acta Astronautica 195 (2022) 234–242
Fig. 12. ROC curves for different numbers of used sensors.
Fig. 13. Scatter plot of the changes when leak occurs 𝒂.
the leak location, as identified by the color. In each location, 𝒂 is also
divided into four, depending on the area of the leak hole 𝐴. Therefore,
by constructing a multi-class detector using 𝒂, it is possible to determine
the location of a leak.
7. Conclusion
Fuel leak is a major failure mode in a liquid rocket engine using
cryogenic propellants. To reduce the time and cost of maintenance,
sensor placement optimization for leak detection and diagnosis was
studied for a reusable liquid rocket engine using liquid hydrogen and
oxygen as propellants. The training data for normal and fault conditions
were obtained using a system-level simulation based on reduced-order
models for each rocket engine component and leak model. The leak
model was introduced based on the experimental results of cryogenic
physics. The relationship between the model parameters in the system-
level simulation and system variation observed in the static firing tests
was modeled, and Monte Carlo simulations were conducted to generate
the training data with realistic system variation.
It was demonstrated that leaks could be detected by performing a
supervised linear projective transformation for the stationary sensor
values, whereas conventional red-line judgment and PCA could not.
Optimum sensors for leak detection were selected using the greedy
approach based on the ROC-AUC score for detection. It was shown that
the performance of the detection remained almost the same even if only
11 out of the 27 sensors were used. Using two of the selected sensors,
(12) the location of the leak could be identified from the difference from the
normal cases. The proposed methodology is robust and can be applied
to any industrial systems as well as any rocket engines.
240
N. Omata et al. Acta Astronautica 195 (2022) 234–242

Acknowledgment [6] M. Sato, T. Hashimoto, M. Shiga, T. Soma, T. Kimura, M. Shin-ichi, Anomaly

detection of liquid propellant rocket engine using system invariant analysis
technology, in: Proceedings of the 70th International Astronautics Congress,
This work was supported by JSPS, Japan KAKENHI grant number
2019, IAC–19–C4.1.7.
JP20K14954. The authors would like to acknowledge Dr. Tomoyuki [7] S. Tsutsumi, M. Hirabayashi, D. Sato, K. Kawatsu, M. Sato, T. Kimura, T.
Hashimoto, Dr. Masaki Sato, and Dr. Toshiya Kimura in JAXA for their Hashimoto, M. Abe, Data-driven fault detection in a reusable rocket rngine
expert advice on the rocket engine. using bivariate time-series analysis, Acta Astronaut. 179 (2021) 685–694, http:
//dx.doi.org/10.1016/j.actaastro.2020.11.035.
[8] M. Schwabacher, N. Oza, B. Matthews, Unsupervised anomaly detection for
Nomenclature
liquid-fueled rocket propulsion health monitoring, J. Aerosp. Comput. Inf.
Commun. 6 (7) (2009) 464–482, http://dx.doi.org/10.2514/1.42783.
𝒂 Difference between normal and fault cases
[9] Y. Nie, Y. Cheng, J. Wu, Liquid-propellant rocket engine online health condition
𝐴 Area of leak hole monitoring base on multi-algorithm parallel integrated decision-making, Proc.
𝐹 Linear relationships approximated between 𝜽 and 𝒖 Inst. Mech. Eng. G: J. Aerosp. Eng. 231 (9) (2017) 1621–1633, http://dx.doi.
𝑘𝑑𝑝 Correction coefficient in orifice model org/10.1177/0954410016656878.
𝐾 Number of sensors [10] J. Wu, Y. Zhang, Q. Chen, Real-time fault detection algorithm based on
neural networks for liquid propellant rocket engine, in: 32nd Joint Propulsion
𝑙 Indicator for location of the leak
Conference and Exhibit, 1996, p. 2591, http://dx.doi.org/10.2514/6.1996-2591.
𝑚̇ Mass flow rate [11] F. Di Matteo, M. De Rosa, M. Onofri, Start-up transient simulation of a
𝑀 Number of cases simulated using Monte Carlo liquid rocket engine, in: Proceedings of the 47th AIAA/ASME/SAE/ASEE Joint
simulation Propulsion Conference & Exhibit, 2011, p. 6032, http://dx.doi.org/10.2514/6.
𝒑, 𝑝𝑖 Projection vector and its component 2011-6032.
[12] A. Ogawara, C. Sezaki, Development and evaluation of transient analysis tool for
𝑃 Pressure
rocket engine, in: Proceedings of the 5th European Conference for Aeronautics
𝑅 Resistance coefficient in orifice model and Space Sciences, EUCASS, 2013, p. 127.
𝑠 Anomaly score [13] P. Sierra, J.M. Tizón, J. Vilas, J.F. Moral, Efficient simulation of liquid pro-
𝑆𝐴 , 𝑆𝑁 Subspace of sensor values for fault and normal cases pellant rocket engine cycle, in: Proceedings of the 53rd AIAA/SAE/ASEE Joint
𝒖, 𝑢𝑖 Sensor values simulated by SLS and its component Propulsion Conference, 2017, p. 5007, http://dx.doi.org/10.2514/6.2017-5007.
[14] D. Satoh, S. Tsutsumi, M. Hirabayashi, K. Kawatsu, T. Kimura, Estimating
𝒖𝐴 , 𝒖𝑁 Fault and the normal case simulated by SLS
model parameters of liquid rocket engine simulator using data assimilation, Acta
𝒖̄ 𝐴 , 𝒖̄ 𝑁 Averaged fault and normal cases simulated by the SLS Astronaut. 177 (2020) 373–385, http://dx.doi.org/10.1016/j.actaastro.2020.07.
𝒙, 𝑥𝑖 Vector of sensor values and component 037.
𝒙𝐴 , 𝒙𝑁 Sensor values for fault and normal case [15] A. Iannetti, J. Marzat, H. Piet-Lahanier, C. Sarotte, G. Ordonneau, C. de la
𝒙̄ 𝐴 , 𝒙̄ 𝑁 Averaged sensor values for fault and normal cases Hunière, Promising HMS approaches for liquid rocket engines, in: Proceedings
of the 7th European Conference for Aeronautics and Space Sciences, EUCASS,
𝛿𝜽, 𝛿𝒖 Variation in 𝜽 and 𝒖 for Monte Carlo simulations 2017, p. 417, http://dx.doi.org/10.13009/EUCASS2017-417.
𝛥𝜽, 𝛥𝒖 Variation in 𝜽 and 𝒖 temporally introduced to [16] J. Cha, S. Ko, S.-Y. Park, E. Jeong, Fault detection and diagnosis algorithms
approximate 𝐹 for transient state of an open-cycle liquid rocket engine using nonlinear Kalman
𝜀𝑖 Measurement noise for sensor 𝑖 filter methods, Acta Astronaut. 163 (2019) 147–156, http://dx.doi.org/10.1016/
j.actaastro.2019.03.075.
𝜽 Set of all parameters for SLS
[17] K. Kawatsu, S. Tsutsumi, M. Hirabayashi, D. Sato, Model-based fault diagnostics
𝜽̂ Most likely parameters set for SLS in an electromechanical actuator of reusable liquid rocket engine, in: Proceedings
𝜌 Density of the AIAA SciTech 2020 Forum, 2020, p. 1624, http://dx.doi.org/10.2514/6.
𝜎𝑖 Standard deviation of the measurement noise for sensor 2020-1624.
𝑖 [18] S.-Y. Park, J. Ahn, Deep neural network approach for fault detection and diag-
nosis during startup transient of liquid-propellant rocket engine, Acta Astronaut.
𝛴𝑁 Covariance matrix of normal sensor values
177 (2020) 714–730, http://dx.doi.org/10.1016/j.actaastro.2020.08.019.
𝛴𝑇 Covariance matrix of system variation in the past test [19] P. Suter, Representation of pump characteristics for calculation of water hammer,
𝜙𝒂̄ , 𝜙𝑁 Projection result of anomaly vector 𝒂̄ and normal Sulzer Tech. Rev. 1966 (4) (1966) 45–48.
subspace 𝑆𝑁 [20] B.J. McBride, M.J. Zehe, S. Gordon, NASA Glenn Coefficients for Calculating
Thermodynamic Properties of Individual Species, NASA Glenn Research Center,
Declaration of competing interest 2002.
[21] D.R. Bartz, Survey of the relationship between theory and experiment for
convective heat transfer from rocket combustion gases, in: S.S. Penner (Ed.),
The authors declare that they have no known competing finan- Advances in Rocket Propulsion, AGARD, Technivision Services, Manchester
cial interests or personal relationships that could have appeared to England, 1968, pp. 291–381.
influence the work reported in this paper. [22] H. Kobayashi, Y. Naruo, Y. Maru, Y. Takesaki, K. Miyanabe, Experiment of cryo-
compressed (90-MPa) hydrogen leakage diffusion, Int. J. Hydrogen Energy 43
(37) (2018) 17928–17937, http://dx.doi.org/10.1016/j.ijhydene.2018.07.145.
References
[23] N. Omata, D. Satoh, S. Tsutsumi, K. Kawatsu, M. Abe, Efficient sensor placement
for health management of a rocket engine using Monte Carlo simulation, in:
[1] K. Manohar, B.W. Brunton, J.N. Kutz, S.L. Brunton, Data-driven sparse sensor
Proceedings of the Annual Conference of the PHM Society 2020, Vol. 12, no. 1,
placement for reconstruction: Demonstrating the benefits of exploiting known
2020, p. 1135, http://dx.doi.org/10.36001/phmconf.2020.v12i1.1135.
patterns, IEEE Control Syst. Mag. 38 (3) (2018) 63–86, http://dx.doi.org/10.
[24] R.A. Fisher, The use of multiple measurements in taxonomic problems, Ann.
1109/MCS.2018.2810460.
Eugen. 7 (2) (1936) 179–188, http://dx.doi.org/10.1111/j.1469-1809.1936.
[2] G.V. Trunk, A problem of dimensionality: A simple example, IEEE Trans. Pattern
tb02137.x.
Anal. Mach. Intell. PAMI-1 (3) (1979) 306–307, http://dx.doi.org/10.1109/
TPAMI.1979.4766926.
[3] M. Sato, T. Hashimoto, S. Takada, T. Kimura, T. Onodera, Y. Naruo, T. Yagishita,
K.-I. Niu, T. Kaneko, K. Obase, Development of main propulsion system for Noriyasu Omata received his Ph.D. degree in Engineering
reusable sounding rocket: Design considerations and technology demonstration, from the University of Tokyo, Japan, in 2019. He worked
Tran. Jpn. Soc. Aeronaut. Space Sci. 12 (ists29) (2014) Tm_1–Tm_6, http://dx. as a postdoctoral fellow at the Japan Aerospace Exploration
doi.org/10.2322/tastj.12.Tm_1. Agency (JAXA) for two years and is working as a researcher
[4] T. Kimura, T. Hashimoto, M. Sato, S. Takada, S. Moriya, T. Yagishita, Y. Naruo, in JAXA. His research interests include data-driven analysis
H. Ogawa, T. Ito, K. Obase, et al., Reusable rocket engine: Firing tests and of flow dynamics and the health management of aerospace
lifetime analysis of combustion chamber, J. Propul. Power (2016) 1087–1094, systems using machine learning.
http://dx.doi.org/10.2514/1.B35973.
[5] W.A. Maul, G. Kopasakis, L.M. Santi, T.S. Sowers, A. Chicatelli, Sensor selection
and optimization for health assessment of aerospace systems, J. Aerosp. Comput.
Inf. Commun. 5 (1) (2008) 16–34, http://dx.doi.org/10.2514/1.34677.

241
N. Omata et al. Acta Astronautica 195 (2022) 234–242

Daiwa Satoh received his Master of Engineering from Kaname Kawatsu received the Master of Engineering in
Hokkaido University, Japan, in 2012. He joined Hitachi, Ltd. Aerospace Engineering from Tohoku University in 2006
in 2012 as a researcher, and developed fans, flow channels, and started his career at the Japan Aerospace Exploration
and heat exchangers of air conditioners to improve perfor- Agency. He is in charge of research and development of risk
mance. In 2018, he was assigned to the Japan Aerospace assessment, design evaluation, and health management with
Exploration Agency (JAXA) and worked as a researcher on spacecraft and launch vehicles by utilizing the multi-physics
a reusable rocket engine. He has developed a method of system-level modeling and simulation technique.
probabilistic visualization for explanatory variables using
machine learning in Hitachi, Ltd. since 2021. His research
interests include system-level modeling and simulation, data
assimilation, and fault detection and diagnosis with machine
learning.
Masaharu Abe received his Master’s Degree in the Depart-
ment of Chemistry, Graduate School of Science at Kyoto
Seiji Tsutsumi received his Ph.D. degree in Aerospace University, Japan, in 2006. He is currently a chief engineer
Engineering from the University of Tokyo, Japan, in 2006. at Ryoyu Systems Co., Ltd. His interests focus on model-
He is an associate senior researcher in the Research and De- based and data-driven fault detection in launch vehicles and
velopment Directorate, Research Unit III, Japan Aerospace spacecraft.
Exploration Agency (JAXA). His research interests include
aeroacoustics, high-speed aerodynamics, CFD techniques us-
ing HPC, and the application of machine learning to health
management in aerospace systems.

242