Automatica 126 (2021) 109451

Contents lists available at ScienceDirect

Automatica
journal homepage: www.elsevier.com/locate/automatica

Brief paper

Reduced-dimensional reinforcement learning control using singular

perturbation approximations✩
∗
Sayak Mukherjee a , , He Bai b , Aranya Chakrabortty a
a
Department of Electrical and Computer Engineering, North Carolina State University, USA
b
Mechanical and Aerospace Engineering Department, Oklahoma State University, USA

article info a b s t r a c t

Article history: We present a set of model-free, reduced-dimensional reinforcement learning (RL) based optimal
Received 8 October 2019 control designs for linear time-invariant singularly perturbed (SP) systems. We first present a state
Received in revised form 10 November 2020 feedback and an output feedback based RL control design for a generic SP system with unknown
Accepted 11 December 2020
state and input matrices. We take advantage of the underlying time-scale separation property of the
Available online xxxx
plant to learn a linear quadratic regulator (LQR) for only its slow dynamics, thereby saving significant
Keywords: amount of learning time compared to the conventional full-dimensional RL controller. We analyze the
Reinforcement learning sub-optimality of the designs using SP approximation theorems, and provide sufficient conditions for
Linear quadratic regulator closed-loop stability. Thereafter, we extend both designs to clustered multi-agent consensus networks,
Singular perturbation where the SP property reflects through clustering. We develop both centralized and cluster-wise block-
Model-free control
decentralized RL controllers for such networks, in reduced dimensions. We demonstrate the details
Model reduction
of the implementation of these controllers using simulations of relevant numerical examples, and
compare them with conventional RL designs to show the computational benefits of our approach.
© 2020 Elsevier Ltd. All rights reserved.

1. Introduction the full-dimensional plant. The specific property of the plant

Reinforcement Learning (RL), originally introduced in Sutton
and Barto (1998), has recently seen a resurgence in optimal
control of dynamical systems through a variety of papers such
as Jiang and Jiang (2012), Lewis and Vrabie (2009), Liu and
Wei (2014), Vamvoudakis (2017), Vrabie et al. (2009), Wu and
Luo (2012) using solution techniques such as adaptive dynamic
programming (ADP), actor–critic methods, Q-learning, etc. Curse
of dimensionality, however, continues to be an ongoing debate
for all of these RL-based control designs. Depending on the size
and complexity of the plant, it may take an unacceptably long
amount of time to perform system exploration. Our goal in
this paper is to counteract this problem by exploiting certain
physical characteristics of the plant dynamics so that learning
only a reduced-dimensional controller is sufficient for stabilizing
✩ The work of the third author was supported by the National Science
Foundation ECCS grant 1940866. The material in this paper was partially
presented at: the 57th IEEE Conference on Decision and Control, December 17–
19, 2018, Miami Beach, Florida, USA, and the 2019 American Control Conference
(ACC), July 10–12, 2019, Philadelphia, PA, USA. This paper was recommended for
publication in revised form by Associate Editor Kyriakos G. Vamvoudakis under
the direction of Editor Miroslav Krstic.
∗ Corresponding author.
E-mail addresses: smukher8@ncsu.edu (S. Mukherjee), he.bai@okstate.edu
(H. Bai), achakra2@ncsu.edu (A. Chakrabortty).
model that we study here is based on singular perturbations (SP),
i.e., plants whose dynamics are separated into two time-scales.
Traditionally, SP theory has been used for model reduction (Chow
and Kokotovic (1985), Kokotovic et al. (1976)), and control (Chow
and Kokotovic (1976)) of large-scale systems, but only by using
knowledge of the full plant model. Its extension to model-free
control using RL has not been addressed. To bridge this gap, we
present several sets of RL-based control designs where we exploit
the underlying SP property of the plant to learn a controller
for only its dominant slow time-scale dynamics, thereby saving
significant amount of learning time. We provide sub-optimality
and stability results for the resulting closed-loop system.
The main contributions are as follows. Three distinct RL con-
trol designs for singularly perturbed systems are presented. The
first design assumes that the slow state variable is either di-
rectly measurable, or can be constructed from the measurements
of the full state vector. Using this assumption, we develop a
modified ADP algorithm which learns a reduced-dimensional RL
controller using only feedback from the slow state variables.
The controller is shown to guarantee closed-loop stability of
the full-dimensional system if the fast dynamics are stable. The
second design extends this algorithm to output feedback control
using a neuro-adaptive state estimator (Abdollahi et al., 2006).
The third design shows the relevance of these two designs to
SP models of multi-agent consensus networks where time-scale

https://doi.org/10.1016/j.automatica.2020.109451
0005-1098/© 2020 Elsevier Ltd. All rights reserved.
S. Mukherjee, H. Bai and A. Chakrabortty
separation arises due to clustering of the network nodes. Along
with a centralized design, a variant is proposed that imposes
a block-diagonal structure on the RL controller to facilitate its
implementation. Numerical results show that our approach saves
significant amount of learning time than the conventional RL
while still maintaining a modest closed-loop performance. All the
designs are described by implementable algorithms together with
theoretical guarantees.
The first design has been presented as a preliminary result
in our recent conference paper Mukherjee et al. (2018). The
second design, however, is completely new. The multi-agent
RL controllers, which were presented only for scalar dynamics
in Mukherjee et al. (2018, 2019), are now extended to vector-
dimensional states. Moreover, unlike prior results, the consensus
model here is more generic as we allow each node to have
self dynamics. The simulation examples presented in Section 5
are much larger-dimensional than in Mukherjee et al. (2018) to
demonstrate the numerical benefits of the designs.
Notations: RH∞ is the set of all proper, real and rational stable
transfer matrices; ⊗ denotes Kronecker product; 1n denotes a
column vector of size n with all ones; ∪ denotes union opera-
tion of sets; blkdiag(m1 , . . . , mn ) denotes a block-diagonal matrix
with m1 , . . . , mn as its block diagonal elements; |M | denotes the
cardinality of set M; ∥.∥ denotes Euclidean norm of a vector and
Frobenius norm of a matrix unless mentioned otherwise.
2. Problem formulation
Consider a linear time-invariant (LTI) system
ẋ = Ax + Bu, x(0) = x0 , q = C x, (1)
where, x ∈ R is the state, u ∈ R is the control input, and q ∈ Rp
n m
is the output. We assume that the matrices A and B are unknown,
although n, m and p are known. The following assumption is
made.
Assumption 1. The system with state–space model (1) exhibits a
singular perturbation property, i.e., there exist a small parameter
1 ≫ ϵ > 0 and a similarity transform T such that by defining
y ∈ Rr and z ∈ Rn−r as
[ ] [ ]
y T
=Tx= x, (2)
z G
the state–space model (1) can be rewritten as
ẏ = A11 y + A12 z + B1 u, y(0) = Tx0 = y0 , (3a)
ϵ ż = A21 y + A22 z + B2 u, z(0) = Gx0 = z0 , (3b)
[ ] [ ]
y y
q = CT −1 =C . (3c)
z z
In the transformed model (3), y(t) represents the slow states
and z(t) represents the fast states. Since A and B are unknown,
the matrices A11 , A12 , A21 , A22 , B1 and B2 are unknown as well.
2.1. Problem statement for state-feedback RL
P1. Learn a control gain K ∈ Rm×r for the singularly perturbed
system (3) without knowing the model using online measure-
ments of u(t) and y(t) such that
u(t) = −Ky(t) = −KTx(t) (4)
minimizes
∫ ∞
J(y(0); u) = (yT Qy + uT Ru)dt , (5)
0
2
Automatica 126 (2021) 109451
s.t. A − BKT ∈ RH∞ . (6)
We assume (A, B) to be stabilizable. We consider y(t) to be di-
rectly measurable, or x(t) to be measurable (i.e. C = I) and T to
be known so that y(t) can be computed at all time t. This is not a
restrictive assumption as in many SP systems the identity of the
slow and fast states are often known a priori (Khalil & Kokotovic,
1978) even if the model is unknown. In some cases, for example
as in the multi-agent model that will be shown later in the paper,
offline knowledge of certain structural properties of the system
can enable the designer to construct T using measurements of
x(t), even when A and B are unknown. The benefit of using y(t)
as the feedback variable is that one has to learn only a (m × r)
matrix instead of a (m × n) matrix if full state feedback x(t) was
used. This will improve the learning time, especially if r ≪ n.
Before proceeding with the control design, we make the following
assumption.
Assumption 2. A22 in Eq. (3b) is Hurwitz.
This assumption means that the fast dynamics of (3) are stable,
which allows us to skip feeding back z(t) in (4).
2.2. Problem statement for output feedback RL
P2. Considering that q(t) is measured and C is known, but A
and B are both unknown in (1), estimate the states ŷ(t), ẑ(t) (or,
equivalently estimate x̂(t) and compute ŷ(t) = T x̂(t) assuming
that T is known), learn a controller K ∈ Rm×r using online
measurements of q(t) ∫and u(t) such that u = −K ŷ = −KT x̂
∞
minimizes J(y(0); u) = 0 (yT Qy + uT Ru)dt .
We assume (A, B) to be stabilizable, and (A, C ) to be detectable.
Our approach would be to estimate the slow states ŷ(t) without
knowing (A, B) using an observer employing a neural structure
that does not require exact information of the state dynamics, and
then using u(t) and ŷ(t) to learn the controller K using adaptive
dynamic programming.
We present the solutions for P1 and P2 with associated stabil-
ity proofs in the following two respective sections.
3. Reduced-dimensional state feedback RL
Following Khalil (2002), the reduced slow subsystem of (3) can
be defined by substituting ϵ = 0, resulting in
ẏs = As ys + Bs us , ys (0) = y(0), u = us + uf , (7)
−1 −1
where As = A11 − A12 A22 A21 and Bs = B1 − A12 A22 B2 . Since our
intent is to only use the slow variable for feedback, we substitute
the fast control input uf = 0, and the slow control input us = u.
If the controller were to use ys (t) for feedback then it would find
u = −K̄ ys (t) to solve:
∫ ∞
minimize J̄(ys (0); u) = (yTs Qys + uT Ru)dt , (8)
0
s.t. As − Bs K̄ ∈ RH∞ . (9)
The optimal solution for the above problem is given by the
following algebraic Riccati equation (ARE):
ATs P̄ + P̄As + Q − P̄Bs R−1 BTs P̄ = 0, K̄ = R−1 BTs P̄ ,
where P̄ = P̄ T ≻ 0. If As and Bs are unknown, then the RL
controller K̄ can be learned using measurements of ys (t) and of
an exploration input u(t) = u0 (t) by the ADP algorithm presented
in Jiang and Jiang (2017), which is a model-free version of Klein-
man’s algorithm (Kleinman, 1968). The control policy u0 (t) must
be persistently exciting, and can be chosen arbitrarily as long as
S. Mukherjee, H. Bai and A. Chakrabortty Automatica 126 (2021) 109451

Algorithm 1 SP-RL using slow dynamics theorem provides a sufficient condition that is required to achieve
Input: Measurements of y(t) and u0 (t) asymptotic stability for the (k + 1)th iteration of Algorithm 1
Step 1 - Data storage: Store data (i.e., y(t) and u0 (t)) for sufficiently large assuming that the control policy at the kth iteration stabilizes (3).
uniformly sampled time instants (t1 , t2 , · · · , tl ), and construct the following
matrices:
[ ]T
Theorem 3. Assume that the control policy u = −Kk y at the kth
δyy = y ⊗ y| t 1 +T
t1, · · · , y ⊗ y|
tl +T
tl , (11) iteration asymptotically stabilizes (3). Consider R ≻ 0 and Q ≻ 0
[∫ ]T with λmin (Q ) sufficiently large. Then the control policy at the (k+1)th
t1 +T ∫ tl +T
Iyy = t (y ⊗ y)dτ , · · · , t (y ⊗ y)dτ , (12) iteration given by u = −Kk+1 y is asymptotically stabilizing for
1 l
[∫ ]T (3) □
t +T ∫ t +T
Iyu0 = t 1 (y ⊗ u0 )dτ , · · · , t l (y ⊗ u0 )dτ , (13)
1 l

such that rank(Iyy Iyu0 ) = r(r + 1)/2 + rm satisfies.

Step 2 - Controller update: Starting with a stabilizing K0 , solve for K iteratively
(k = 0, 1, · · ·) following the update equation:
] v ec(Pk )
[ ]
δyy = −Iyy v ec(Qk ) .
[
−2Iyy (Ir ⊗ KkT R) − 2Iyu0 (Ir ⊗ R)
 v ec(Kk+1 )
    
Θk
The stopping criterion for this update is ∥Pk − Pk−1 ∥ < γ , where γ is a chosen
small positive threshold.
Step 3 - Applying control: After P and K converge, remove u0 and apply
u = −Ky.
the system states remain bounded. For example, one choice of u0
is a sum of sinusoidal signals.
In reality, however, ys is not accessible as ϵ ̸ = 0. We, therefore,
recall the following theorem from Chow and Kokotovic (1976),
which will allow us to replace ys (t) with y(t) in the learning
algorithm.
Theorem 1. (Chow & Kokotovic, 1976; Khalil, 2002) Consider the
two systems (3) and (7). There exists 0 < ϵ ∗ ≪ 1 such that for
all 0 < ϵ ≤ ϵ ∗ , the trajectories y(t) and ys (t) satisfy uniformly for
t ∈ [0, t1 ]
y(t) = ys (t) + O(ϵ ).
Algorithm 1 shows how the controller K is learned using y(t) and
u0 (t).
The condition rank(Θk ) = r(r + 1)/2 + rm can be satis-
fied, for example, by utilizing data from at least twice as many
sampling intervals as the number of unknowns. We next provide
the analytical guarantees of Algorithm 1 related to the SP-based
approximations.
3.1. Sub-optimality and stability analysis
The optimal controller parameters P , K can be written as P =
P̄ + ∆P , K = K̄ + ∆K , where P̄ , K̄ are the optimal solutions if ys (t)
were available for design, and ∆P , ∆K are matrix perturbations
resulting from the fact that ϵ ̸ = 0. The following theorem
establishes the sub-optimality of the learned controller using y(t).
Proof. Please see Theorem 4 in Mukherjee et al. (2018).
Remark 1 (Design Trade-off). The proof of Theorem 3 is based
on Lyapunov stability analysis, where Q compensates for the
error due to O(ϵ ) approximation of the fast dynamics such that
(14)
Q − O(ϵ ) ≻ 0. This translates to the requirement of a sufficiently
Φk
large λmin (Q ). In practice, assuming that a reliable upper bound
for ϵ is known to the designer (even if the exact value of ϵ may
not be known), one can start the off-policy RL iteration with
a Q such that λmin (Q ) is greater than that upper bound, and
adjust Q , if necessary, to obtain satisfactory stability and transient
performance. Note that the only underlying assumption about
the physics of the system is the existence of a strong time-scale
separation, i.e., the existence of a sufficiently small ϵ . Estimates of
ϵ , or at least of an upper bound for it, can be obtained from offline
spectral analyses of the state trajectories, or even from a subset
of known network parameters. Also note that our approach is
different from classical robust control designs where a nominal
model is used to compute stabilizing controls with a known upper
bound on the uncertainty. The physics-informed nature of our
design helps with scalability and stronger analytical guarantees.
Finally, the requirement for an initial stabilizing control is
common for policy iterations (Jiang & Jiang, 2012). For an open-
(10) loop stable system, learning can be theoretically achieved with-
out any initial stabilizing control. However, an initial stabilizing
control may improve convergence and accuracy.
4. Reduced-dimensional output feedback RL
We next address the RL design when the full state information
is not available. We consider that the system model is in the
singularly perturbed form (3), and then design an observer to
estimate the state x as x̂(t) = [ŷ(t); ẑ(t)]. The idea then is to
simply replace y(t) by ŷ(t) in Algorithm 1. If the system is not
directly in the singularly perturbed form (3), but in the generic
form (1), then a full-order observer is required to estimate x̂(t)
followed by computation of the slow state estimate ŷ(t) using
the knowledge of the projection T . In Section 4.2 we will present
one such observer. We first analyze the stability properties of the
output feedback design.

Theorem 2. Assume that ∥ys (t)∥ and ∥u0 (t)∥ are bounded for a
finite time t ∈ [0, t1 ]. The solutions of Algorithm 1 are given by
P = P̄ + O(ϵ ), K = K̄ + O(ϵ ), and J = J̄ + O(ϵ )
Proof. See theorems 2 and 3 in Mukherjee et al. (2018).
Theorem 2 shows that the controller obtained from Algorithm
1 is O(ϵ ) close to that obtained from the ideal design (where
ϵ = 0) which also holds for the respective costs. Moreover,
it is shown in Chow and Kokotovic (1976) that the reduced-
order control cost is O(ϵ )-perturbed from the full-order control
cost. This holds for the model-free case as well. As Algorithm 1
is constructed first by learning the ideal slow sub-system, and
then by replacing ys (t) with y(t) for the implementation, we
can quantify the sub-optimality of this approximation. The next
3
4.1. Sub-optimality and stability analysis
Lemma 1. Define e(t) = x(t) − x̂(t). If e is uniformly ultimately
bounded (UUB) with a bound b for all t ≥ t0 + T for some initial
time t0 , then there exist positive constants ϵ ∗ and k such that for all
0 < ϵ ≤ ϵ∗
∥ŷ(t) − ys (t)∥ ≤ k̄|ϵ| + b := c(ϵ, b) (15)
holds uniformly for t ∈ [t2 , t1 ].
Proof. Since e(t) is UUB, there exists positive constants b and
b̂, independent of t0 ≥ 0, and for every a ∈ (0, b̂), there exists
T1 = T1 (a, b), independent of t0 , such that ∥ŷ(t0 ) − y(t0 )∥ ≤ a,
S. Mukherjee, H. Bai and A. Chakrabortty Automatica 126 (2021) 109451

which implies that 4.2. Neuro-adaptive observer

∥ŷ(t) − y(t)∥ ≤ b, ∀t ≥ t0 + T1 := t2 .
From Theorem 1, it follows that there exist positive constants k
and p such that,
∥y(t) − ys (t)∥ ≤ k̄|ϵ| ∀t ∈ [t0 , t1 ], t1 > t2 , ∀|ϵ| < p.
Combining (16) and (17), for t ∈ [t2 , t1 ] we have
∥ŷ(t) − ys (t)∥ ≤ k̄|ϵ| + b := c(ϵ, b).
This completes the proof. □
Corollary 1. If e(t) = O(ϵ ) for t ∈ [t2 , t1 ], then ŷ(t) = ys (t) + O(ϵ ).
Proof. The proof directly follows from Lemma 1. □
We know that if ys (t) were available for feedback then P̄ , K̄
would be the optimal solutions. However, due to the state esti-
mation error bound b and the singular perturbation error O(ϵ ),
the actual solutions are given as P = P̄ + ∆P, K = K̄ + ∆K , where
∆P and ∆K are matrix perturbations resulting from non-ideal
feedback.
(16) A candidate observer to estimate ŷ(t) without knowing (A, B)
is the neuro-adaptive observer proposed in Abdollahi et al. (2006)
that employs a neural network (NN) structure to account for
the lack of dynamic model information. This observer guarantees
boundedness of e(t), which, with proper tuning, can also be made
(17)
arbitrarily small. Recalling the mechanism of this observer, we
rewrite (1) as
ẋ = Âx + (Ax − Âx) + Bu, q = C x, (19)
(18)   
g(x,u)
where  is a Hurwitz matrix, and (C , Â) is observable. We do not
have proper knowledge about g(x, u), and a NN with sufficiently
large number of neurons can approximate g(x, u), as g(x, u) =
W σ (V x̄) + η(x). Here, x̄ = [x, u], while σ (.) and η(x) are the
activation function and the bounded NN approximation error,
respectively. W and V are the ideal fixed NN weights. We choose
G such that Ac = Â − GC is Hurwitz. The observer dynamics follow
as
x̂˙ = Âx̂ + g(x̂, u) +G(q − C x̂), q̂ = C x̂, (20)
  
=Ŵ σ (V̂ x̄)
ˆ

Proposition 1. Perturbations ∆P , ∆K are bounded, i.e., there exist where Ŵ , V̂ are NN weights when driven by x̂, and are updated
two positive constants ρ, ρ1 , dependent on b and ϵ , such that based on the modified Back Propagation (BP) algorithm. The
∥∆P ∥ ≤ ρ, ∥∆K ∥ ≤ ρ1 . Moreover, if e(t) = O(ϵ ) for t ∈ [t2 , t1 ], observer (20) requires the knowledge of C . Accordingly, we define
then we will recover P = P̄ + O(ϵ ), K = K̄ + O(ϵ ). the output error as q̃ = q − C x̂. The objective function is to
minimize J = 12 (q̃T q̃). Following Abdollahi et al. (2006), the
Proof. Please see Mukherjee et al. (2020). update law follows from gradient descent as:
˙
If e(t) can be made sufficiently small by proper tuning of the Ŵ = − η1 (q̃T C A−
c ) (σ (V̂ x̄)) −ρ1 ∥q̃∥Ŵ ,
1 T ˆ T (21)
observer gain then we would recover the design characteristics of
  
Algorithm 1. To this end, we present the following stability result. η1 ( ∂ J )
∂ Ŵ
˙
V̂ = − η2 (q̃ C Ac Ŵ (I − Λ(V̂ x̄)))
T −1 ˆ T −ρ2 ∥q̃∥V̂ ,
ˆ T sgn(x̄)
Theorem 4. Assume that the control policy u = −Kk ŷ is asymp-   
totically stabilizing for the kth iteration. Then, there exist sufficiently η2 ( ∂ J )
small b∗ , and 0 < ϵ ∗ ≪ 1 such that for b ≤ b∗ , 0 < ϵ ≤ ϵ ∗ , with ∂ V̂

Q ≻ 0, R ≻ 0, u = −Kk+1 ŷ will asymptotically stabilize (3) at the
(k + 1)th iteration.
Proof. Please see Appendix A.
As shown in Appendix A, the estimation error enters the
closed-loop system as an exogenous disturbance. Since Kk+1 is
stabilizing, the states converge to a neighborhood of the origin for
sufficiently small b∗ and ϵ ∗ . Note that the designer does not need
the explicit knowledge of ϵ ∗ . Assuming the existence of a small
enough ϵ is sufficient. From the physical laws of the dynamics, the
states in Eq. (3b) are inherently sufficiently faster than the states
in Eq. (3a) ensuring stability. The design takes advantage of this
structure to construct a sample efficient reduced-dimensional
learning control methodology.
Remark 2. The convergence of the observer and the RL itera-
tions are handled sequentially. The observer gathers a sufficient
amount of data samples to meet the rank condition rank(Θ̂k ) =
r(r + 1)/2 + rm, after which the control gain is computed it-
eratively. Θ̂k has same structure as Θk but with y(t) replaced
by ŷ(t). The designer may start gathering data samples after a
few initial time-steps over which the observer has converged
close to its steady-state. The observer is designed to achieve
fast convergence, as discussed next. The state estimation error
that may be present in the observer output has been taken into
consideration in the sub-optimality and the stability analysis, as
discussed in Proposition 1 and Theorem 4.
4
where η1 , η2 > 0 are learning rates and ρ1 , ρ2 are small positive
numbers, Ac = A − GC , σ (·) is the activation function, and x̄ˆ =
[x̂, u]. Considering k neurons we have Λ(V̂ x̄) ˆ = diag(σ 2 (V̂i x̄))
i
ˆ ,i =
1, 2, . . . , k, where σi (V̂i x̄)
ˆ is the ith element of σ (V̂ x̄),
ˆ and sgn(·)
is the sign function. The update law (21) depends on the knowl-
edge of C . This observer guarantees the following boundedness
property.
Theorem 5 (Abdollahi et al., 2006, Theorem 1). With the update law
described as (21), the state estimation error x̃ = x − x̂ and weight
estimation errors W̃ = W − Ŵ , Ṽ = V − V̂ are uniformly ultimately
bounded (UUB).
The size of the estimation error bound can be made arbitrarily
small by properly selecting the parameters and learning rates
as shown in Abdollahi et al. (2006). For example, with higher
learning rates the convergence rate can be increased but these
parameters need to be properly tuned to avoid overshoot, and
selecting  to have fast eigenvalues will also keep the state
estimation error small.
5. Applying to clustered multi-agent networks
We next describe how SP-based RL designs can be applied
for the control of clustered multi-agent consensus networks,
e.g., power systems, robotic swarms, and biological networks. The
LTI model of these networks can be brought into the standard SP
form (3) by exploiting the time-scale separation in its dynamics
arising from the clustering of nodes.
S. Mukherjee, H. Bai and A. Chakrabortty
Fig. 1. Centralized and block-decentralized control architectures.
5.1. SP representation of clustered networks
Consider a network of n agents, where the dynamics of the ith
agent is given by
∑ dts
ẋi = Fxi + aij (xj − xi ) + bi ui ,
j∈Ni
where xi ∈ Rs is the state, ui ∈ Rp is the input, and Ni denotes
the set of agents that are connected to agent i, for i = 1, . . . n.
The graph between agents is assumed to be connected and time-
invariant. The constants aij = aji > 0 denote the coupling
strengths of the interaction between agents i and j, and vice versa.
The matrix F ∈ Rs×s models the self-feedback of each node. The
overall network model is written as
ẋ = Ax + Bu, x(0) = x0 ,
ns
where, x ∈ R is the vector of all agent states, u ∈ R is the ns
control input, B = diag(b1 , . . . , bn ), A = In ⊗ F + L ⊗ Is , L ∈ Rn×n
being the weighted network Laplacian matrix satisfying L1n = 0.
Assumption 3. F is marginally stable.
Let the agents be divided into r non-empty, non-overlapping,
distinct groups I1 , . . . , Ir such that agents inside each group
are strongly connected while the groups themselves are weakly
connected, i.e., aij ≫ apq for any two agents i and j inside a group
and any other two agents p and q in two different groups. This
type of clustering has been shown to induce a two-time scale
behavior in the network dynamics of (22) (Chow & Kokotovic,
1985). Fig. 1a shows an example of such a clustered dynamic
network. The clustered nature of the network helps decompose L
as L = LI +ϵ LE , where LI (a block-diagonal matrix) and LE (a sparse
matrix) represent the internal and the external connections, and
ϵ is the singular perturbation parameter arising from the worst-
case ratio of the coupling weights inside a cluster to that between
the clusters. The slow and fast variables are defined as
[ ] [ ] [ ]
y T y
= x, x = (U G† ) , (24)
z G z
where, T = T1 ⊗ Is , G = G1 ⊗ Is . Here, T1 ∈ Rr ×n = Na−1 U T ,
Na = diag(n1 , n2 , . . . , nr ), where ni denotes the number of agents
in group i, U = diag(U1 , U2 , . . . , Ur ), Uα = 1nα . The matrix G1 , fol-
lowing from Chow and Kokotovic (1985), is not required for our
design as we are only interested in constructing the slow states.
We can see that T is simply an averaging operation. Applying this
Automatica 126 (2021) 109451
transformation to (23), and redefining the time-scale as ts = ϵ t,
the following SP form is obtained:
dy
= A11 y + A12 z + B1 u, (25a)
(22) dz
ϵ = A21 y + A22 z + B2 u, (25b)
dts
A11 = T (LE ⊗ Is )U + (Ir ⊗ F )/ϵ, A12 = T (LE ⊗ Is )G† ,
A21 = G(LE ⊗ Is )U , A22 = G(LI ⊗ Is )G† + (In−r ⊗ F )+
ϵ G(LE ⊗ Is )G† , B1 = TB/ϵ, B2 = GB.
The detailed derivation is shown in Mukherjee et al. (2020). All six
matrices are assumed to be unknown. Following Assumption 2,
we assume that A22 is Hurwitz.
(23)
5.2. Projection of control to agents
One important distinction between controlling the multi-agent
system (25) and a generic SP system (3) is that the control input
u for the former has a physical meaning in terms of each agent.
Therefore, u(t), although designed in lower dimension, must be
actuated in its actual dimension. One way to design u(t) is to
use u = M ũ where ũ ∈ R(rp)×(rs) is the actual control signal
learned using ADP, and the matrix M is a projection matrix of
the form M = blkdiag(M 1 , . . . , M r ), M i = M̄ i ⊗ Is , M̄ i = 1|Ii | ,
which projects the reduced-dimensional controller to the full-
dimensional plant. M is constructed by the designer with the
assumption that the designer knows the cluster identity of each
agent. We assume (A, BM) to be stabilizable.
6. Block-decentralized multi-agent RL
The controllers learned in Sections 3 and 4 need to be com-
puted in a centralized way. In this section we show that for the
clustered consensus model (25) the clustered nature of the sys-
tem can also aid in learning a cluster-wise decentralized RL con-
troller. Fig. 1 describes the centralized and block-decentralized
architectures.
6.1. Cluster-wise representation
Let the states of the agents in cluster α be denoted as
(xα1 , xα2 , . . . , xαnα ) ∈ Rnα s . Following Chow and Kokotovic (1985),
the transformation matrix T in (24) is an averaging operation on
5
S. Mukherjee, H. Bai and A. Chakrabortty Automatica 126 (2021) 109451

the states of agents inside a cluster, which implies that the slow Algorithm 2 Cluster-wise Decentralized ADP
variable for the cluster α is For area α = 1, 2, . . . , r
1 α Step 1: Construct matrices δyα yα , Iyα yα , Iyα uα0 having similar structures as
yα = (x + xα2 + · · · + xαnα ), α = 1, . . . , r , (26) δyy , Iyy , Iyu0 but with y(t) replaced by yα (t).
nα 1 Step 2: Starting with a stabilizing K0α , Solve for Kkα+1 iteratively (k = 0, 1, . . . )
y = [y1 ; y2 ; . . . ; yr ]. (27) once matrices δyα yα , Iyα yα , Iyα uα0 are constructed and iterative equation can be
written for each small learning steps as,
For the cluster-wise decentralized design, the starting point is to
δyα yα −2Iyα yα (Is ⊗ KkαT Rα ) − 2Iyα uα0 (Is ⊗ Rα ) ×
[ ]
consider the scenario if all clusters were decoupled from each
other. We denote the states in cluster α in that scenario as
  
Θkα
xαd1 , xαd2 , . . . , xαdnα ∈ Rnα s , and the concatenated state vector [
v ec(Pkα )
]
considering all the clusters are denoted as xd . For this decoupled α = −Iyα yα v ec(Qkα ) . (34)
v ec(Kk+1 )
scenario, yαd and yd are similarly defined following (26) and (27).
  
Φαk
Then we will have,
The stopping criterion for this update is Pkα − Pkα−1  < γ1 , where γ1 is a chosen
 

ẋd = (In ⊗ F + LI ⊗ Is )xd + Bu, (28) small positive threshold.

Step 3: Next ũα = −K α yα is applied and uα0 source is removed.
ẏd = T ẋd = (T1 ⊗ Is )(In ⊗ F + L ⊗ Is )xd + B̃1 u,
I
End For

where B̃1 = TB. As xd = Uyd + G† zd , (28) is reduced to

ẏd = (Ir ⊗ F )yd + B̃1 u.
The controller can be represented cluster-wise as u = In this section we analyze the sub-optimality and stability
[u1 ; u2 ; . . . , ur ]. Using the projected controller discussed in Sec-
tion 5.2, we can design uα (t) as
uα = M α ũα , M α = M̄ α ⊗ Is , M̄ i = 1|Ii | ,
α
where ũ is the controller learned in cluster α , α = 1, . . . , r.
Taking a hint from the cluster-wise decentralized structure of
yd -dynamics in (29), we next state our design problem as follows.
P3. Consider the multi-agent consensus model (23) where A and
B are unknown. Learn a control gain K α for every area α , α =
1, . . . , r, using yα (t) and ũα (t) such that uα = M α ũα = −M α K α yα
stabilizes the closed-loop system and minimizes the following
individual cluster-wise objectives
∫ ∞
α α α αT α α αT α α
J (y (0); ũ ) = (y Q y + ũ R ũ )dt ,
0 Proof. This proof directly follows from the analysis performed
for α = 1, . . . , r. We assume that (A, BM) is stabilizable.
6.2. RL algorithm
We exploit a different O(ϵ ) separation existing between the
trajectories of the actual average variable of an area and the same
variable when the areas are decoupled. We start by providing a
lemma proving how the actual average variable yα is related to
the decoupled average variable yαd for an area α .
Lemma 2. The cluster-wise average variable yα (t) and the decoupled
average variable yαd (t) are related as,
(29) 6.2.1. Analysis and stability for the decentralized design
aspects of the area-wise decentralized controller learned from
Algorithm 2. The learned controller K α ∈ R for all the areas will
be perturbed from the controller computed using yαd , i.e.,
(30)
P α = P̄ α + ∆P α , K α = K̄ α + ∆K α , (35)
α α
where P̄ , K̄ are the optimal solutions if the clusters were decou-
pled and yαd (t) were available for design, and ∆P α , ∆K α are ma-
trix perturbations. The following theorem shows that the matrix
perturbations are O(ϵ ) small.
Theorem 6. Assuming ∥yαd (t)∥ and ∥uα0 (t)∥ are bounded, the area-
wise decentralized solutions satisfy for α = 1, . . . , r
P α = P̄ α + O(ϵ ), K α = K̄ α + O(ϵ ), J α = J̄ α + O(ϵ ). (36)
(31)
for Theorem 2. Here the time-scale separation exists between the
decoupled average variable yαd and the actual average variable yα .
Using Lemma 2, these variables are O(ϵ ) apart, which leads to (36)
following the analysis of Theorem 2, and Corollary 1. □
Next we analyze the closed-loop stability conditions for the
block-decentralized design.
Theorem 7. Assume that the control policy uα = −M α Kkα yα for
area α at the kth iteration is asymptotically stable. Then the control
policy at the (k + 1)th iteration given by uα = −M α Kkα+1 yα is
asymptotically stable with Rα ≻ 0 and Q α ≻ 0, if ϵ is sufficiently
small. □

yα (t) = yαd (t) + O(ϵ ), ∀t ∈ [0, t1 ]. (32) Proof. The proof is given in Appendix B.

Proof. The proof is presented in Mukherjee et al. (2020).

7. Numerical simulations
We first consider decoupled clusters with the according T for
averaging. The decoupled slow dynamics is given in (28). The 7.1. Centralized state feedback design
controller for area α uses the yαd (t) feedback and implements
ũα = −K̄ α yαd (t) so that the decoupled dynamics are stabilized A singularly perturbed system in the form of (3) is considered
and the following objective is minimized for area α with the ARE with two fast and two slow states. We choose ϵ = 0.01, Q =
solution P̄ α ≻ 0 and the optimal control gain K̄ α : 10I2 , R = I, the initial conditions as [1, 2, 1, 0], and the learning
∫ ∞ time-step as 0.01 s. The model matrices are taken from Chow and
J̄ α (yαd (0); ũα (0)) = (yαd T Q α yαd + ũα T Rα ũα )dt . (33) Kokotovic (1976) as
0
0.4 −0.524
[ ] [ ] [ ]
0 0 0 0
As the decoupled system is fictitious, based on Lemma 2, it is A11 = , A12 = , A21 =
0 0 0.345 0 0 0
plausible to replace yαd (t) with yα (t) in the learning algorithm and
−0.465 0.262
[ ] [ ]
then follow the same procedure as the Kleinman’s algorithm. The 1
resulting algorithm is given in Algorithm 2.
A22 = , B1 = B2 = .
0 −1 1
6
S. Mukherjee, H. Bai and A. Chakrabortty
Fig. 2. Convergence of P and K for the standard SP system.
Fig. 3. Comparison of slow state 1 with ϵ = 0.01, 0.001 and reduced slow
subsystem for standard SP system.
The system is persistently excited by exploration noise follow-
ing Jiang and Jiang (2017). The control gain is learned as K =
[3.80 1.38], producing a closed-loop objective value J = 7.72
units. The convergence plots for P and K are shown in Fig. 2. For
the ideal slow system (i.e., when ϵ = 0), the following controller
is learned: K̄ = [3.1623 1.9962], J̄ = 7.2950 units. Fig. 3
compares closed-loop responses learned by ADP for the ideal
reduced slow system (ϵ = 0) versus the system with ϵ ̸ = 0. The
top panel of Fig. 3 shows this comparison for ϵ = 0.01, while the
bottom panel shows this for ϵ = 0.001. It can be seen that the
responses of the ideal and non-ideal reduced-order systems get
closer to each other over time as ϵ decreases.
We next consider a 5-cluster, 25-agent network, each agent
with scalar state with F = 0. Therefore the network has 4
slow eigenvalues, one zero eigenvalue and the rest are the fast
eigenvalues. The slow eigenvalues are −0.128, −0.195, −0.196,
and −0.2638. The control architecture is shown in Fig. 1(a). Each
7
Automatica 126 (2021) 109451
Fig. 4. Learned controller with full-state feedback.
Fig. 5. Improved closed-loop response for the clustered network (Top panel -
Q = 10I5 , Bottom panel - Q = 1000I5 ).
Table 1
Reduction in learning and CPU run times for the slow state feedback-based
design with 25 agents.
Min. learning time CPU run times
(T = 0.01 s)
Full-state feedback 18.75 s 72.19 s
Reduced-dim state feedback 0.75 s 1.34 s
cluster is assumed to have a local coordinator that averages the
states from inside the cluster, and transmits the average state to a
central controller, which learns the reduced-dimensional control
input ũ(t) ∈ R5 and subsequently back-projects it to individual
agents.
Fig. 4 shows the learning of the full-dimensional optimal LQR
controller. It takes at least 18.75 s to learn K ∈ R25×25 . The ex-
ploration signal here is a sum of sinusoidal signals with different
frequencies. With r = 5, the reduced-order controller, on the
other hand, requires only r 2 + 2r 2 = 75 samples for learning. It
dominantly affects the slow poles, and with Q = 10I5 , the closed-
loop slow poles are placed at −3.14, −3.18, −3.17, −3.15, and
−3.16. Dynamic performance is improved with increase in the
weights of Q as shown in Fig. 5. Table 1 gives a comparison
between the full and the reduced-order designs in terms of min-
imum learning time based on the sample requirement and CPU
run times.
S. Mukherjee, H. Bai and A. Chakrabortty Automatica 126 (2021) 109451

Fig. 8. Dynamic performance with cluster-wise decentralized design (Top panel

Fig. 6. Decentralized controllers for ideal decoupled clusters.
— Q = 10I5 for all areas, Bottom panel — varying Q across different areas).

Fig. 7. Convergence of K and P for the cluster-wise decentralized design.

7.2. Cluster-wise decentralized state feedback design

Considering the same multi-agent example, we first perform

the ADP-based learning of the controller when the clusters are
fully decoupled (i.e., the ideal decentralized scenario). Each area Fig. 9. Convergence of K and P for the standard SP system (OFRL).
is equipped with an aggregator. Note that the average of all the
cluster states represents the decoupled slow state yαd for cluster
α . The state evolution of one representative area is shown in
Fig. 6. We consider similar coupling strengths between the agents
inside all the clusters with Q = 10, R = 1 but with different
initial conditions. The computed scalar control gain for each area
is K = 3.1623, and the corresponding objective values are J̄ 1 =
1.317, J̄ 2 = 0.745, J̄ 3 = 1.765, J̄ 4 = 0.8451 and J̄ 5 = 0.5244.
Thereafter, the decentralized ADP computation is performed
on the actual system following Algorithm 2. The average states
from each cluster is used as the feedback signal for the ADP
computation block as shown in Fig. 1(b). Fig. 7 shows the fast
convergence of the ADP iterations. With Q = 10, R = 1 for all the
areas, the cluster-wise decentralized control gains are computed
as K 1 = 3.139, K 2 = 3.195, K 3 = 3.130, K 4 = 3.187, K 5 =
3.173, with the objective values as J 1 = 1.308, J 2 = 0.754, J 3 =
1.7478, J 4 = 0.8524 and J 5 = 0.5261. In Fig. 8, we can see
that with the increasing value of Q α , α = 1, . . . , 5, the dynamic
performance of the agent states increases. The dynamic perfor-
mance of different cluster states can be controlled independently
using different Q for the different areas. The learning time is also
decreased because of the reduced number of feedback variables.
The exploration is performed for only 0.2 s.

7.3. Output feedback RL (OFRL) design Fig. 10. Slow state trajectories for the standard SP system (OFRL).

We first consider the singularly perturbed system as in Sec-

tion 7.1 with ϵ = 0.01, initial condition [1, 2, 1, 0]. We consider
C = [1, 1, 0, 0; 0, 0, 1, 1]. The learning time step is 0.01 s. Data is
gathered for 0.7 s with the system being persistently excited with
exploration noise. Fig. 9 shows the convergence of P and K during
8
the ADP-based computations using the estimated states. Fig. 10
and Fig. 11 show the actual versus estimated state trajectories
using the NN observer. For the design of the NN observer, the
S. Mukherjee, H. Bai and A. Chakrabortty
Fig. 11. Fast state trajectories for the standard SP system (OFRL).
Fig. 12. Comparison with state feedback for the ϵ = 0 system (OFRL).
Hurwitz matrix  is considered to be of SP structure but different
than the original state matrix. We can see from Figs. 11–12 that
the estimation error is small, and the ADP controller using these
estimates maintains closed-loop stability. Also, Fig. 12 compares
the output feedback control responses with the ideal (ϵ = 0) state
feedback responses.
We next consider the 5-cluster, 25-agent consensus network.
For the estimator design, the Hurwitz matrix  is taken to be of
similar structure as A but the coupling between the agents in a
same cluster is 20% off from the original, while the inter-cluster
strengths are 50% off from the original. For the full-order system,
Fig. 13 shows few examples of the state estimation, where the
learning takes approximately 20 s. In the reduced-order design,
using the NN observer estimates the aggregator generates the
average states for each cluster. These average states and inputs
are used for the reduced-order ADP iterations. Fig. 14 shows that
the reduced-order design using the NN observer requires ap-
proximately 1 second of exploration time. The savings in sample
9
Automatica 126 (2021) 109451
Fig. 13. Learning with full state estimates for the clustered network.
Fig. 14. Learning with slow state estimates for the clustered network.
complexity (i.e., minimum learning time) here will be the same as
that for the state feedback case once the observer converges. The
CPU run time for the reduced-dimensional design is only 13.82 s
compared to that for the full-scale design, which is 298 s.
8. Conclusion
The paper presented RL based optimal control designs incor-
porating ideas from model reduction following from time-scale
separation properties in LTI systems. Both state feedback and
output feedback RL designs are reported. The designs are ex-
tended to clustered multi-agent networks for which an additional
cluster-wise block-decentralized RL control is also discussed. Sub-
optimality and stability analyses for each design are performed
using SP approximation theorems. For the state feedback de-
signs only the SP approximation error affects the sub-optimality,
whereas for the output feedback designs the state estimation
error adds to it. Results are validated using multiple simulation
case studies.
S. Mukherjee, H. Bai and A. Chakrabortty
Appendix A. Proof of Theorem 4
We know that ∥ŷ − y(t)∥ ≤ b, therefore by use of slack
variables, we can write ŷ = y(t) + b1r − ∆b (t). Also, we have
∥Kk+1 − K̄k+1 ∥ ≤ ρ1 , implying Kk+1 = K̄k+1 + ρ1 I − ∆ρ . Let us
denote b2 = b1r −∆b (t), ρ2 (b, ϵ ) = ρ1 I −∆ρ . The feedback control
is given by, u = −Kk+1 ŷ = −Kk+1 (y + b2 ) which will make (3) as
ẏ = A11 y + A12 z + B1 (−Kk+1 (y + b2 )), (A.1)
ϵ ż = A21 y + A22 z + B2 (−Kk+1 (y + b2 )). (A.2)
We next re-derive the slow subsystem by substituting ϵ = 0. The
1 −1
slow manifold is given as zs = −A−
22 A21 − B2 Kk+1 ys + A22 B2 Kk+1 b2 .
Therefore, the slow-subsystem dynamics using Kk+1 = K̄k+1 + ρ2
follows as
ẏs = (As − Bs K̄k+1 + ρ3 )ys + ρ4 (t). (A.3)
where As = A11 − A12 A22 A21 , Bs = B1 − A12 A22 B2 , ρ3 = −Bs ρ2 and
−1 −1
ρ4 (t) = −Bs (K̄k+1 b2 (t) + ρ2 (t)b2 (t)). Here, ρ4 (t) acts as a distur-
bance to the dynamics: ẏs = (As − Bs K̄k+1 + ρ3 )ys . Therefore, we
investigate stability by analyzing the disturbance-free dynamics.
One can consider the dynamics ẏs = (As − Bs K̄k+1 + ρ3 )ys as a
perturbed version of the nominal dynamics, ẏs = (As − Bs K̄k+1 )ys .
Considering a Lyapunov function Vk (t) = yTs P̄k ys , and computing
its time-derive along ẏs = (As − Bs K̄k+1 )ys , we get
V̇k (t) = yTs [P̄k (As − Bs K̄k+1 ) + (As − Bs K̄k+1 )T P̄k ]ys ,
which, using the proof of Theorem 2, can be shown to reduce to
V̇k (t) = −yTs [(K̄k − K̄k+1 )T R(K̄k − K̄k+1 )]yTs
− yTs [Q + K̄kT+1 RK̄k+1 ]ys . (A.4)
With Q ≻ 0, closed-loop will be asymptotically stable. The
dynamics ẏs = (As − Bs K̄k+1 +ρ3 )ys is basically ẏs = (As − Bs K̄k+1 )ys
perturbed by ρ3 ys vanishing at ys = 0. If the estimation error is
small with sufficiently small ϵ then we will have a sufficiently
small upper bound ∥ρ3 ∥ ≤ ρ̄3 , and the vanishing perturbation
g(t , ys ) = ρ3 ys will satisfy ∥g(t , ys )∥ ≤ ρ̄3 ∥ys ∥. With these
considerations, we apply Khalil (2002, Lemma 9.1) and conclude
that the ys = 0 is exponentially stable for a sufficiently small
ϵ and state estimation error. Disturbance ρ4 (t) depends on the
state estimation error bound b and the controller gain Kk+1 . With
arbitrarily small estimation error, the norm of the disturbance can
be bounded by sufficiently small upper-bound ∥ρ4 (t)∥ ≤ ρ̄4 .
Appendix B. Proof of Theorem 7
We first show that the learned decentralized control gain
Kkα+1 can stabilize the decoupled yd dynamics when ϵ is small
with a sufficiently large Q . Let the area-wise control be uα =
−M α Kkα+1 yα . Therefore, u = −MKk+1 y, where Kk+1 = diag(Kk1+1 ,
. . . , Kkr+1 ). From Theorem 6, we have Kkα+1 = K̄kα+1 +O(ϵ ), implying
Kk+1 = K̄k+1 +O(ϵ ). Using the learned gains Kk+1 for the decoupled
dynamics with F1 = In ⊗ F we get ẏd = F1 yd − B̃1 (−M K̄k+1 yd ) −
O(ϵ )yd . Next, consider the Lyapunov function Vk (t) = yTd P̄k yd with
P̄k ≻ 0, and its time derivative along ẏd as,
V̇k (t) = yTd [P̄k (F1 − B̃1 M K̄k+1 − O(ϵ ))
+ (F1 − B̃1 M K̄k+1 − O(ϵ ))T P̄k ]yd . (B.1)
Using the ARE, ATk P̄k + P̄k Ak = −(K̄kT RK̄k + Q ) with Ak =
F1 − B̃1 M K̄k , and K̄k+1 = R−1 M T B̃T1 P̄k , it can be shown that V̇k (t)
10
Automatica 126 (2021) 109451
becomes
V̇k (t) = −yTd (K̄k − K̄k+1 )T R(K̄k − K̄k+1 ) + Q
[
+K̄kT+1 RK̄k+1 − O(ϵ ) yd .
]
(B.2)
We conclude that with a sufficiently small ϵ , if Q =
diag(Q 1 , . . . , Q r ) has a sufficiently large λmin (Q ) > 0 then V̇k (t)
will be negative definite, stabilizing the decoupled dynamics.
Next, consider the reduced slow sub-system dynamics of the
actual system. Using the learned feedback in (25) we get
dy
= ((Ir ⊗ F )/ϵ )y + H11 y + H12 z + B̃1 /ϵ (−MKy), (B.3)
dts
dz
ϵ = ϵ H21 y + (H̃2 + ϵ H22 )z + B̃2 (−MKy), (B.4)
dts
where, H̃2 = H2 + (In−r ⊗ F ). By substituting ϵ = 0 and using
the slow manifold variable zs = −H̃2−1 B˜2 (−MKys ), we obtain the
reduced sub-system as
dys (Ir ⊗ F ) B̃1
= ys + H11 ys + ( − H12 H̃2−1 B̃2 )(−MKys ).
dts ϵ ϵ
Reverting back to the original time-scale ts = ϵ t, we get
dys
= (Ir ⊗ F )ys + ϵ H11 ys + (B̃1 − ϵ H12 H̃2−1 B̃2 )(−MKys ),
dt
= (Ir ⊗ F )ys − B̃1 MKys + ϵ (H11 + H12 H̃2−1 B̃2 ) ys . (B.5)
  
H̃
The dynamics (B.5) can be viewed as the decoupled dynamics
ẏd = F1 yd − B̃1 MKyd perturbed by an O(ϵ ) term vanishing at
ys = 0. The vanishing perturbation term given by g(t , ys ) =
ϵ H̃ys satisfies ∥g(t , ys )∥ ≤ ϵ∥H̃ ∥∥ys ∥. With these considerations,
we apply Khalil (2002, Lemma 9.1) and conclude that ys = 0
is exponentially stable for a sufficiently small ϵ . As the slow
reduced sub-system model is the perturbed version of the de-
coupled model with the above-mentioned bound, the learned
decentralized controller will exponentially stabilize the slow sub-
system dynamics, which in turn stabilizes the entire system with
the assumption that the fast sub-system is stable. □
References
Abdollahi, F., Talebi, H. A., & Patel, R. V. (2006). A stable neural network-based
observer with application to flexible-joint manipulators. IEEE Transactions on
Neural Networks, 17(1), 118–129.
Chow, J., & Kokotovic, P. (1976). A decomposition of near-optimum regulators for
systems with slow and fast modes. IEEE Transactions on Automatic Control,
21(5), 701–705.
Chow, J., & Kokotovic, P. (1985). Time scale modeling of sparse dynamic
networks. IEEE Transactions on Automatic Control, 30(8), 714–722.
Jiang, Y., & Jiang, Z.-P. (2012). Computational adaptive optimal control
for continuous-time linear systems with completely unknown dynamics.
Automatica, 48, 2699–2704.
Jiang, Y., & Jiang, Z.-P. (2017). Robust adaptive dynamic programming. Wiley-IEEE
press.
Khalil, H. (2002). Nonlinear systems. Prentice-Hall, New York.
Khalil, H., & Kokotovic, P. (1978). Control strategies for decision makers using
different models of the same system. IEEE Transactions on Automatic Control,
23(2), 289–298.
Kleinman, D. (1968). On an iterative technique for Riccati equation computations.
IEEE Transactions on Automatic Control, 13(1), 114–115.
Kokotovic, P., O’malley, R., & Sannuti, P. (1976). Singular perturbations and order
reduction in control theory: An overview. Automatica, 12, 123–132.
Lewis, F., & Vrabie, D. (2009). Reinforcement learning and adaptive dynamic
programming for feedback control. IEEE Circuits and Systems Magazine, 9(3),
32–50.
Liu, D., & Wei, Q. (2014). Policy iteration adaptive dynamic programming
algorithm for discrete-time nonlinear systems. IEEE Transactions on Neural
Networks and Learning Systems, 25(3), 621–634.
Mukherjee, S., Bai, H., & Chakrabortty, A. (2018). On model-free reinforcement
learning of reduced-order optimal control for singularly perturbed systems.
In IEEE conference on decision and conrol 2018. Miami, FL, USA.
S. Mukherjee, H. Bai and A. Chakrabortty
Mukherjee, S., Bai, H., & Chakrabortty, A. (2019). Block-decentralized model-
free reinforcement learning control of two time-scale networks. In American
control conference 2019. Philadelphia, PA, USA.
Mukherjee, S., Bai, H., & Chakrabortty, A. (2020). Reduced-dimensional rein-
forcement learning control using singular perturbation approximations. arXiv
preprint arXiv:2004.14501.
Sutton, R., & Barto, A. (1998). Reinforcement learning - An introduction. Cambridge:
MIT Press.
Vamvoudakis, K. (2017). Q-learning for continuous-time linear systems: A
model-free infinite horizon optimal control approach. Systems & Control
Letters, 100, 14–20.
Vrabie, D., Pastravanu, O., Abu-Khalaf, M., & Lewis, F. (2009). Adaptive opti-
mal control for continuous-time linear systems based on policy iteration.
Automatica, 45, 477–484.
Wu, H., & Luo, B. (2012). Neural network based online simultaneous policy
update algorithm for solving the HJI equation in nonlinear H∞ control. IEEE
Transactions on Neural Networks and Learning Systems, 23(12), 1884–1895.
Sayak Mukherjee received the B.E. degree in Electri-
cal Engineering from Jadavpur University, India with
the medal for second highest percentage in 2015
and the Ph.D. in Electrical Engineering from NC State
University, USA in 2020. He recently joined Pacific
Northwest National Laboratory as a Post-doctorate Re-
search Associate. His research interests include learning
based optimal control techniques, scalable reinforce-
ment learning for dynamic systems with applications
related to power and energy systems.
11
Automatica 126 (2021) 109451
He Bai received his B.Eng. degree from the Depart-
ment of Automation at the University of Science and
Technology of China, Hefei, China, in 2005, and the
M.S. and Ph.D. degrees in Electrical Engineering from
Rensselaer Polytechnic Institute (RPI) in 2007 and 2009,
respectively. From 2009 to 2010, he was a Post-doctoral
Researcher at Northwestern University, Evanston, IL.
From 2010 to 2015, he was a Senior Research and
Development Scientist at UtopiaCompression Corpora-
tion. In 2015, he joined the Mechanical and Aerospace
Engineering Department at Oklahoma State University
as an assistant professor. He has published over 80 peer-reviewed journal and
conference papers related to control and robotics and a research monograph
‘‘Cooperative control design: a systematic passivity-based approach’’ in Springer.
He holds one patent on monocular passive ranging. His research interests in-
clude reinforcement learning, distributed optimization and learning, multi-agent
systems, and autonomous systems.
Aranya Chakrabortty received the Ph.D. degree in
Electrical Engineering from Rensselaer Polytechnic In-
stitute, Troy, NY in 2008. From 2008 to 2009 he
was a postdoctoral research associate in University of
Washington, Seattle. From 2009 to 2010 he was an
Assistant Professor of Electrical Engineering at Texas
Tech University. Since 2010 Aranya has joined the
Electrical and Computer Engineering Department at
North Carolina State University, where he is currently
a Professor. His research interests are in all branches
of control theory with applications in electric power
systems. He received the NSF CAREER award in 2011.