A Reinforcement Learning Fuzzy Controller for
the Ball and Plate System
Nima Mohajerin, Mohammad B. Menhaj, Member, IEEE, Ali Doustmohammadi
Abstract—In this paper, a new fuzzy logic controller, namely
Reinforcement Learning Fuzzy Controller (RLFC), is proposed
and implemented. Based on fuzzy logic, this newly proposed
online-learning controller is capable of improving its behavior by
learning from experiences it gains through interaction with the
plant. RLFC is well established for hardware implementation
with or without a priori knowledge about the plant. To evidence
this claim, a hardware implementation of Ball and Plate system
was established, and RLFC was then developed and applied to it.
The obtained results are illustrated in this paper.
Index Terms—Fuzzy Logic Controller, Reinforcement
Learning, Ball and Plate system, Balancing Systems, Model-free
optimization
I. INTRODUCTION
B ALANCING systems are one of the most popular and
challenging test platforms for control engineers. Such
systems are the traditional cart-pole system (inverted
pendulum), the ball and beam (BnB) system, the multiple
inverted pendulums, the ball and plate system (BnP), etc.
These systems are the promising test-benches for investigating
the performance of both model-free and model-based
controllers. Considering those complicated ones (such as
multiple inverted pendulums or BnP) even if one bothers to
mathematically model them, the resulted model is likely to be
too complicated to be used in a model-based design. One
would highly prefer to use an implemented version of such a
system (if available and not risky) and observe its behavior
while the proposed controller is applied to it. This paper is
devoted to the efforts done for a project in which the main
goal is to control a ball over a flat rotary surface (the plate)
mimicking human’s behavior controlling the same plant, i.e.
the BnP system. The proposed controller neither should be
dependent on any physical characteristics of the BnP system
nor should it be supervised by an expert. It should learn an
optimal behavior from its experiences interacting with the BnP
system and improve its action-generation strategy; however,
some prior knowledge about the overall behavior of the BnP
Manuscript received February 5, 2010.
Nima Mohajerin is with the School of Science and Technology of Örebro
University, Örebro, Sweden. (e-mail: nima.mohajerinh091@student.oru.se).
Mohammad B. Menhaj, is with the Electrical Engineering Department of
Amirkabir University of Technology, (e-mail: tmenhaj@ieee.org).
Ali Doustmohammadi, is with the Electrical Engineering Department of
Amirkabir University of Technology, (e-mail: doustm@aut.ac.ir).
978-1-4244-8126-2/10/$26.00 ©2010 IEEE
Authorized licensed use limited to: Inha University. Downloaded on July 27,2022 at 10:13:55 UTC from IEEE Xplore. Restrictions apply.
system may also be available and be used in order to reduce
the needed time for reaching the goal.
Those few published papers on the BnP system are mainly
devoted to achieve the defined goals regarding the BnP itself
rather than how to achieve those goals [3, 4, 5, 8, 9]. They can
be categorized into two main groups: those which are based on
mathematical modeling (with or without hardware
implementation), those proposing model-free controllers.
Since the simplification in the mathematical modeling of the
BnP system yields two separate BnB systems, the first
category is goal-oriented and is of no interest for the current
project [4, 5]. On the other hand, the hardware apparatus used
in the second category is CE151 [6] (or in some rare cases
another apparatus [9]) which all use image feedback for ball
position sensing. However, among all, [5] is devoted to a
mechatronic design of the BnP system controlled by a classic
model-based which benefits a touch-sensor feedback and [3,
4] used the CE151 apparatus. Note that the image feedback is
a time bottleneck which will be discussed in Section III. In [3],
a fuzzy logic controller (FLC) is designed which learns online
from a conventional controller. Although the work in [4] is
done through mathematical modeling and is applied to CE151
apparatus, it is of more interest because it tackles the problem
of trajectory planning (to be stated in Section III). Reports in
[8] and [9] are focused on motion planning and control though
they are less interesting for us.
To achieve the desired goal, a different approach will be
demonstrated in this paper. This approach is based on fuzzy
logic controller which learns on the basis of reinforcement
learning. Additionally, a modified version of the BnP
apparatus is also implemented in this project as a test platform
for the proposed controller.
In this paper, the fundamental concepts of RL are embodied
into fuzzy logic controlling methodologies. This leads to a
new controller, namely Reinforcement Learning Fuzzy
Controller (RLFC) which is capable of learning from its own
experiences. Inherited from fuzzy controlling methodologies,
RLFC is a model-free controller and, of course, previous
knowledge about the system can be included in RLFC so as to
decrease the learning time. However, as it will be seen,
learning in RLFC is not a separate phase of its functioning
phase.
This paper is divided into six sections. After this
introduction, in section II RLFC is completely explained, both
conceptually and mathematically. In section III, the BnP
system is introduced and the hardware specification of the
implemented version of this system is also outlines. In section
IV some modifications on RLFC in order to be applicable to
the BnP system is fully discussed. Section V is dedicated to
x = [x 1 x 2 xn]
T
(1)
illustrate and analyze the results of RLFC performance on the
implemented BnP system. In this section, RLFC performance Assume that in the universe of discourse of input variable
is also compared with that of a human controlling the same xi , i.e. U i , a number of ni term sets are defined. The lth
plant. Finally, section VI concludes the paper. *
fuzzy rule is : (for now forget about the consequence part)
II. CONTROLLER DESIGN IF x1 is A1l1 AND x2 is A2l2 AND … AND xn is Anln
(2)
This section is dedicated to explain the idea and THEN y is B l
mathematics of the proposed controller, i.e. RLFC. First, the l
where: 1 ≤ li ≤ ni , and Ai i is the l i th defined fuzzy set on the
behavior of RLFC is outlined conceptually and then the
mathematics will be detailed. universe of discourse of xi , (1 ≤ i ≤ n ) . The output variable
A. Outline is y and there are M fuzzy sets defined on the universe of
According to Fig. 1, RLFC consists of two main blocks, a discourse of y ( M ∈ ) - V is the universe of discourse of
controller (FLC) and a critic. The task of the controller is to
generate and apply actions in each given situation as well as y. Note that (2) is a fuzzy relation defined on U × V [11]
improving its state-action mapping while the critic has to whereU = U 1 ×U 2 × ×U n . Note also that in real world, all
evaluate the current state of the plant. Neither controller nor of the above variables express physical quantities. Therefore
critic necessarily knows anything about how the plant all the corresponding universes of discourse are bounded and
responds to actions, how the actions will influence its states can be covered by a limited number of fuzzy sets.
and what the best action is in any given situation. There is also For hardware implementation, what matters first is the
no need for the critic to know how the actions in the controller processing speed of the controller. In other words, we have to
are generated. The most important responsibility of the critic establish a fuzzy controller architecture that leads to an
is to generate an evaluating signal (reinforcement signal), optimum performance versus its complexity. For this reason,
which best represents the current state of the plant. The we propose the following elements for the FLC structure:
reinforcement signal is then fed to the controller. singleton fuzziyfier, product inference engine and center
Once the controller receives a reinforcement signal, it
should realize whether its generated action was indeed a good
average defuzzyfier [11]. In this case given an input vector x∗
the output of the controller is given by:
action or a bad one, and in both cases how good or how bad it
∑ y ( ∏ A (x ) )
L −l n li ∗
was. These terms are embodied into a measure named l =1 i i
∗ ∗ i =1
reinforcement measure which apparently is a function of the y = f (x ) = (3)
∑ ( ∏ A (x ) )
L n li ∗
reinforcement signal. Then according to this measure, the l =1 i =1 i i
controller attempts to update its knowledge of generating
−l l
actions, i.e. improves its mapping from states to actions. The where y is the center of normal fuzzy set B . However, as
process of generating actions is separate from the process of mentioned earlier, other FLC structures may be considered.
updating the knowledge and thus they can be assumed as two Apparently, only those rules are participated in generating y
parallel tasks. This implies that while the learning procedure is that have a non-zero premise (IF-part), i.e. the fired rules. This
a discrete-time process, the controlling procedure can be fact is not dependent on the FLC structure.
continues-time. However, without loss of generality it is On the design stage, the controller does not know which
assumed that the actions are generated in a discrete-time states it will observe, or in other words, the designer hardly
manner and each action is generated after the reinforcement may know what rules are useful to be embodied in the fuzzy
signal - describing consequences of the previously generated rule base. Thus, all rules with premises made by all possible
action - has been reported and the parameters have been combinations of input variables, using AND operator, are
updated. The dashed line in Fig. 1 implies the controller included in the fuzzy rule base. The number of these rules is:
awareness of its generated actions. Although apparently the n
controller is aware of its generated actions, in case of L = ∏ ni (4)
hardware implementation, inaccuracies in mechanic structure i =1
and electronic devices and other unknown disturbances may
impose distortions on the generated actions.
B. The Controller
The aforementioned concept is as general as to be
applicable to any fuzzy controller scheme; however, we
assume that the fuzzy IF-THEN rules and controller structure
are of Mamdani type [9]. Imagine that the input to the fuzzy
inference engine (FIS) is an n element vector, each element is Fig. 1. A typical application of the RLFC.
a fuzzy number produced by the fuzzyfication block [11]: *
Please note that superscripts are not powers, unless it is mentioned to be
so.

Authorized licensed use limited to: Inha University. Downloaded on July 27,2022 at 10:13:55 UTC from IEEE Xplore. Restrictions apply.
 Obviously, L grows exponentially with respect to the
number of variables and defined term sets. Consequently, the
processing time drastically increases. To solve this, we assume
that for any given value to each variable, there are at most two
term sets with non-zero membership values. This condition –
which will be referred by the covering condition – is necessary
and if it is held then the number of fuzzy rules which
contribute to generate the actual output, i.e. the fired rules, is
equal or less than 2n, i.e. 2 is powered by n. Noticeably, to
reduce the needed time for discovering the fired rules we
implement a set of conditional statements rather than an
extensive search among all of the rules.
C. The Decision Making
As previously mentioned, the controller is discrete-time. So,
in each iteration, as the controller observes the plant state, it
distinguishes the fired rules. From now on, the L-rules FLC
shrinks to a 2n-rules FLC where 2n << L . The key-point in
generating output, i.e. decision making, is how the
consequences (THEN parts) of these fired rules are selected.
Noticing (2), the assigned term set to the consequence of the
l
lth rule is B . It was also mentioned that there are M term sets
defined on the universe of discourse of the output variable.
Assume that these term sets are referred by W i where
i = 1, 2,..., M . Having fired the lth rule in the kth iteration,
the probability of choosing W i for its consequence is:
Pkl ( j = i ) (5)
where j is a random variable with an unknown discrete
distribution over indices. The aim of reinforcement learning
algorithm, which is discussed in the next sub-section, is to
learn this probability for each rule such that the overall
performance of the controller is (nearly) optimized.
l
Our objective in this section is to sensibly define Pk to be
well suited for applying the reward/punishment procedure and
also for software programming. Fulfilling these objectives, for
each rule a bounded sub-space on ℜ is chosen. Note that ℜ
represents the set of real numbers. Factors for how this one
dimensional sub-space should be chosen are discussed in
section IV. Let the related sub-space to lth rule be:
Ωl = ⎡⎣a0l , aMl ) (6)
This distance is divided into M sub-distances as shown by
(8-a), each of which is assigned to an index i where
i = 1, 2, , M . We have:
M y (k − 1) , etc. Thus, Δr (k ) is not immediately applicable
Ω l = ∪ ωrl (7)
r =1
a) ωrl = ⎡⎣ arl −1 , arl ) , r = 1,..., M
b) a0 ≤ as ≤ as +1 ≤ aM , s = 1,..., M − 2
(8)
c) ∀ l ∈ {1, 2,..., L} & p, q ∈ {1, 2,..., M } p ≠ q :
ω lp ∩ ωql = ∅
We calculate the probability P l by (9):
Authorized licensed use limited to: Inha University. Downloaded on July 27,2022 at 10:13:55 UTC from IEEE Xplore. Restrictions apply.
ωil
Pl (j = i ) = (9)
Ωl
where ω rl represents the numeric length of sub-distance
ωrl and is calculated by (10).
ωrl = arl − arl −1 (10)
Till now the iteration counter, k, is omitted. However since
the reinforcement learning procedure is done by tuning the
above parameters, they are all a function of k. Thus, (9) turns
to:
ωil ( k )
Pkl ( j = i ) = l (11)
Ω (k )
Or:
ail (k ) − ail −1 (k )
Pkl ( j = i ) = (12)
aMl ( k ) − a0l ( k )
By observing (7), (8) and (11) it is obvious that
Pkl ( j = i ) is a probability function and satisfies the
necessary axioms.
D. Reinforcement Learning Algorithm
In this sub-section the proposed algorithm for tuning the
above defined parameters is depicted. This algorithm is based
on reinforcement learning methods and satisfies the six
axioms mentioned by Barto in [12]. Let r ( k ) be the
reinforcement signal generated by the critic module in the kth
iteration. Note that it represents the effect of the previously
generated action by the FLC, i.e. y ( k − 1) , on the plant and
before generating the kth action the parameters of the related
fired rules should be updated. In other words, this scalar can
represent the change in the current state of the plant.
To be more specific, as a general assumption, imagine that
the smaller reinforcement signal represents a better state of
the plant. Obviously, the change in the current state of the
plant is stated by (13). Therefore, an improvement in the
plant situation is indicated by Δr (k ) > 0 while Δr ( k ) < 0
indicates that the plant situation is worsened.
Δr (k ) = r ( k − 1) − r ( k ) (13)
However, since (13) is based solely on the output of the
critic, it does not contain information about which rules have
been fired, which term sets have been chosen for generating
for updating the parameters. The mentioned
reward/punishment updating procedure means that if the
generated action resulted in an improvement in the plant
state, the probabilities of choosing the same term set for the
premises of each corresponding fired rule should be
increased. However, if this action caused the plant state to be
worsened, these probabilities should be decreased.
Δr (k ) will be mathematically manipulated in order to
 shape up the reinforcement measure. This measure is the
amount by which the mentioned probabilities, (11), would be
affected.
At the first step, a simple modification is done on Δr (k ) .
This step may be ignored if Δr (k ) is already a suitable
representation of change in the system state A comprehensive
example will illustrate this case in section IV. This
manipulation is done by f (.) noticing that f : ℜ ⎯⎯
→ℜ :
Δr ′( k ) = f ( Δr (k ) )
To modify the probabilities defined by (11), the
corresponding sub-distances, (8) are changed.The amount of
change in the sub-distances relating to the lth rule is defined
by ϑil (k ) in (15).
ϑil ( k ) = g .ε (l , i ).α (l ).Δr ′(k ) (15)
Regarding (15), g is gain and is a scaling factor, ε (l , i )
represents the exploration/exploitation policy [1], α (l ) is the
firing rate of the lth rule and is obtained by replacing x in the
membership function obtained from the premise part of the lth
rule. Note that this factor expresses the contribution of this
rule in generating the output.
There are a variety of exploration/exploitation policies [1, 2,
, 12], however here we propose a simple one:
⎛ θ ⎞ that Δωi ( k ) > 0 . Using (5) we have:
ε (l , i ) = ⎜ 1 − (k ) ⎟
n il
⎝ e ⎠ ΔPkl ( j = i ) = Pkl ( j = i ) − Pkl −1 ( j = i )
where n il ( k ) counts how many times W i is chosen for the
lth rule and θ is a scaling factor. Considering a particular
rule, as W i is chosen more for this rule, ε (l , i ) grows
exponentially to one, letting ϑi ( k ) → g .α (l ).Δr ′( k ) .
l
The reinforcement measure introduced in subsection II.A is
given by (17).
⎧⎪ϑil (k ) , Δr (k ) ≥ 0
Δωil (k ) = ⎨ (17)
{ l l
}
⎪⎩max ϑi (k ), −ωi (k ) , Δr (k ) < 0
In (17), max operator is used in case of Δr ( k ) < 0 . This
is due to avoid large reinforcing those sub-distances that have
not been chosen. The reason is clearer if (18) is studied.
Equation (18) depicts the updating rule:
⎧⎪ aql (k − 1) ,q < i
aql ( k ) = ⎨
⎪⎩ max {aq −1 ( k ), aq ( k − 1) + Δωi (k )} , q ≥ i
l l l
where q = 0,1,..., M .
Regarding (18) there are several points to be noted:
1- i is the index of the chosen term set for the
consequence part of the lth fired rule , i.e. B l = W i .
2- q is a counter which starts from i and ends to M.
Apparently, there is no need to update those
parameters which are not modified and q may starts
from i. This will indeed reduce the processing time.
3- There are M parameters for each rule in the rule base.
Authorized licensed use limited to: Inha University. Downloaded on July 27,2022 at 10:13:55 UTC from IEEE Xplore. Restrictions apply.
But only the parameters of fired rules are modified.
Hence there are at most M .2 n modifications per
iteration.
4- By (18) it should be understood that only the length of
subspaces ωil and Ωl are modified. Although,
subspaces ωql for q = i + 1,..., M are shifted, but their
length remains unchanged.
5- Modification of Ωl always let other subspaces (and
(14) hence other indices) to be chosen. As the system learns
more, a dominant subspace will be found, but the
length of un-chosen subspaces are still not zero as long
as the effect of choosing them is not observed as a
worsening result. This feature is useful in case of
slightly changing plants.
Theorem1. By (18) if a term set receives
reward/punishment, then the probability of choosing that
term set is increased/decreased.
Proof.
a) Reward. Assume that W i has been chosen for lth rule
and the resulting action has an improving effect. Thus,
Pkl ( j = i ) should increase. Note that in this case
Δr (k ) > 0 and by (13),(14),(15),(16) and (17) it is obvious
l
(16)
ωil (k ) ωil (k − 1) (19)
= −
l
Ω (k ) l
Ω (k − 1)
In case of an improvement ΔPkl ( j = i ) must be a positive
scalar. Using (9), (10), (11) and (12) in (19) we obtain:
a l (k ) − ail −1 (k ) ail (k − 1) − ail −1 (k − 1)
ΔPkl ( j = i ) = i −
aml (k ) aml (k − 1)
According to (18) the above equation yields:
a l + Δω l (k ) − ail −1 ail − ail −1
ΔPkl ( j = i ) = i l
am + Δω l (k )
−
aml
In which we omit (k-1) on the right-hand side of the above
equation. It can easily be seen that:
Δωil (k ) ⎡⎣aml − ail + ail −1 ⎤⎦
ΔPkl ( j = i ) =
aml ⎡⎣aml + Δω l (k ) ⎤⎦
(18) This equation with regard to (8) implies that:
ΔPkl ( j = i ) > 0
b) Punishment. In this case Δr (k ) < 0 .
Hence:
Δωil (k ) = max {ϑil (k ), −ωil (k )}
Since a negative-length sub-distance is undefined, the max
operators used in the above equation and (17) assures us that
update rule (18) does not yield to undefined sub-distance.
The max operators are used to satisfy (8-b).
 l
In this case ΔPk ( j = i ) in (19) has to be a negative
scalar. The procedure is the same as part a.
III. BALL AND PLATE SYSTEM
A Ball and Plate system, aforementioned BnP, is an
evolution of the traditional Ball and Beam (BnB) system [13,
14]. It consists of a plate which its slope can be manipulated
in two perpendicular directions and a ball over it. The
behavior of the ball is of interest and can be controlled by
tilting the plate. According to this scheme, various structures
may be proposed and applied in practice. Usually an image
feedback is used to locate the position of the ball. However,
due to its less accuracy and slower sampling rate comparing
touch screen sensors (or simply touch sensors), we opt for a
touch sensor.
The implemented hardware structure in this project, as
outlined in Fig.2, consists of five blocks. However, the whole
system can be viewed as a single block which its input is a
two element vector, i.e. the target angles – (20). The output
of the assumed block is a six element vector which contains
the position and velocity of the ball and the current angles of
the plate. Table I illustrates the related parameters.
TABLE I
PARAMETERS OF BALL AND PLATE SYSTEM AND THEIR UNITS
Symbol Parameter
( xd , y d ) Target position of the ball
( x, y) Current position of the ball
(vx , vy ) Current velocity of the ball
(α x , α y ) Current angels of the plate
αx Plate angel with regard to x axis
αy Plate angel with regard to y axis
(u x , u y ) Control signal
ux plate target angel with regard to x axis
uy plate target angel with regard to y axis
T The third and main section of the BnP system is its
u = ⎡⎣α x , α y ⎤⎦
In this paper we are interested in the following problem:
Simple command of the ball. The objective is to place the
ball on any desired location on the plate surface starting
from an arbitrary initial position.
Before explaining the control system, a summary of the
hardware specifications of the implemented BnP for this
project is given below done in hardware implementation of the
BnP in this project is stated. A complete or even brief
description of the efforts how we made this apparatus is
beyond scope of this paper. But a summary is needed to show
that the plant for RLFC is made roughly and there are many
inaccuracies that a classic controller will definitely be unable
to control the plant. Referring back to Fig.2 each block
Authorized licensed use limited to: Inha University. Downloaded on July 27,2022 at 10:13:55 UTC from IEEE Xplore. Restrictions apply.
function is summarized next.
The Actuating Block
• The actuating block consists of high accuracy
□ stepping motors equipped with precise incremental
encoders (3600 ppr*) coupled to their shafts plus
accurate micro-stepping-enhanced drivers.
• The original step size of the steppers is 0.9 degrees
and reducible by the drivers down to 1 200 of step
size, i.e. 4.5 × 10 −3 degree per step.
• Taking into account the mechanical limitations, the
smallest measurable and applicable amount of
rotation is 0.1 degrees.
Fig. 2. Hardware structure of the implemented BnP system. The dashed
square separates the electronics section from the mechanical parts.
The Sensor Block
• The sensor is a 15 inches touch-screen sensor.
Unit • Sensor output is a message packet sent through
Pixel RS-232 serial communication with 19200 bps†.
• Thus the fastest sampling rate of the whole sensor
Pixel
block is 384 samples per second. This implies that
Pixel per
second the maximum available time for decision making is
−3
Angel step
1
384 ≅ 2.604 ×10 second.
Angel step
• The area of the surface of the touch sensor on
which pressure can be sensed is 30.41 × 22.81cm.
Angel step
• The sensor resolution is 1900×1900 pixels. If the
Angel step sensor sensitivity is uniformly distributed on its
sensitive area, then each pixel is assigned to an area
Angel step
approximately equals to 1.8 × 1.2 mm2 of the surface of
Angel step the sensor.
The Interface Block
(20)
electronic interface. This interface receives commands from
an external device in which the controller has been
implemented and then takes necessary corresponding actions.
Each decision made by the controller algorithm is translated
and formed into a message packet which is then sent to the
interface via a typical serial communication (RS232) or other
communication platforms. Then, the interface sends necessary
signals to actuators. In addition to some low-level signal
manipulation (such as applying low-pass filter to sensor
reading and noise cancelling), upon request from the main
controller, the interface sends current information, such as ball
position and velocity or position of the actuators, to the main
controller.
*
ppr: pulse per rotation
†
bps: bit per second. A measuring unit for serial communication.
 IV. MODIFICATION OF RLFC FOR BALL AND PLATE In Fig.3, the corresponding defined term sets are presented
In section II, RLFC was discussed completely. In this graphically. Number, shape and distribution of the defined
section, necessary modifications for making it applicable to term sets are chosen based on the logical sense and practical
control the implemented BnP system for solving the second experience. There is no obligation for the system not to
problem depicted in the previous section are expressed. perform well if these selections are changed.
Assuming that the covering condition is satisfied, as it is the
A. Primary Design Stage 6
case in Fig.3, in each iteration, there are at most 2 =64 fired
Fig.1 depicts the control architecture as well as signal flows. rules. However, if an extensive search is done among 4900
With regard to Table I, illustrated signals are explained next: rules to discover fired rules, i.e. to calculate each premise
Control signal vector: membership value and check whether it is zero or not, then the
T
u = ⎡⎣u x u y ⎤⎦ (21) processing time would grow out of tolerable bounds. Instead,
since we know the exact location of each term set, if we locate
Plant state vector: each measured value of input variables on their corresponding
T
s = ⎡⎣ x y ux u y ⎤⎦ (22) universe of discourses then those term sets with non-zero
membership value are discovered. Doing this procedure for all
Plant target state vector: six input variables, there will be at most two discovered term
sd = [ xd 0 0]
T
yd (23) sets for each of them. Hence the combinations of these term-
sets using AND operator directly shows us which rules are
Error vector: fired. The mentioned locating procedure can be coded into a
T
e = ⎡e x ey ev x ev y ⎤ (24) software programming language using a set of conditional
⎣ ⎦ statements. Apparently, if there are n defined term sets on a
Where: particular universe of discourse then (n+1) conditional
ex = x − x d , e y = y − y d statements are needed in order to discover the fired rules.
Hence instead of L arithmetic calculations, we are faced with
And since we want to make the ball steady:
S logical comparisons, where:
evx = vx , ev y = v y
S = ∑ i =1 ( n i + 1)
n
(26)
In Fig.1 it is obvious that there are six input variables to the
For our case study: S=32.
controller section of RLFC. Let us arrange them in x vector as
There is no special constraint to be applied to distribution of
written in (25).
T
term sets on the universes of discourse of the output variables.
x = ⎡⎣e x ey vx vy αx α y ⎤⎦ (25) There are M=51 triangular term sets which are uniformly
distributed over each universe of discourse. Fig. 4 illustrates
the location of these term sets.

(a)

Fig. 4 The defined term sets on the universes of discourses of output

variables.

B. Critic: Generation of Reinforcement Signal

Until now, the reinforcement signal is only defined to be an
(b) (c)
Fig. 3 shows the defined term sets on the universe of discourse of (a) evaluation of the system current situation generated by the
e x and e y , (b) v x and v y , (c) α x and α y . For the respective units refer to critic block. No other precise definition could have been given
Table I. by now since the nature of this signal directly depends on the
On the universe of discourse of each input variable, a nature of the plant to be controlled. Referring back to Fig.1, it
specific number of term sets are defined. Let these numbers be is obvious that the input to critic is the Error Signal, (24).
n x , n y , nvx , nvy , nα x and nα y . According to (4), the fuzzy According to this figure the reinforcement signal is a function
rule base contains L rules where: of this vector:
L = n x .n y .nvx .nvy .nα x .nα y r = g (e ) = g (e x , e y ,v x ,v y ) (27)
For our very specific implementation following quantities Apparently, the critic is defined by g(.) and is the
are assigned to the above variables: designer’s choice. The only necessary condition is that this
function should represent the current state of the plant as well
( nx = ny = 7, nvx = nvy = 5, nα x = nα y = 2 ) ⇒ L = 4900 possible as it can be. A proposed general form of this function

Authorized licensed use limited to: Inha University. Downloaded on July 27,2022 at 10:13:55 UTC from IEEE Xplore. Restrictions apply.
is depicted in (28).
r = cx (ex2 + cv vx2 ) + c y (ey2 + cv v y2 )
Equation (28) can be regarded as a revised version of LMS.
Note that the aim of RLFC is to minimize (28). In (28) three
coefficients are defined which are explained next.
balancing coefficient between the velocity of the ball and its
position error. As cv increases, the controller gives more
credit to stabilizing the ball rather than guiding it to the
desired location. cx and c y represent the mutual interaction
between the two actuators. This interaction comes from the
inevitable impreciseness of the mechanical structure of BnP.
According to this mutual interaction, the motion of the ball in
each direction is not only a function of the corresponding
angle of the plate. However, exact values of cx and c y are
one of the mechanical specifications. They can be chosen and
then tuned experimentally or to be learnt.
C. Reinforcement Measure
Having proposed the reinforcement signal, we are seeking
for a suitable function to produce Δωil ( k ) according to (17).
Equation (29) is a proposed function for (14):
⎛ Δr ( k ) sgn ( Δr (k ) ) ⎞
Δr ′(k ) = f ( r ( k ) ) = ⎜ β +λ ⎟ (29)
⎝ rmax r (k ) ⎠
This function consists of two terms, the first one scales the
pure reinforcement signal received from the critic, and the
second one tunes the learning sensitiveness when the plant is
around the target state; actions receive more
reward/punishment as they affect the plant state when it is
around the desired target state. Note that rmax can be easily
calculated using (28).
According to (29) and (15), the reinforcement measure is
given by (30) replaced in (17):
⎛ θ ⎞ ⎛ Δr (k ) sgn(Δr ) ⎞
Δϑil = g. ⎜1 − nl ( k ) ⎟ . ⎜ β +λ ⎟ (30)
⎝ e i
⎠ ⎝ rmax r (k ) ⎠
D. Adding a Priori Knowledge
From a very general point of view, the proposed algorithm
is a search in the space of possible actions. However, it is
possible to add a priori knowledge in order to increase the
learning speed. To describe that, it would be a great help to
explain how the random selection of output term sets takes
place. Digital processors can produce uniformly distributed
random numbers and this is also used in RLFC. First, a
random number is generated by the processor and then it is
checked to which sub-distance (note equation (8)) it belongs.
Then the corresponding number of that sub-distanfce will be
the index of the chosen term set. Let the randomly generated
number for the lth rule be j l . Equation (31) must hold:
j l ∈ Ωl
Authorized licensed use limited to: Inha University. Downloaded on July 27,2022 at 10:13:55 UTC from IEEE Xplore. Restrictions apply.
By this concept, adding a priori knowledge in term of
modifying bounds on j l is a simple procedure. For a typical
(28) example regarding our implemented RLFC, observe the
following rule form:
⎧ex is A11 AND ey is whatever AND ⎫
⎪⎪ ⎪⎪
cv is the IF ⎨vx is A32 AND ey is whatever AND ⎬ THEN u x is B jl
⎪ ⎪
⎪⎩α x is whatever AND α y is whatever ⎪⎭
Apparently, this applies to a set of rules which the first and
third conditions are fixed as mentioned. By referring back to
the depicted term sets in Fig.3 and Fig.4, This clearly indicates
that the sensible choice for this condition is a high deviation in
αx towards a positive direction with regard according to
Cartesian co-ordination system. Thus:
j l ∈ [35,50] (31)
This implies that those insensible choices are ignored for
the mentioned rules.
V. PERFORMANCE RESULTS OF RLFC ON BALL AND PLATE
After the modified RLFC is implemented it is experimentally
applied to the implemented BnP system. The illustrated graphs
in this section are the results of a series of experiments. In all
the experiments, the position of the ball versus time is
collected using the monitoring section of the implemented
BnP. Then these row data is modified by a sort of software
enhancing procedures to avoid huge amount of confusing
graphs. After omitting the time, the x position versus y is
obtained. Then, the obtained points corresponding to a series
of iterations are drawn on a single graph. Units of x and y are
pixels and the origin of the coordinating system is selected as
seen by the touch sensor. Each figure illustrates the touch-
sensitive area of the plate. Thus illustrates the 1900×1900
pixels of the touch sensor. Location of the ball is illustrated
approximately around the mean of the acquired points. Dark
areas on the figures illustrate presence of the ball over the
corresponding areas of the plate. The more each space of the
figure darkens, the more the ball was in the corresponding area
over all observed iterations. The target (desired) location of
the ball in all the illustrated experiences is the centre of the
plate.
In Fig. 5, improvement in behavior of the ball under control
of RLFC is shown. It is observed that after approximately
70000 iterations, an acceptable performance is obtained. Note
that the needed time for iterations normally varies. In our
experiments, 70000 iterations took around 20 minutes. Since
the performance of the system is satisfactory around 70000th
iteration, at this stage RLFC is regarded as a trained system.
However, since we do not omit the learning procedure
afterwards, the comments under the next figures do not
include the term “trained”. Instead, the 70000th iteration is
mentioned as a reference point for a good train of the system.
The control signals relating to the best performance illustrated
in Fig. 5 is shown in Fig. 6.
(31)
 Fig. 7 . Different starting point. Fig. 8 . Human best performance.

VI. CONCLUSION
The main idea of the discussed work was to propose a
human-like controller, capable of learning from the past and
own obtained experiences as well as embedding some prior
knowledge and reasonable facts. Although still far away from
an exact human-like behavior, the result of applying this
controller to a very complex and uncertain plant resulted in
satisfactory performance, especially when compared with a
good human behavior trying to control the same plant. There
are varieties of extensions and modifications to the proposed
Fig. 5. Improvement in behavior of the ball under RLFC control. Top left method, from the form of fuzzy IF-THEN rules to the method
figure corresponds to the first 15000 iterations, where initially a priori of tuning various defined parameters.
knowledge is embedded. Top right and bottom left figures are showing
improvement in performance. After approximately 70000 iterations,
bottom right performance is regarded as acceptable. REFERENCES
[1] R. S. Sutton A.G. Barto , Introduction to reinforcement learning,
MIT Press/Bradford Books, Cambridge, MA, 1998.
In all the aforementioned experiments, the ball is initially [2] L. P. Kaelbling, M. L. Littman, A. W. Moore “Reinforcement
located in the same place. To show that this is not a necessary Learning: A Survey” Journal of Artificial Intelligence. vol. 4, pp.
condition, in Fig. 7 another start point is chosen. It is seen that 237-285, May 1996.
[3] A. B. Rad, P. T. Chan, Wai Lun Lo, C. K. Mok “An Online
since RLFC has not experienced these new states enough Learning Fuzzy Controller” IEEE Trans. Industrial Engineering, vol
before, at first it could not perform well. However, as the ball 50, no. 50, pp. 1016-1021, October 2003.
touches a previously enough-experienced state (shown by an [4] X. Fan, N. Zhang, S. Teng, “Trajectory planning and tracking of ball
arrow), the behavior comes under the control. The number of and plate system using hierarchical fuzzy control scheme” Elsevier
Journal of Fuzzy Sets and Systems, vol. 144, pp. 297-312, 2003.
iterations in this figure is about 3500. [5] S. Awtar, K. C. Craig, “Mechatronic Design of a Ball on Plate
Balancing System” Proc. 7th Mechatronics Forum International
Conference, Atlanta, GA, 2000.
[6] Humusoft User’s Manual “CE 151 Ball&Plate Apparatus”,
Humusoft,
[7] E. H. Mamdani, “Application of fuzzy algorithms for simple
dynamic plant,” Proceedings of IEE, vol. 121, pp. 1585-1588, 1974.
[8] M. Bai, H. Lu, J. Su, Y. Tian, “Motion Control of Ball and Plate
System Using Supervisory Fuzzy Controller,” WCICA 2006, vol. 2,
pp. 8127-8131, June, 2006.
[9] Wang, H. Tian, Y. Sui, Z. Zhang, X. Ding, “Tracking Control of
Ball and Plate System with a Double Feedback Loop Structure,”
Fig. 6 . Control inputs with respect to the best performance in Fig.5. ICMA 2007, August, 2007.
[10] R. Bellman, Adaptive Control Processes: A Guided Tour, Princton
In order to compare the performance of RLFC with that of a University Press, 1961.
human, 10 individuals (all healthy, normal, and matured with [11] L. X. Wang, A Course in Fuzzy Systems and Control , Prentice-Hall
no apparent nervous or muscular disorder) were selected and International Inc.,1997.
[12] A. G. Barto, “Reinforcement Learning”, The Handbood of Brain
asked to control the implemented BnP system. Each individual Theory and Neural Networks, M. A. Arbib, Jaico Publishing House,
was allowed to try the system 10 times. Note that in all The MIT Press, Cambridge, MA, USA, 2006, pp. -804-809.
experiences, the steppers are released and the individuals [13] H. K. Lam, F. H. F. Leung, P. K. S. Tam, “Design of a fuzzy
controller for stabilizing a ball and beam system”, The proceedings
controlled the plate by their own hands. This would indeed of IECON 1999, vol. 2, pp. 520 – 524.
omit the most complicated nonlinearity and impreciseness of [14] E. Laukanen, S. Yurkovich, “A Ball and Beam testbed for Fuzzy
the system: the actuators. In Fig.8 the best performance is identification and Control design,” Proc. American Control Conf.,
June 1993, pp 665-669.
illustrated which was the 7th try of the individual who
possessed the best control over the BnP system.

Authorized licensed use limited to: Inha University. Downloaded on July 27,2022 at 10:13:55 UTC from IEEE Xplore. Restrictions apply.