Journal of Cleaner Production 65 (2014) 184e193

Contents lists available at ScienceDirect

Journal of Cleaner Production

journal homepage: www.elsevier.com/locate/jclepro

Potential capability estimation for real time electricity demand

response of sustainable manufacturing systems using Markov
Decision Process
Zeyi Sun, Lin Li*
Department of Mechanical and Industrial Engineering, University of Illinois at Chicago, 842 W Taylor St. ERF 3057, Chicago, IL 60607, USA

a r t i c l e i n f o a b s t r a c t

Article history: Electricity demand response has been considered as a critical methodology to realize the strategy of
Received 27 March 2013 sustainable development for manufacturing enterprises by effectively reducing the increasing electricity
Received in revised form demand and Greenhouse Gas emissions. Most existing studies about the electricity demand response
27 August 2013
implementation focus on either the supply side management, e.g., policy making, price setting, or the
Accepted 28 August 2013
customer side applications for the end-users in residential and commercial building sectors. As for the
Available online 6 September 2013
industrial sector, only a few papers utilizing the long term scheduling methodology to reduce the
electricity consumption during peak periods are available. Little work has been implemented on the
Keywords:
Real time
decision-making for the real time electricity demand response in industrial manufacturing systems
Electricity demand response considering system throughput constraint. In this paper, an analytical model is established to identify the
Sustainable manufacturing systems optimal energy control actions and estimate the potential capacity of power demand reduction of typical
Markov Decision Process manufacturing systems during the period of demand response event without compromising system
production. Markov Decision Process is used to model the complex interaction between the adopted
demand control actions and the system state evolutions. A numerical case study on a section of an
automotive assembly line is used to illustrate the effectiveness of the proposed approach.

2013 Elsevier Ltd. All rights reserved.

1. Introduction Therefore, it is of high interest for both industry and academia

The electricity demand in the United States is expected to grow
at an average rate of 1.5% per year and the total consumption is
projected to increase by 45% by 2030 (EIA, 2006). It is estimated
that about 347 GW of new generation capacity, which costs about
$300 billion financial investment (Edison Electric Institute, 2006)
are needed to satisfy the growing demand. Additionally, the huge
volume of Greenhouse Gas (GHG) emissions is also expected since
the power generation industry is thought to be the most concen-
trated source of GHG emissions. The industrial sector is a main
contributor to this increasing trend of electricity demand.
Approximately over one-quarter of electricity is consumed by the
industrial sector in the United States (EIA, 2011). Manufacturing
activities dominate industrial energy consumption (Duflou et al.,
2012). It is reported that about 84% of energy-related industry
carbon dioxide emissions and 90% of industry energy consumption
are contributed by manufacturing sector (Schipper, 2006).
* Corresponding author. Tel.: þ1 312 9963045; fax: þ1 312 4130447.
E-mail address: linli@uic.edu (L. Li).
to reduce the huge investment and GHG emissions by sustain-
ability strategy that jointly considers economic, environmental,
and social aspects. The concerns of sustainability in manufacturing
sector have gained wide attention and interests recently. A great
deal of research focusing on the sustainability strategy on the
business level for the whole factory has been conducted. For
instance, Baldwin et al. (2005) developed a manufacturing evolu-
tion model to guide transformations and explore the evolutionary
differences between sustainable and non-sustainable organiza-
tions. Aguado et al. (2012) developed a general approach to help
firms harmonize efficiency and sustainability based on environ-
mental innovation. Heikkurinen and Bonnedahl (2013) proposed a
new sustainability-orientated business strategy to replace the
traditional stakeholder or market-oriented ones. Despeisse et al.
(2012) analyzed environmental principles and industrial practice
to develop a conceptual manufacturing ecosystem model as a
foundation to improve environmental performance. Despeisse
et al. (2013) presented a tactics library to provide a connection
between the generic sustainability concepts and more specific
examples of operational practices for resource efficiency in
factories.

0959-6526/$ e see front matter
2013 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.jclepro.2013.08.033
 Z. Sun, L. Li / Journal of Cleaner Production 65 (2014) 184e193
1.1. Introduction of demand side management-energy efficiency
and demand response
Besides the methods towards sustainability on general business
level for the manufacturing companies, Demand Side Management
(DSM) is also regarded as an effective approach to guide the daily
operations of manufacturing companies and so the goal of sus-
tainability can be achieved. The potential impact of DSM is esti-
mated to be 157e218 GW reduction of non-coincident summer
peak by 2030, or 14e20% below projected levels by Electric Power
Research Institute in a recent study as shown in Fig. 1 (Electric
Power Research Institute, 2009).
Generally, two main types of DSM methods exist: one is energy
efficiency improvement, and the other is electricity demand
response. The objective of energy efficiency related research is to
achieve the same amount of output with less energy consumption
through economic efficiency on a permanent basis. It is universally
regarded as the ‘first fuel’ in making energy decisions due to its
cheaper, cleaner, faster, and easier realization than any other re-
sources (Friedrich et al., 2009). Many investigations about the
policies and the barriers of the implementation of energy efficiency
programs in different countries have been conducted (Bunse et al.,
2011; Trianni et al., 2013; Walsh and Thornley, 2012).
As for the technical point of view, much literature can also be
found. For example, Draganescu et al. (2003) developed the statistic
modeling of machine tool efficiency and specific consumed energy
in machining using the experimental data by Response Surface
Methodology. The energy efficiency for single process was defined
as the ratio of the energy consumed by process itself and the total
energy consumed by the manufacturing machine during process
(including the energy required by other auxiliary systems like
coolant or hydraulic system) (Dietmair and Verl, 2009a, 2009b). The
general energy consumption model of individual manufacturing
process was established by separating the total energy into two
parts, i.e., the fixed part that ensures the readiness of operation, and
the variable part for the process (Dahmus and Gutowski, 2004;
Gutowski et al., 2006). Based on the general energy consumption
model, an online power consumption monitoring model was
developed by Hu et al. (2012). The energy consumption profiles
were also identified to characterize the relationship between pro-
cess variables and energy consumption of milling process (Li et al.,
2013) and predict the total energy consumption of a selected ma-
chine tool performing a turning operation (Li and Kara, 2010).
Furthermore, the strategies of energy efficiency improvement
for single machine manufacturing system were analyzed and
discussed for different manufacturing processes (Abele et al., 2011;
Li et al., 2011). The optimal machining parameters were investi-
gated for the desired power consumption and tool life using the
1200
1000
800
GW
600
400
200
0
Projected Value The Value in Realistic The Value in
Achievable Scenario Maximum Achievable
Scenario
Fig. 1. U.S. summer peak demand saving potentials by demand side management in
2030 under different scenarios.
185
techniques of design of experiment in computer numerical control
(CNC) turning (Bhushan, 2013). The joint improvement of produc-
tivity, cost efficiency, and energy savings was verified in a CNC
machining environment by experiments with different material
removal rates (Anderberg et al., 2010). The production schedule for
a single machine with multiple objective of minimizing both total
energy consumption and tardiness was identified by utilizing a
greedy randomized adaptive search meta-heuristic (Mouzon and
Yildirim, 2008).
Recently, the research on the methodology for the multi-
machine level energy efficiency improvement has also been initi-
ated. The feasibility and energy saving potentials for multi-machine
manufacturing systems have been explored (Li et al., 2012a; Sun
et al., 2011). The energy control opportunities for the multi-
machine manufacturing systems have been analyzed and esti-
mated (Sun and Li, 2013). An analytical model aiming at reducing
the energy waste of the machines during idle periods for multi-
machine manufacturing systems has been established (Li and
Sun, 2013).
Electricity demand response, in contrast, encourages customers
to change their regular usage patterns in response to the variation
of energy cost over time, which has attracted growing attention
from the electric utilities throughout the United States. It is
considered as a key technology enabler (Li, et al., 2012b) and
application area (DeGroff, 2010) of smart grid. The functionality of
the management sub system of the smart grid can be enhanced by
shaping demand profile (Shafiullah et al., 2013). The voltage
collapse of a power system can be effectively prevented (Wang
et al., 2011). The wide adoption of renewable electricity which
plays a critical role in maintaining world average temperature
(Battaglini et al., 2009) can also be further facilitated. The impacts
of the demand response programs are enormous. The average en-
ergy saving ratio of demand response programs was reported to be
65 kWh per kW of peak demand reduction (Siddiqui, 2008). Federal
Energy Regulatory Commission (FERC) has estimated that the
existing demand response resources are about 41,000 MW, repre-
senting 5.8% of 2008 summer peak demand (FERC, 2008). It is
projected an increase to 138,000 MW, representing 14% of peak
demand by 2019 (FERC, 2009).
Generally, existing demand response programs can be catego-
rized into two groups as shown in Fig. 2, price-driven and event-
driven (Goldman et al., 2010). In price-driven program, e.g., Time
of Use (TOU), Critical Peak Pricing (CPP), and Real Time Pricing
(RTP) (Goldman et al., 2010), the end-user electricity rates vary over
time to encourage customers to change their patterns of energy
consumptions. In event-driven program, the customers can be
rewarded for reducing their power consumption in response to
specific triggering events, e.g., extreme local weather, regional
Demand Response
Price-Driven Event-Driven
Time of Use Triggering Events:
Real Time Pricing Local Weather
Critical Peak Pricing Reliability
Fig. 2. Two types of demand response programs.
186 Z. Sun, L. Li / Journal of Cleaner Production 65 (2014) 184e193
transmission congestion, and generation equipment failures. The
decisions on the response policy usually need to be made on a real
time basis since most triggering events cannot be accurately pre-
dicted with a long prediction window and thus the reduction re-
quests are usually issued on short notice. Event frequency and
duration are variant and determined by different applications. For
example, ancillary service applications may entail 80 to 120 events
per year with durations of 10e20 min each (Goldman et al., 2010).
1.2. State-of-the-art of the research on electricity demand response
Extensive studies on electricity demand response have been
conducted. The investigations on general policy of demand response
programs have been concentrated (Greening, 2010; Vassileva et al.,
2012). The supply side management issues in demand response
programs, e.g., price policy (Doostizadeh and Ghasemi, 2012; Faria
et al., 2011), and real time optimal pricing model (Yousefi et al.,
2011; Yu et al., 2012) have been widely investigated. The demand
response simulator that allows studying demand response actions
and schemes in distribution networks has also been established
(Faria and Vale, 2011). In addition, much work on the applications of
the end-users of residential and commercial building sectors has
been implemented. For example, Motegi et al. (2007) introduced a
set of general control strategies and techniques for demand
response of commercial buildings. The real time pricing models for
residential customers have been studied (Chen et al., 2012; Yi et al.,
2013). The methods of thermal storage utilization (Braun, 1990;
Henze et al., 2004) and cooling/heating requirement prediction
(Braun and Chaturvedi, 2002) were developed to reduce the elec-
tricity load of the buildings during the peak periods. The user
comfort and the energy saving or power reduction were emphasized
(Corno and Razzak, 2012; Wang et al., 2012).
As for the end-users in industrial sector, most existing studies
focus on the implementation of the price-driven demand response
programs. For example, Ashok and Banerjee (2001) and Ashok
(2006) developed the mathematical models for obtaining optimal
production schedule with minimum operation costs and energy
costs for a flour plant and a steel plant respectively. Fernandez et al.
(2013) established a “Just-for-Peak” buffer inventory methodology
to reduce the power demand during the peak periods without
compromising system throughput for typical manufacturing sys-
tems. Bego et al. (2013) developed a methodology for the applica-
tion of CPP program offered by San Diego Gas and Electric for
manufacturing enterprises. Wang and Li (2013) proposed a systems
approach to production scheduling so the manufacturing system
can effectively respond to the given Time-of-Use pricing profile
under the production target constraint.
For the event-driven program, Chao and Chen (2005) and Chao
and Zipkin (2008) studied Optional Binding Mandatory Curtail-
ment (OBMC) Plan offered by Pacific Gas & Electric (PG&E)
from the manufacturer’s perspective to identify the optimal pro-
duction strategies when that program is offered. However, the
manufacturing system is simplified and reduced into a single ma-
chine model and thus it cannot address the problem for typical
manufacturing systems with multiple machines and buffers. It can
be seen that the existing literature on the electricity demand
response pays little attention to the applications of event-driven
demand response program for the typical manufacturing systems
with multiple machines and buffers on a real-time basis.
1.3. Challenges to the implementation of event-driven demand
response program for manufacturing enterprises
In fact, most demand response programs in effect today are
event-driven (Goldman et al., 2010). The time duration for the
event is usually notified by a specific set of messages sent to either
the individual end-users or the end-user groups via email or
automated phone call (New York ISO, 2010). Several different types
of notifications may be distributed, e.g., in-day advisory, 2-h
deployment notification, immediate deployment, extending an
event, or termination of an event or test (New York ISO, 2010). After
receiving the notification, the customers have to respond 1) if they
are expected to have their resources participating and 2) the power
reduction they are able to provide. A general implementation
procedure of event-driven demand response program is shown in
Fig. 3.
For those commercial or residential customers where the
quality of service, e.g., the comfort of people, is usually emphasized,
it is not difficult to obtain the answers to the two questions. The
prevalent method is to set the target value of a certain comfort
gauge, e.g., temperature, as an interval rather than a single point.
The action can be usually obtained from a predetermined table.
More sophisticatedly, the tradeoff between the comfort and cost
can be jointly formulated and thus an optimal thermostat control
policy can be obtained to respond to electricity price variation for
the purpose of demand response (Liang et al., 2012).
In industrial sector, the answers can only be identified for some
peripheral or simple operations of specific industrial systems
where the equipment is usually operating under a relatively iso-
lated environment and the interrelationship among the equipment
is low (Lewis, 2007; Lewis et al., 2009; McKane et al., 2008). There is
no effective method to quantitatively estimate the demand reduc-
tion capability under the constraint of production invariant for the
event-driven programs for the complex manufacturing systems
with multiple machines and buffers. The complexity of multi-
machine manufacturing systems leads to high interconnections
among the machines in the system. The random failures of the
machines cannot be accurately predicted. The operation state of a
single machine is jointly determined by both its own reliability and
its adjacent machines and buffers (see Section 2.1 for details).
Therefore, it is difficult to clearly describe the states of the ma-
chines in the system when demand response is implemented and
thus the reduction capability under the constraint of system
throughput invariant is very hard to be obtained on a real time
basis. Consequently, the concerns about production impact, a key
barrier towards industrial participation (Ghatikar et al., 2012),
cannot be effectively resolved and manufacturers are reluctant to
venture their throughput to participate in demand response
programs.
1.4. Objective
The above mentioned status quo motivates the authors to
launch the research on the event-driven demand response program
for the manufacturing enterprises with typical manufacturing
systems with multiple machines and buffers. In this paper, we will
General Implementation Procedure of Event-Driven Demand Response Program
Customer Customer Feedback
Utility Issues
Triggering
Reduction
Decision Decision to Utility
Events Occur Making on Making on and
Request
“Yes/No” “How much” Execution
1 2 3 4 5
Initialization and Notification Decision Making from Execution
from Utility Side Customer Side from Utility
Fig. 3. General implementation procedure of event-driven demand response program.
 Z. Sun, L. Li / Journal of Cleaner Production 65 (2014) 184e193
focus on the real time application for the event-driven demand
response program. An analytical model will be established to
identify the optimal energy control actions and estimate the po-
tential capacity of power demand reduction of typical
manufacturing systems during the period of demand response
event without compromising system production. The estimation
results of the potential demand reduction capability are expected to
facilitate industrial participation in real time demand response
programs to reduce overall energy cost and grid pressure during
peak consumption periods. It can provide a concrete direction for
the realization of sustainability for manufacturing enterprises by
considering interests of both their own production targets and their
power suppliers. The complicated interaction between the adopted
demand response actions and system state evolutions is modeled
by Markov Decision Process (MDP). A numerical case study on a
section of an automotive assembly line is used to illustrate the
effectiveness of the proposed approach. The rest of the paper is
organized as follows. Section 2 introduces system modeling and
solution technique. Section 3 uses a numerical case to verify the
effectiveness of the proposed approach. Conclusions and future
work are discussed in Section 4.
2. Proposed methodology
2.1. Introduction of Markov Decision Process and modeling
assumptions
In Section 2, we develop a dynamic power control model for
effective real time electricity demand response for a typical
manufacturing system under an MDP framework. MDP provides a
mathematical framework for modeling decision-making in the
situations where outcomes are partly random and partly under the
control of a decision maker (Ye, 2011). Both stochastic and deter-
ministic properties of the system can be captured by the model
(Chatterjee and Doyen, 2011). More precisely, at each decision
epoch t under the framework of MDP, the system is in a certain state
st, and the decision maker may choose any action at that is available
in state st. The system responds at the next decision epoch t þ 1 by
0
randomly moving into a new state s and incurring a corresponding
immediate cost C(st, at). The probability that the system moves into
0
the new state s is influenced by both adopted action at and state st.
Specifically, it is given by the state transition probability
0
P(st þ 1 ¼ s jst, at). A value function is also formulated to integrate the
incurred immediate cost between the current and the next decision
epoch and the expectation of the subsequent cost between the next
and the final decision epoch so that different actions can be eval-
uated and therefore the objective function can be established
accordingly (Givan and Parr, 2013).
Today MDP is widely used in a variety of areas including robotics
(Karami et al., 2009; Laroche et al., 1999), automated control (Chang
et al., 2003; Konstantinos and Sascha, 2003), economics (Lave,
1966; Mirman et al., 2008) and manufacturing (Ambani et al.,
2009; Li and Sun, 2013). Recently, Li and Sun (2013) developed a
mathematical model to describe the dynamic energy control in a
typical manufacturing system with multiple machines and buffers
under MDP framework by utilizing machine’s idle periods to reduce
energy consumption. It focused on the opportunity of the blockage/
starvation (idle) machines in the system by identifying an optimal
energy consumption state during each idle period, while the
reduction opportunity for the rest of the machines in the system is
not considered. In addition, it defined the buffer state by using “full,
empty, and in-between” rather than the detail number of parts in
the buffer location. The buffer state transition was assumed to
follow specific probability distribution rather than the evolution of
machine state and action adopted (Li and Sun, 2013). Therefore, the
187
real system behavior cannot be accurately described. In this paper,
MDP is further advanced by improving the above drawbacks to
model the dynamic power control for demand response imple-
mentation for a multi-machine system considering the power
reduction opportunities for all the machines in the system. In the
proposed model, the system state is composed of machine opera-
tion state, machine energy consumption state, machine produc-
tivity state, and buffer state. The power control strategy is
considered as the action of MDP model. The transition probability is
used to describe the uncertainty of system state evolution when
power control actions are performed due to the random failure of
the machines. The objective is to find a set of optimal actions that
can make the cost, the system power demand during the demand
response event, be minimized under the constraint of system
throughput invariant. Consequently, an approximate method is
used to find the near-optimal solution of a series of power control
actions and thus the power reduction potential can be obtained.
Before we delve into our model, several assumptions are listed
as follows:
1) The duration of the demand response event in this model is
relatively short, e.g., 10e20 min, and thus the power demand
during the demand response event is calculated by dividing the
total electricity usage by the time length of the demand
response event;
2) The cycle times of all the machines in the system are the same;
3) The machine failure is time-dependent;
4) The time length of the period between the arrival of the noti-
fication of the demand response event and the ending of the
demand response event equals to the product of a positive
integer and system cycle time;
5) The time horizon is slotted with the durations equal to the
system cycle time, the decision epoch is at the beginning or
ending of each cycle;
6) The transitions of machine operation state and buffer state are
assumed to occur at the beginning or ending of each cycle;
7) For machine energy state, besides the three conventional en-
ergy states, i.e., full operation, ready for operation, and turned-
off (Li and Kara, 2010), we consider hi different energy hiber-
nation states with partial power consumption of “ready for
operation” state;
8) No production activities can be implemented when machine is
in hibernation energy mode;
9) No additional power control actions can be adopted when
machine is in the power state transition process; and
10) The first machine is never starved and the last machine is never
blocked.
2.2. Markov Decision Process modeling
2.2.1. System state variables and state space
Consider a typical tandem manufacturing system with I ma-
chines (denoted by squares) and I-1 buffers (denoted by circles) as
shown in Fig. 4 (Li and Meerkov, 2008).
The system state space includes both machine state and buffer
state. Here we advance the method used in our previous study (Li
and Sun, 2013) to define the system state as follows. For the ma-
chine state, three kinds of information are recorded: operation
state, energy state, and productivity state.
M1 B1 M2 M B MI
Fig. 4. A tandem manufacturing system with I machines and I-1 buffers.
188 Z. Sun, L. Li / Journal of Cleaner Production 65 (2014) 184e193

Machine operation state includes operation, blockage, starva-
tion, and breakdown (Li and Meerkov, 2008). Blockage means that
the machine itself is not failed while the completed part cannot be
delivered to the downstream buffer due to the breakdown of spe-
cific downstream machines. Starvation means that the machine
itself is not failed while there is no incoming part from the up-
stream buffer due to the breakdown of specific upstream machines.
Let OPi, DNi, BLi and STi denote the above four operation states of
machine i. Accordingly, the set of operation states of machine i can
be described as SOi ¼ {OPi, BLi, STi, DNi},i ¼ 1, 2, .I.
For machine energy state, besides the three conventional energy
states, i.e., full operation, ready for operation, and turned-off (Li and
Kara, 2010), we consider hi different energy hibernation states with
partial power consumption of “ready for operation” state as shown
in Fig. 5. Let Hqi, qi ¼ 1, .hi, denote these hibernation states. Thus,
the set of energy states of machine i can be denoted as SEi ¼ {Fi, Ri,
Hqi, Oi}, qi ¼ 1, .hi and i ¼ 1, 2, .I, where Fi, Ri, and Oi represent the
state of full operation, ready for operation, and turned-off of ma-
chine i respectively. In this research, we also assume that the en-
ergy state of machine i is Ri at the beginning of each cycle if the
machine operation state is not breakdown and no power control
strategy is implemented. On the one hand, it can be automatically
switched to Fi if the operation state of machine i is OPi and switched
back to Ri at the ending of the cycle when the processing is
completed (Note that the transition between Ri and Fi is assumed to
be instantaneous). On the other hand, it will be kept in Ri if the
operation state of machine i is BLi/STi.
Machine productivity state is newly defined in this research to
describe if machine i is throughput bottleneck or not since we
consider the energy control opportunity for all the machines
rather than only blockage and starvation machines in (Li and Sun,
2013). Let BNi denote that machine i is the throughput bottleneck
of the system and NBNi denote that machine i is not the
throughput bottleneck of the system. Therefore, the set of
productivity states of machine i can be represented by
SPi ¼ {BNi,NBNi},i ¼ 1, 2, .,I.
For buffers, there is no energy consumption and the buffer state
is defined as the number of the jobs waiting for being processed in
the buffer location, which is more detailed than the definition of
“full, empty and in-between” in (Li and Sun, 2013). Hence, the set of
states for buffer i can be described asBi ¼ {0, 1, 2, .,Ni}, i ¼ 1, 2,
.,I  1.
Max
Full Operation State: Fi
Ready for Operation State: Ri
Energy Consumption
In summary, the set of system states can be denoted by S ¼ {SOi,
SPi, SEi, Bi}, where i ¼ 1, 2, .,I for SOi, SPi, and SEi; i ¼ 1, 2, .,I  1 for
Bi.
2.2.2. Power control action
Power control action for the purpose of demand response for
machine i can be divided into three categories. The first one is to
keep the original energy state (K-action), which can be adopted
under the following three situations: (1) the power state cannot be
adjusted to respond to the demand reduction requirement due to
some specific constraints; (2) the machine can be kept on a certain
hibernation power level; and (3) the operation state of machine i is
DNi.
The second one (H-action) is to adjust the energy state from Ri to
Hqi or Oi. It notifies machine i to operate on a lower power level Hqi
or Oi as a demand response action.
The third one (W-action) is to adjust the energy state of machine
i from Hqi or Oi toRi. It is used to waken the hibernation machines to
resume operation. Table 1 summarizes the possible power adjust-
ment actions applied for different scenarios.
Therefore the set of power control actions for machine i can be
denoted as Ai ¼ {Ki, Hi, Wi},i ¼ 1, 2, .,I, where Ki, Hi, and Wi denote
K-action, H-action, and W-action for machine i respectively. The
actions for the system at decision epoch t can be described as
At ¼ ðAt1 ; :::; Ati ; :::; AtI Þ, whereAti is the action adopted for machine i
at decision epoch t.
2.2.3. System state transition
We assume that the productivity state of machine i during the
period of demand response event can be modeled as a static
property which is obtained by the short-term operational records
of previous shift by the data driven method developed in (Li et al.,
2009). The energy state of machine i at the next decision epoch is
jointly determined by both its operation state and power control
action executed at the current epoch. Since the outcome of energy
state strictly follows power control action, we focus on the calcu-
lation of the transition of buffer state and machine operation state
as follows.
The buffer state at decision epoch t þ 1 can be obtained by (1)
based on the states and the power control actions adopted at de-
cision epoch t of upstream and downstream machines.



Btþ1
i
¼ Bt
i þ I St
O i
; S t
E i
; At
i  I S t
O iþ1
; S t
E iþ1
; At
iþ1 ; 0  Bti  Ni
(1)
where
(

 1 StOi ¼ OPi and StEi ¼ Ri and Ati ¼ Ki
I StOi ; StEi ; Ati ¼
0 StOi sOPi or StEi ¼ Hqi =Oi or Ati ¼ Hi

(2)
Energy Hibernation State: H1

Table 1
Energy Hibernation State: H2 Power adjustment actions for different scenarios.

Operation state Energy state Power control action Target power level

OPi Ri K-action Ri
OPi Ri H-action Hqi or Oi
OPi Hqi or Oi K-action Hqi or Oi
Energy Hibernation State: OPi Hqi or Oi W-action Ri
BLi/STi Ri K-action Ri
BLi/STi Ri H-action Hqi or Oi
Power Off State: Oi BLi/STi Hqi or Oi K-action Hqi or Oi
Min BLi/STi Hqi or Oi W-action Ri
DNi Oi K-action Oi
Fig. 5. Machine energy states.
 Z. Sun, L. Li / Journal of Cleaner Production 65 (2014) 184e193 189

Refer to the literature focusing on the statistical methods for Since the power demand during demand response event is
machine reliability (Meeker, and Escobar, 1998), we assume Li, the assumed to be obtained by dividing the electricity consumption by
random variable of lifetime of machine i, follows Weibull distri- the time duration of the event, the objective function can be
bution with shape parameter and scale parameter considering the formulated as a minimization problem about the incurred elec-
non-memoryless characteristic of machine lifetime. The probability tricity consumption from decision epoch t to the ending of the
that machine i goes into failure or not failure at the next decision demand response event as shown in (10).
epoch t þ 1, given it is not in failure at the current decision epoch t
!
can be described by (3) and (4) respectively. X

min CðS t ; At Þ þ PrðS tþ1 ¼ S jS t ; At ÞVtþ1 S ’
0
(10)

  At
 S ’ ˛S
Pr Stþ1
Oi ¼ DN StOi sDN ¼ PrðLi < t þ TCÞ (3)
where

 
 t


 
Pr Stþ1
Oi sDN SOi sDN ¼ PrðLi  t þ TCÞ (4)
rq rq rq q
CðSt ; At Þ ¼ Si˛IHt min Ti ; TC $Pi þ max TC  Ti ; 0 $Pi
At the same time, we also assume Di, the random variable of


 
repair time of machine i, follows Exponential distribution (Dallery, þ Si˛IWt min Tiqr ; TC $Piqr þ max TC  Tiqr ; 0 $Pir
1994). The probability that machine i completes or does not com- f
plete the repair at the next decision epoch t þ 1, given it is in repair
þ Si˛IOP Pi $TC þ Si˛IBS Pir $TC
Kt Kt

at the current decision epoch t can be described by (5) and (6) (11)
respectively.

  IKOPt is the set of machines in operation state and K-action is adopted
 t at decision epoch t, IKBSt is the set of machines in blockage or star-
Pr Stþ1
Oi sDNi SOi ¼ DNi ¼ PrðDi < t þ TCÞ (5)
vation state and K-action is adopted at decision epoch t, IHt is the set

  of machines that H-action is adopted at decision epoch t, and IWt is
 t
Pr Stþ1
Oi ¼ DNi SO i
¼ DN i ¼ PrðDi  t þ TCÞ (6) the set of machines that W-action is adopted at decision epoch t.
PrðS tþ1 ¼ S ’ j:S t ; At Þ is the probability of St þ 1 given St and At; and
The probability that machine i is in a starvation or blockage state Vt þ 1(S0 ) is the energy consumption from decision epoch t þ 1 to the
can be described by (7) and (8) respectively. ending of the demand response event.



 To ensure that the production is not influenced due to the
Pr Stþ1
Oi ¼ STi ¼ Pr Stþ1 tþ1
Oi sDNi $Pr Bi1 ¼ 0 $
demand response actions, the system throughput constraint


 need be considered. When the production of a certain machine is
Pr Stþ1 tþ1
Oi1 ¼ DNi1 þ Pr SOi sDNi $ (7) strategically stopped for peak demand reduction purpose, the


 downstream buffer contents will decrease. One previous study
Pr Btþ1 tþ1
i1 ¼ 0 $Pr SOi1 ¼ STi1 (Chang et al., 2007) showed that full utilization of buffer con-
tents for maintenance purposes significantly influences the



 system throughput. To overcome this challenge, we refer to the
Pr Stþ1
Oi ¼ BLi ¼ Pr S tþ1
Oi sDN i $Pr Btþ1
i ¼ Ni $
results of (Sun and Li, 2013) and consider a partial buffer utili-



Pr Stþ1 tþ1 zation policy as shown in (12) and (13) to ensure the throughput
Oiþ1 ¼ DNiþ1 þ Pr SOi sDNi $ (8)


 invariant.
Pr Btþ1
i
¼ Ni $Pr Stþ1
Oiþ1 ¼ BLiþ1
Ati sHi if Btþ1
i < ð1  ui Þ$Ni ; when StEi ¼ Ri and StOi ¼ OPi
The probability that machine i is in operation state can thus be
described by (9).
(12)




Pr Stþ1
Oi ¼ OPi ¼ Pr Stþ1
Oi sDNi  Pr SOi
tþ1
¼ STi


 Pr Stþ1 ¼ BLi
Oi
Therefore, the probability of system operation state transition
between the current decision epoch and the next decision epoch
can be calculated by using (1) to (9) when At ¼ ðAt1 ; :::; Ati ; :::; AtI Þ is
adopted based on a given St. Note that in this research, the structure
of the buffer state transition is advanced to be determined by the
machine state evolution and energy control action adopted rather
than the assumed probability distribution used in (Li and Sun,
2013). Thus, a set of more general formulations of the probability
of the system state transition rather than specific (state-action)
scenario-based formulations (Li and Sun, 2013) can be obtained
accordingly.
2.2.4. Optimization formulation
The optimal power control actions for the system during the
period of the demand response event need be determined based on
online information in order to identify the potential electricity
demand reduction under the constraint of system throughput.
(9)
Ati ¼ Wi if Btþ1
i < ð1  ui Þ$Ni ; when StEi ¼ Hqi =Oi and StOi ¼ OPi
(13)
where ui is the utilization ratio of buffer i. Formula (12) implies that
the H-action that makes the downstream buffer level be lower than
the predetermined value cannot be adopted when the operation
state of machine i is OPi and energy state is Ri . Formula (13) shows
that a W-action has to be adopted when the downstream buffer
level is expected to be lower than the predetermined value when
the hibernation machine is in operation state.
In addition, since the throughput bottleneck machine influences
the system throughput with the greatest impact; a high priority to
continue operations should be given to minimize the impact on
throughput. The constraint can be described by (14).
Ati ¼ Ki if SPi ¼ BNi (14)
It implies that H-action cannot be adopted for the throughput
bottleneck machine.
190 Z. Sun, L. Li / Journal of Cleaner Production 65 (2014) 184e193

M1 B1 M2 B2 M3 B3 M4 B4 M5

Fig. 6. A five-machine and four-buffer serial line.

0
Finally, the last machine of the system cannot be stopped for d) Initialize value function V t ðS t Þ for all St with zero, t˛TD
demand response purpose; otherwise the system throughput is 2. Choose the sample path un
influenced during the demand response event. It is formulated as 3. For t ¼ 0 to tP, or t˛TP , follow the sample path chosen in step 2
shown in (15). to obtain the system state S nt (t ¼ tD). (In other word, we keep
taking K-actions for all the t’s belonging to TP.)
AtI ¼ KI (15) 4. For t ¼ tD to T, or t˛TD _
a) Choose a random sample of outcomes Utþ1;z 3U repre-
senting possible realizations of the machine up/down state
2.3. Solution technique evolution between t and tþ1
b) Solve the Equation (16)
In Sections 2.2, the joint production-energy model for a typical
manufacturing system with multiple machines and buffers has
h  
Ant ¼ arg minn Ct Snt ; At
been established by MDP framework. Dynamic programming is At ˛At
usually used as a typical tool to solve MDP. The algorithm begins at n1
i
the final decision epoch and steps backward by looping over all the þS b tþ1;z ptþ1 ðutþ1 ÞV tþ1 ðStþ1 Þ (16)
utþ1 ˛ U
possible states and available actions until the optimal action for
current epoch is obtained (Bellman, 1957). where pt þ 1(ut þ 1) is the probability that the evolution realization
Although the concept of the backward recursion is elegant, the between decision epoch t and t þ 1 is ut þ 1, which is approximately
usefulness is limited due to the “curse of dimensionality”, which obtained as 1/z, andS tþ1 ¼ ST ðS nt ; At ; utþ1 Þ, whereST($) denote the
requires the algorithm to loop over all the states and actions, system state transition function
n1
leading to computational intractability even for very small prob- c) Update value function V t ðS t Þ for next sample path by (17)
lems (Powell, 2011). The alternative is to use a forward method
which proceeds by estimating the approximation of value functions ( _n
n vt S t ¼ S nt
iteratively to obtain an approximate solution (Powell, 2011). A set of V t ðS t Þ ¼ n1 (17)
sample paths is randomly generated to simulate the evolution of V t ðS t Þ otherwise
system state. The algorithm runs from first decision epoch to the
final one along with each sample path iteratively. The value func- where
tion is updated accordingly with each step and used for the next 0 1
iteration. X
_n B n1 C
The benefit of the aforementioned forward algorithm is that it vt ¼ minn @Ct ðS nt ; At Þ þ ptþ1 ðutþ1 ÞV tþ1 ðS tþ1 ÞA
can get around the difficulty of looping over all states and the At ˛At _
utþ1 ˛Utþ1;z
challenge of expectation calculation. The detail procedure of the
solution technique referring to literature of Powell (2007) for de- (18)
cision making is listed as follows.
  
d Compute S ntþ1 ¼ ST S nt ; Ant ; untþ1 (19)
1. Initialization
a) Index decision epochs t from 0 to T
b) Randomly generate M different sample paths un (n ¼ 1, 2,
.M) to describe M different possible paths of up/down 5. n ¼ n þ 1. If n  M, go to step 2, otherwise calculate the
state evolution for all the machines in the system from the reduction potential by (20)
given state when notification arrives to the last decision
_n1
epoch. Let U be the set of all sample paths,
_ vt
REDP ¼ PREG  (20)
Ut ðt ¼ 1; 2; :::; TÞ be the subset of U that includes M evo- TDR
lution
_
realizations between the _ decision epoch t  1 and t,
Ut;z be the non-empty subset of Ut that includes z elements
(z  M), and unt ðt ¼ 1; 2; :::; TÞ be the specific evolution
realization of sample path n between the decision epoch where REDP is the power reduction potential, PREG is the regular
t  1 and t. power consumption level during the period of demand response
c) Set n ¼ 1 event, and TDR is the duration of demand response event.

Table 2
Basic settings of each machine.

MTBF (min) Scale Shape MTTR (min) Exponential Power of ready for Cycle time (min)
parameter gi parameter ki parameter li operation state (kW)

M1 100 111.39 1.5766 4.95 0.2020 21 0.5

M2 45.6 51.1 1.6532 11.7 0.0855 14 0.5
M3 98.8 110.9 1.7174 15.97 0.0626 20 0.5
M4 217.5 239.1 1.421 27.28 0.0367 16 0.5
M5 109.4 122.1 1.591 18.37 0.0544 13 0.5
 Z. Sun, L. Li / Journal of Cleaner Production 65 (2014) 184e193 191

Table 3 Table 5
Basic settings of each buffer. Comparison of power consumption & throughput between baseline model and
demand response model.
Buffer1 Buffer2 Buffer3 Buffer4
Baseline Demand Difference
Capacity 70 40 30 42
model response
Initial contents 50 8 22 28
model
(number of parts)
Power Mean 80.21 62.08 Power consumption
demand reduction (kW)/
3. Case study (kW) Reduction percentage
95% (77.43, (58.78, 18.13/22.59%
confidence 82.99) 65.38)
Consider a five-machine and four-buffer manufacturing line as interval
shown in Fig. 6. The real data from our industrial partner like mean Throughput Mean 623.13 621.66 Mean throughput
time between failures (MTBF), mean time to repair (MTTR), cycle (unit) difference
time of each machine, and the assumed power of “ready for oper- 95% (590.13, (591.66, 0.2%
confidence 656.13) 651.66)
ation state” is listed in Table 2. Machine 2 is identified as the
interval
throughput bottleneck machine in the system by the method
developed by Li et al. (2009). The real data for each buffer including
buffer capacity and initial contents are listed in Table 3. The
Table 6
assumed different power modes are demonstrated in Table 4.
Comparison of buffer contents (number of parts) at the end of the 8-h shift.
Assume a demand response event that is notified at the 45th
minute of an 8-h shift, 15 min in advance of its beginning. The Buffer 1 Buffer 2 Buffer 3 Buffer 4
duration of the event is 15 min. The reduction capability of system Baseline Mean 67.3 9.4 5.5 14.7
power demand need be identified and replied to the utility. In the model 95% confidence (62.9, 71.6) (6.3, 12.5) (2.7, 8.3) (8.3, 21.2)
case study, the simulation model is established by ProModel to interval
Demand Mean 67.7 10.4 5.3 13.6
simulate the manufacturing system as described by Fig. 6 with response 95% confidence (64.5, 70.8) (7.2, 13.6) (3.2, 7.4) (6.9, 20.5)
parameter settings shown from Table 2 to Table 4. ProModel is a model interval
discrete event simulation software package developed by ProMo-
del, Inc (ProModel, 2013). It is used for evaluating, planning or
designing manufacturing, warehousing, logistics and other opera- and estimate the potential capacity of demand reduction based on
tional and strategic situations. The algorithm proposed in Section II online information of typical manufacturing systems during the
is called when the demand response event occurs. The number of period of demand response event without compromising system
random sample paths M is set to be 30 and z is set to be 10. The production. The energy reduction opportunity for all the machines
algorithm is executed by a desktop with Intel(R) Core TM(2) Quad in the system is considered. An approximate dynamic program-
2.83GHZ processor, and 4 GB memory. The computation time is ming method utilizing random generated sample paths is used to
about 600 s. The optimal demand control actions during the pe- find the near optimal solutions for the problem on a real time basis.
riods of event can thus be identified and used as the feedback to the The demand reduction potentials can thus be obtained by applying
simulation model. the developed techniques, which can help manufacturers quanti-
After running 30 replications of an 8-h shift for both baseline tatively estimate their capability of demand response.
model (demand response is not executed) and demand response For future work, we may consider the demand response event
model with buffer utilization ratio of 0.5, the system throughput with longer duration where the problem formulation of the power
and power consumption during the demand response event are demand during the event is more complex rather than just taking
shown in Table 5. average through the short duration. The integration of both price-
It can be observed that the power demand during the period of driven and event-driven programs can also be studied. In addi-
the demand response event can be cut by about 22% and system tion, we could focus on a joint cost model considering both energy
throughput of the whole shift is almost unchanged. and throughput in order to identify an optimal tradeoff between
Concern about the buffer consumption by implementing demand the two factors for the decision making of manufacturing
response, we also compare the buffer level at the end of the 8-h shift companies.
between the baseline model and the demand response model as
shown in Table 6. The buffer at the end of the shift when demand
response is executed is very close to the scenario where no demand
Abbreviations
response is implemented; therefore it can be inferred that the impact
on the system throughput of the next shift can be ignored.
At demand response actions adopted for the whole system
at decision epoch t
4. Conclusion
Bti the content of buffer i at decision epoch t, i ¼ 1, 2, .,I  1
Fi ”full operation” energy state of machine i
In this paper, using Markov Decision Process, an analytical
hi number of energy hibernation states of machine i
model is established to identify the optimal energy control actions
Hqi a certain energy hibernation state of machine i
i index of the machines in the system, i ¼ 1, 2, ..., I
Table 4
Different power states.
Ni the capacity of buffer i
f
Pi power of machine i in “full operation” state
Energy Operation Ready for Shallow Median Deep Turned-off
Piq power of machine i in a certain Hqi state
consumption operation sleep sleep sleep rq
state Pi average transition power of machine i from “ready for
operation” state to Hqi state, qi ¼ 1, .hi
Power level 105% 100% 50% 30% 10% 0%
Pir power of machine i in “ready for operation” state
192 Z. Sun, L. Li / Journal of Cleaner Production 65 (2014) 184e193
qr
Pi average transition power of machine i from Hqi state to
“ready for operation” state, qi ¼ 1, ...hi
Pit power level of machine i at decision epoch t
qi energy hibernation state index of machine i, qi ¼ 1, 2, .,hi
Ri “ready for operation” energy state of machine i
St system state at decision epoch t
t index of the decision epochs, t ¼ 0, 1, .,tP, tD, .,T
tD the first decision epoch belongs to TD
tP the last decision epoch belongs to TP
TD the set of decision epochs between the time that demand
response event begins and ends
TP the set of decision epochs between the time that the
notification of demand response event arrives and
demand response event begins
Tiqr transition time of machine i from Hqi state to “ready for
operation” state, qi ¼ 1, .hi
Tirq transition time of machine i from “ready for operation”
state to Hqi state, qi ¼ 1, .hi
TC system cycle time
ui utilization ratio of buffer i
References
Abele, E., Sielaff, T., Schiffler, A., Rothenbucher, S., 2011. Analyzing energy con-
sumption of machine tool spindle units and identification of potential for im-
provements of efficiency, Glocalized Solutions for Sustainability in
Manufacturing. In: Proceedings of the 18th CIRP International Conference of
Life Cycle Engineering, May 2e4, 2011. Technische University Braunschweig,
Braunschweig, Germany.
Aguado, S., Alvarez, R., Domingo, R., 2012. Model of efficient and sustainable im-
provements in a lean production system through processes of environmental
innovation. J. Clean. Prod. 47, 141e148.
Ambani, S., Li, L., Ni, J., 2009. Condition-based maintenance decision-making for
multiple machine systems. J. Manuf. Sci. E-T. ASME 131 (3), 031009-1.
Anderberg, S.E., Kara, S., Beno, T., 2010. Impact of energy efficiency on computer
numerically controlled machining. Proc. IMechE., Part B: J. Eng. Manufacture
224, 531e541.
Ashok, S., 2006. Peak-load management in steel plants. Appl. Energ. 83 (5),
413e424.
Ashok, S., Banerjee, R., 2001. An optimization mode for industrial load management.
IEEE T. Power Syst. 16 (4), 879e884.
Baldwin, J.S., Allen, P.M., Winder, B., Ridgway, K., 2005. Modeling manufacturing
evolution: thoughts on sustainable industrial development. J. Clean. Prod. 13,
887e902.
Battaglini, A., Lilliestam, J., Haas, A., Patt, A., 2009. Development of SuperSmart
Grids for a more efficient utilisation of electricity from renewable sources.
J. Clean. Prod. 17, 911e918.
Bego, A., Li, L., Sun, Z., 2013. Identification of reservation capacity in critical peak
pricing electricity demand response program for sustainable manufacturing
systems. Int. J. Energ. Res.. http://dx.doi.org/10.1002/er.3077.
Bellman, R., 1957. Dynamic Programming. Princeton University Press, Princeton, U.S.
Bhushan, R.K., 2013. Optimization of cutting parameters for minimizing power
consumption and maximizing tool life during machining of Al alloy SiC particle
composites. J. Clean. Prod. 39, 242e254.
Braun, J.E., 1990. Reducing energy costs and peak electrical demand through
optimal control of building thermal storage. ASHRAE Trans. 96 (2), 839e848.
Braun, J.E., Chaturvedi, N., 2002. An inverse Grey-Box model for transient building
load prediction. Int. J. HVAC&R Res. 8 (1), 73e100.
Bunse, K., Vodicka, M., Schönsleben, P., Brülhart, M., Ernst, F.O., 2011. Inte-
grating energy efficiency performance in production management e gap
analysis between industrial needs and scientific literature. J. Clean. Prod.
19, 667e679.
Chang, H.S., Fard, P.J., Marcus, S.I., Shayman, M., 2003. Multi-time scale Markov
decision processes. IEEE T. Automat. Contr. 48 (6), 976e987.
Chang, Q., Ni, J., Bandyopadhyay, P., Biller, S., Xiao, G., 2007. Maintenance oppor-
tunity planning system. J. Manuf. Sci. E-T. ASME 129 (3), 661e668.
Chao, X., Chen, F.Y., 2005. An optimal production and shutdown strategy when a
supplier offers an incentive program. Manufacturing Serv. Operations Manage.
7 (2), 130e143.
Chao, X., Zipkin, P.H., 2008. Optimal policy for a periodic-review inventory system
under a supply capacity contract. Oper. Res. 56 (1), 59e68.
Chatterjee, K., Doyen, L., 2011. Energy and mean-payoff parity Markov Decision
Processes. In: Proceedings of the 36th International Symposium on Mathe-
matical Foundations of Computer Science (MFCS’11), August 2011, pp. 206e218.
Warsaw, Poland.
Chen, Z., Wu, L., Fu, Y., 2012. Real-time price-based demand response management
for residential appliances via stochastic optimization and robust optimization.
IEEE T. Smart Grid 3 (4), 1822e1831.
Corno, F., Razzak, F., 2012. Intelligent energy optimization for user intelligible goals
in smart home environments. IEEE T. Smart Grid 3 (4), 2128e2135.
Dahmus, J., Gutowski, T., 2004. An environmental analysis of machining. In: Pro-
ceedings of IMECE04 2004 ASME International Mechanical Engineering
Congress and Exposition, November 13e20, 2004, pp. 643e652. Anaheim,
California, U.S.
Dallery, Y., 1994. On modeling failure and repair times in stochastic models of
manufacturing systems using generalized exponential distributions. Queueing
Syst. 15, 199e209.
DeGroff, D., 2010. Green environment: decision-making and power utility optimi-
zation towards smart-grid options. Smart Grid Renew. Energy 1, 32e39.
Despeisse, M., Ball, P.D., Evans, S., Levers, A., 2012. Industrial ecology at factory level
e a conceptual model. J. Clean. Prod. 31, 30e39.
Despeisse, M., Oates, M.R., Ball, P.D., 2013. Sustainable manufacturing tactics and
cross-functional factory modeling. J. Clean. Prod. 42, 31e41.
Dietmair, A., Verl, A., 2009a. A generic energy consumption model for decision
making and energy efficiency optimization in manufacturing. Int. J. Sust. E. 2
(2), 123e133.
Dietmair, A., Verl, A., 2009b. Energy consumption forecasting and optimization for
tool machines. Mod. Machinery Sci. J. 2 (2), 62e67.
Doostizadeh, M., Ghasemi, H., 2012. A day-ahead electricity pricing model based on
smart metering and demand-side management. Energy 46, 221e230.
Draganescu, F., Gheorghe, M., Doicin, C.V., 2003. Models of machine tool efficiency
and specific consumed energy. J. Mater. Process. Tech. 141 (1), 9e15.
Duflou, J.R., Sutherland, J.W., Dornfeld, D., Herrmann, C., Jeswiet, J., Kara, S.,
Hauschild, M., Kellens, K., 2012. Towards energy and resource efficient
manufacturing: a processes and systems approach. CIRP Ann.-Manuf. Techn. 61,
587e609.
Edison Electric Institute, 2006. Rising Electricity Costs: a Challenge for Consumers,
Regulators, and Utilities. Washington D.C., U.S http://www.eei.org/whatwedo/
PublicPolicyAdvocacy/StateRegulation/Documents/rising_electricity_costs.pdf
(06.04.2013).
EIA (Energy Information Administration), 2006. Annual Energy Outlook 2006.
Washington D.C., U.S http://www.scag.ca.gov/rcp/pdf/publications/1_2006Ann
ualEnergyOutlook.pdf (06.04.2013).
EIA (Energy Information Administration), 2011. Electric Power Annual 2010.
Washington D.C., U.S http://www.eia.gov/electricity/annual/pdf/epa.pdf (07.04.
2013).
Electric Power Research Institute, 2009. Assessment of Achievable Potential from
Energy Efficiency and Demand Response Programs in the U.S. (2010-2030). Palo
Alto, CA, U.S http://www.edisonfoundation.net/IEE/Documents/EPRI_Summary
AssessmentAchievableEEPotential0109.pdf (06.04.2013).
Faria, P., Vale, Z., 2011. Demand response in electrical energy supply: an optimal real
time pricing approach. Energy 36 (8), 5374e5384.
Faria, P., Vale, Z., Soares, J., Ferreira, J., 2011. Demand response management in
power systems using a particle swarm optimization approach. IEEE Intell. Syst..
http://dx.doi.org/10.1109/MIS.2011.35.
FERC (Federal Energy Regulatory Commission), 2008. Assessment of Demand
Response and Advanced Metering. Washington D.C., U.S http://www.ferc.gov/
legal/staff-reports/12-08-demand-response.pdf (06.04.2013).
FERC (Federal Energy Regulatory Commission), 2009. A National Assessment of
Demand Response Potential. Washington D.C., U.S http://www.ferc.gov/legal/
staff-reports/06-09-demand-response.pdf (06.04.2013).
Fernandez, M., Li, L., Sun, Z., 2013. Just-For-Peak buffer inventory for peak electricity
demand reduction of manufacturing systems. Int. J. Prod. Econ.. http://dx.doi.
org/10.1016/j.ijpe.2013.06.020.
Friedrich, K., Eldridge, M., York, D., Witte, P., Kushler, M., 2009. Saving Energy Cost-
effectively: a National Review of the Cost of Energy Saved through Utility-sector
Energy Efficiency Programs, American Council for an Energy-efficient Economy.
Washington D.C., U.S http://dandelion-patch.mit.edu/afs/athena.mit.edu/dept/
cron/project/urban-sustainability/Energy%20Efficiency_Brendan%20McEwen/
LG%20EE%20Policy%20Sectors/Costs/Friedrich%20et%20al%202009%20-%20EE%
20Costs.pdf (06.04.2013).
Ghatikar, G., McKane, A., Goli, S., Therkelsen, P., Olsen, D., 2012. Assessing the
Control Systems Capacity for Demand Response in California Industries. Law-
rence Berkeley National Laboratory, Berkeley, CA, U.S. http://drrc.lbl.gov/sites/
drrc.lbl.gov/files/LBNL-5319E.pdf (06.04.2013).
Givan, B., Parr, R., 2013. An Introduction to Markov Decision Processes. http://www.
cs.rice.edu/wvardi/dag01/givan1.pdf (06.04.2013).
Goldman, C., Reid, M., Levy, R., Silverstein, A., 2010. National Action Plan for Energy
Efficiency, Coordination of Energy Efficiency and Demand Response. United
States Environmental Protection Agency, Washington D.C., U.S. http://www.epa.
gov/cleanenergy/documents/suca/ee_and_dr.pdf (06.04.2013).
Greening, L.A., 2010. Demand response resources: who is responsible for imple-
mentation in a deregulated market? Energy 35 (4), 1518e1525.
Gutowski, T., Dahmus, J., Thiriez, A., 2006. Electrical energy requirements for
manufacturing processes. In: Proceedings of 13th CIRP International Conference
of Life Cycle Engineering, May 31-June 2, 2006. Leuven, Belgium.
Heikkurinen, P., Bonnedahl, K.J., 2013. Corporate responsibility for sustainable
development: a review and conceptual comparison of market-and stakeholder-
oriented strategies. J. Clean. Prod. 43, 191e198.
Henze, G.P., Felsmann, C., Knabe, G., 2004. Evaluation of optimal control for active
and passive building thermal storage. Int. J. Therm. Sci. 43, 173e183.
Hu, S., Liu, F., He, Y., Hu, T., 2012. An on-line approach for energy efficiency moni-
toring of machine tools. J. Clean. Prod. 27, 133e140.
 Z. Sun, L. Li / Journal of Cleaner Production 65 (2014) 184e193
Karami, A.B., Jeanpierre, L., Mouaddib, A.I., 2009. Partially observable Markov De-
cision Process for managing robot collaboration with human. In: Proceedings of
21st International Conference on Tools with Artificial Intelligence, November
2e4, 2009, pp. 518e521. Newark, NJ, U.S.
Konstantinos, V.K., Sascha, E.E., 2003. Markov Decision Processes with delays and
asynchronous cost collection. IEEE T. Automat. Contr. 48 (4), 568e574.
Laroche, P., Charpillet, F., Schott, R., 1999. Mobile robotics planning using abstract
Markov decision processes. In: Proceedings of 11th IEEE International Confer-
ence on Tools with Artificial Intelligence, November 9e11, 1999, pp. 299e306.
Chicago, IL.
Lave, R.E., 1966. A Markov Decision Process for economic quality control. IEEE T.
Syst. Sci. Cyb. 2 (1), 45e54.
Lewis, G., 2007. Strategies to Increase California Food Processing Industry Demand
Response Participation: a Scoping Study. Lawrence Berkeley National Labora-
tory, Berkeley, CA, U.S.. http://drrc.lbl.gov/system/files/lbnl-63668.pdf (06.04.
2013).
Lewis, G., Rhyne, I., Atkinson, B., 2009. California Food Processing Industry,
Wastewater Demonstration Project: Phase I Final Report. Lawrence Berkeley
National Laboratory, Berkeley, CA, U.S. http://drrc.lbl.gov/sites/drrc.lbl.gov/files/
lbnl-2585e.pdf (06.04.2013).
Li, J., Meerkov, S.M., 2008. Production Systems Engineering. Springer, New York, U.S.
Li, L., Chang, Q., Ni, J., 2009. Data driven bottleneck detection of manufacturing
systems. Int. J. Prod. Res. 47 (18), 5019e5036.
Li, L., Sun, Z., 2013. Dynamic energy control for energy efficiency improvement of
sustainable manufacturing systems using Markov Decision Process. IEEE T. Syst.
Man Cy. A. http://dx.doi.org/10.1109/TSMC.2013.2256856 (in press).
Li, L., Sun, Z., Yang, H., Gu, F., 2012a. Simulation-based energy efficiency improve-
ment for sustainable manufacturing systems. In: Proceedings of the 2012 ASME
International Manufacturing Science and Engineering Conference (MSEC), Pa-
per Number MESC 2012-7242, June 4e8, 2012, pp. 1033e1039. Notre Dame, IN,
U.S.
Li, L., Yan, J., Xing, Z., 2013. Energy requirements evaluation of milling machines
based on thermal equilibrium and empirical modeling. J. Clean. Prod. 52,
113e121.
Li, W., Kara, S., 2010. An empirical model for predicting energy consumption of
manufacturing processes: a case of turning process. Proc. IMechE, Part B: J. Eng.
Manufacture 225, 1636e1646.
Li, W., Zein, A., Kara, S., Hermann, C., 2011. An investigation into fixed energy
consumption of machine tools, glocalized solutions for sustainability in
Manufacturing. Technische University Braunschweig, Braunschweig, Germany.
Li, Y., Ng, B.L., Trayer, M., Liu, J., 2012b. Automated residential demand response:
algorithmic implications of pricing models. IEEE T. Smart Grid 3 (4), 1712e1721.
Liang, Y., Levine, D., Shen, Z., 2012. Thermostats for the smart grid: models,
benchmarks, and insights. Energy J. 33 (4), 61e96.
McKane, A., Rhyne, I., Lekov, A., Thompson, L., Piette, M.A., 2008. Automating De-
mand Response: the Missing Link in the Electricity Value Chain. Lawrence
Berkeley National Laboratory, Berkeley, CA, U.S. http://drrc.lbl.gov/sites/drrc.lbl.
gov/files/lbnl-2736e.pdf (06.04.2013).
Meeker, W.Q., Escobar, L.A., 1998. Statistical Methods for Reliability Data. John Wiley
& Sons, New York, U.S.
Mirman, L.J., Morand, O.F., Reffett, K.L., 2008. A qualitative approach to Markovian
equilibrium in infinite horizon economies with capital. J. Econ. Theory 139 (1),
75e98.
Motegi, N., Piette, M.A., Watson, D.S., Kiliccote, S., Xu, P., 2007. Introduction to
Commercial Building Control Strategies and Techniques for Demand Response,
193
Environmental Energy Technologies Division. Lawrence Berkeley National
Laboratory, Berkeley, CA, U.S.. http://gaia.lbl.gov/btech/papers/59975.pdf
(06.04.2013).
Mouzon, G., Yildirim, M.B., 2008. A framework to minimize total energy con-
sumption and total tardiness on a single machine. Int. J. Sust. E. 1 (2), 105e116.
New York ISO, 2010. Emergency Demand Response Program Manual. NY, U.S http://
www.nyiso.com/public/webdocs/products/demand_response/emergency_
demand_response/edrp_mnl.pdf (06.04.2013).
Powell, W.B., 2011. Approximate Dynamic Programming: Solving the Curses of
Dimensionality, second ed. John Wiley & Sons, Hoboken, U.S.
ProModel, Inc. http://www.promodel.com/(06.04.2013).
Schipper, M., 2006. Energy-related Carbon Dioxide Emissions in U.S. Manufacturing.
U.S. Energy Information Administration, Washington D.C., U.S. DOE/EIA-
0573(2005) http://www.eia.doe.gov/oiaf/1605/ggrpt/pdf/industry_mecs.pdf
(07.04.2013).
Shafiullah, G.M., Amanullah, M.T.O., Shawkat Ali, A.B.M., Wolfs, P., 2013. Smart grid
for a sustainable future. Smart Grid Renew. Energy 4, 23e34.
Siddiqui, O., 2008. The Green Grid e Energy Savings and Carbon Emissions Re-
ductions Enabled by a Smart Grid. Electric Power Research Institute, Palo Alto,
CA, U.S. http://www.smartgridnews.com/artman/uploads/1/SGNR_2009_EPRI_
Green_Grid_June_2008.pdf (06.04.2013).
Sun, Z., Biller, S., Gu, F., Li, L., 2011. Energy consumption reduction for sustainable
manufacturing systems considering machines with multiple-power states. In:
Proceedings of the 2011 ASME International Manufacturing Science and Engi-
neering Conference (MSEC), Paper Number MSEC 2011-50069, June 13-17, 2011,
pp. 99e103. Corvallis, OR, U.S.
Sun, Z., Li, L., 2013. Opportunity estimation for real time energy control of sus-
tainable manufacturing systems. IEEE T. Autom. Sci. Eng. 10 (1), 38e44.
Trianni, A., Cagno, E., Thollander, P., Backlund, S., 2013. Barriers to industrial energy
efficiency in foundries: a European comparison. J. Clean. Prod. 40, 161e176.
Vassileva, I., Wallin, F., Dahlquist, E., 2012. Understanding energy consumption
behavior for future demand response strategy development. Energy 46,
94e100.
Walsh, C., Thornley, P., 2012. Barriers to improving energy efficiency within the
process industries with a focus on low grade heat utilization. J. Clean. Prod. 23,
138e146.
Wang, L., Wang, Z., Yang, R., 2012. Intelligent multiagent control system for energy
and comfort management in smart and sustainable buildings. IEEE T. Smart
Grid 3 (2), 605e617.
Unpublished results Wang, Y., Li, L., 2013. Time-of-Use Based Production Scheduling
towards Energy-efficient, Demand-responsive, and Cost-effective Manufacturing.
submitted to Energy, revision.
Wang, Y., Pordanjani, I.R., Xu, W., 2011. An event-driven demand response scheme
for power system security enhancement. IEEE T. Smart Grid 2 (1), 23e29.
Ye, Y., 2011. The simplex and policy-iteration methods are strongly polynomial for
the Markov decision problem with fixed discount rate. Math. Oper. Res. 36 (4),
593e603.
Yi, P., Dong, X., Iwayemi, A., Zhou, C., Li, S., 2013. Real-time opportunistic scheduling
for residential demand response. IEEE T. Smart Grid 4 (1), 227e234.
Yousefi, S., Moghaddam, M.P., Majd, V.J., 2011. Optimal real time pricing in an agent-
based retail market using a comprehensive demand response model. Energy 36
(9), 5716e5727.
Yu, R., Yang, W., Rahardja, S., 2012. A statistical demand-price model with its
application in optimal real-time price. IEEE T. Smart Grid 3 (4), 1734e1742.