Operator Training Simulator (OTS) – Grid Reliability

Sreenivas Badri
Manager, EMMS Production Support

09/25/2019
 Agenda

• ERCOT Region, Peak Demand Records

• OTS Environments, Architecture and Data flows
• OTS Case Development Process overview
• New Training Facility
• Operator Training Cycles
• Integrated OTS system capabilities
• Control Room Operators Training – Summer Readiness

PUBLIC 2
 The ERCOT Region

The interconnected
electrical system serving
most of Texas, with
limited external
connections
• 90% of Texas electric
load; 75% of Texas land
• 74,531 MW peak,
August 12, 2019
• More than 46,500 miles
of transmission lines
• 650+ generation units
(excluding PUNs) 220 MW with SPP

600 MW with SPP

ERCOT connections to other grids Texas RE

are limited to ~1,250 MW of direct
current (DC) ties, which allow 30 MW with CENACE
control over flow of electricity at Eagle Pass
100 MW with CENACE
at Laredo 300 MW with CENACE at McAllen

PUBLIC 3
 Current Records
Peak Demand Record: 74,531 megawatts (MW)*
 August 12, 2019, 4-5 p.m.
Weekend Peak Demand Record: 71,864 MW*
 Sunday, August 11, 2019, 5-6 p.m.
Winter Peak Demand Record: 65,915 MW
 Jan. 17, 2018, 7-8 a.m.
Wind Generation Records (instantaneous)
 Output: 19,672 MW
‒ Jan. 21, 2019, 7:19 p.m.
 Penetration (load served): 56.16%
‒ January 19, 2019, 3:10 a.m.
‒ Total MW Served by Wind = 17,406 MW
PUBLIC
Recent Monthly Peak Demand Records
2019
• March: 60,692 MW (March 5, 7-8 a.m.)
• August: 74,531 MW (August 12, 4-5 p.m.)*
2018
• January: 65,915 MW (Jan. 17, 7-8 a.m.)
• May: 67,265 MW (May 29, 4-5 p.m.)
• June: 69,123 MW (June 27, 4-5 p.m.)
• July: 73,473 MW (July 19, 4-5 p.m.)
• November: 56,317 MW (Nov. 14, 7-8 a.m.)
*New records are preliminary, subject to change in final settlement
4
 Operator Training Simulator (OTS) – Environments

• 6 OTS Environments, each environment contains multiple systems

• 1 – Test, 2- Bastrop Production and 3- Taylor Production environments

PI OT01-Test OT02 OT03 OT04 OT05 OT06

EMS EMS EMS EMS EMS EMS

PI Buffer & Archive

Webfg/ETA Webfg/ETA Webfg/ETA WebFG/ETA Webfg/ETA Webfg/ETA

PI Buffer & Archive

DSA/VSAT/TSAT DSA/VSAT/TSAT
PI Buffer & Archive DSA/VSAT/TSAT DSA/VSAT/TSAT DSA/VSAT/TSAT DSA/VSAT/TSAT

MMS MMS MMS MMS MMS MMS

MA App MA App MA App MA App MA App MA App
PI AF/Coresight

MMS MMS MMS MMS MMS MMS

PI AF/Coresight MOI Web MOI Web MOI Web MOI Web MOI Web MOI Web
MMS LDAP

PI AF/Coresight MMS MMS MMS LDAP Servers

MI/OS MMS MMS MMS
MI/OS MI/OS MI/OS MI/OS MI/OS

MMAP MMAP MMAP MMAP MMAP MMAP

EMS & MKT Oracle EMS& MKT Oracle

EMS & MKT Oracle EMS & MKT Oracle EMS & MKT Oracle EMS & MKT Oracle
Databases Database
Databases Databases Databases Databases

PUBLIC 5
 Operator Training Simulator (OTS) – Environments
• Test OTS environment is used for OTS software changes testing before deploying
into Production OTS environments
• Why we need 5 Production OTS environments ?

o To have flexibility to run 2-3 simulations in parallel in a training cycles

o To troubleshoot any existing training cycle simulator issues
o To develop and prepare next training cycle simulation
o To train control room operators on latest software release changes before
the changes hits production

• All OTS Production environments (except being used for current Training Cycles)
OTS systems & applications software release versions are in sync with actual
production versions.

• OTS environments are treated as critical with production level support and
maintenance considering the significance of Operator Training and back to back
training cycles is being planned throughout the year.

PUBLIC 6
 OTS Systems Architecture
MMAP Windows
Sever OTS EMS System - Linux Server

DSA/VSAT -
Windows Server
EMS & OTS Data
Applications Interface
PI Buffer & Archive –
Windows Server

PI AF/Coresight – Windows Server

Windows Server WebFG E-terraArchiver
Display
Server Application

Web Based PI Ops

Tools
Database (AIX

Database (AIX
MMS Oracle
EMS Oracle

Server)

Server)
OTS MMS System - AIX

Market Operator MMS Application Server

Interface (MOI) – Display Tibco (SCED, HRUC, ASM, VDI
Server etc.)

MMS/OS MI
Server(Market/Outage
Scheduler Events)

PUBLIC 7
 OTS Applications Data flows
Market Management System
HRUC/AS Manager/ MOI Displays – COP/
SASM DCTIE User Interface
EMS - Network Applications
TCM SCED
COP/DCTIE/RUC/ERS/AS
Deployments/Outages
EMS - Generation Applications (MMS Database)

STNET LF

RDB
DSA/VSAT/TSAT Interface

RTCA RLC

RTNET
LFC iGridInterface
QKNET (Reliability Risk
Desk Operations)
Wind & Solar
SCADA Forecast
(EMS Database)

OTS - Simulation Engine

Outage Scheduler
(MMS Database) DTSPSM QSE Simulator Data Interface
EMS/OTS
HDR OTS Events
RTOP
MOI Displays – Event
Outages creation/ Scenarios DTSPSM also mimics TO Role & TO Telemetry
Update- User Interface QSE Simulator mimics QSE Role & QSE Telemetry

PUBLIC 8
 OTS Case Development Process Overview
• OTS case development process involves multiple teams, most of the tasks are being
performed by EMMS Production and Operator Training Teams

• End to End Case development, all OTS systems setup takes around 2 weeks
depending on number of simulations are planned in a given training cycle.

• Training Team provides following training cycle requirements and determines new OTS
case is required.

 Production Snapshot Date, Peak Demand

 Systems and Applications, Tools to be functional
 Market Simulation requirements

• EMMS Production develops OTS case, performs following testing on the validated
case to ensure simulator is working as expected before adding OTS events.
 OTS (DTSPSM, QSE Simulator etc.) applications testing
 EMS & MMS application testing and MMS to EMS integration testing
 Basic scenario testing (Frequency Recovery, Large Unit Trips, 345KV equipment outages, RRS/Non-
Spin Deployments etc.)

• Tested case is provided to Training Team to develop OTS events and perform case
tuning to simulate required events during operator training cycles.

PUBLIC 9
 OTS Case Development Process Overview
High level steps in OTS Case Development
• EMS save cases, SCADA HDR files preparation and setup based on Simulation Date and time.

• Run OTS Save Case Preparation Application to do following

o EMS Save Cases load
o Validate all Model Clones
o Load validated save cases into Real Time Clones

• Case updates and tuning – run scripts to populate OTS specific data and set OTS specific parameters

• Bring up OTS system and Verify OTS applications specifically QSE Simulator and DTSPSM

• Ensure that DSA/VSAT runs and transfer Stability Limits to EMS

• Perform Market Database Refresh based on Simulation date and time.

• Perform CIM Model Load in Market System based on Simulation date

• Setup EMS/MMS Interfaces and ensure EMS to MMS dataflow is functional

• Enable OTS Market Integration, ensure Market Management System (MMS) applications SCED/HRUC/ASM/VDI
are working as expected based on basic scenario testing.

• Perform OTS Integration Testing

• Prepare PI data and ensure PI Archive, PI AF/CoreSite, PI Ops Tools, Control Room Spreadsheets are functional
based on Simulation time.

• Ensure Macomber Map is setup and integrated with EMS.

PUBLIC 10
 OTS Case Development Lifecycle
Simulation Case Development Lifecycle
Overall development timeline depends on accurate estimate of required work in each department
Operations
System

Training/ Ops
Simulation
Performanc e procedures/
Start indic ations Yes No
simulation tool is sues
acc eptable?
requested identified?

Yes
System Operations Training

Complete
Sim dev Submit to
Simulation Case Simulation Event Conduct training Create crew
process Create s imul ati on Operations
Preparation performanc e Yes performanc e Yes No and s upport performanc e
effi ciencies training s cript Improvement for
Checkli st and acc eptable? acc eptable? training feedbac k report
identified? analysis
submit to E MMS

Yes
No

Update Simulation
Evaluate case Cas e Preparation End
Checkli st
No

No
EMMS Production Support

Mar ket data

No Develop c ase Tes t cas e Fixes requir ed? No
refresh req’d?

Yes

RFC for code

Create RFC for
No changes Yes
Dev
required?
Yes
Database Hosting
Services

Perform Market
data refresh
Development
EMMS

Create c ode or
tuning changes
and deli ver to
EMMS PS

Highlighted boxes represent individual department-specific documented processes/procedures/controls which require

department commitments
Red arrows represent the optimal path

PUBLIC 11
 Taylor Training Facility

• ERCOT’s new training center

officially opened in January.

• Black Start and Operator

Training Seminars will
take place in the new
building in 2019.

PUBLIC 12
 Operator Training Facility

• Operator Training Facility Control Room is replica of actual Control

Room

• Operators from every desk are trained in OTS environment on all the
EMS/VSAT/MMS/PI/MMAP etc. applications and situational awareness
tools they use in actual control room.

PUBLIC 13
 Operator Training Cycles
• 6 Operator Training Cycles every year

 Black Start Training with Market Participants every year - Cycle 2 (March – April) –
800 Operators (including ERCOT Operators) attending this year Black Start
training this year.
 Summer Readiness Training Cycles – Cycle 3 and Cycle 4 (May to August)
 It also includes SOLs & IROLs, EEA and Loss of Primary Control Center related
training simulations
 Operator Training Seminar – Cycle 5 (OTS is not used)
 Emergency Weather Drills (Alternate between Hurricane and Winter Storm) –
Cycle 6 (October – November)
 Grid Ex – Cycle 6

• Operators are trained in OTS environment on EMS/MMS/PI Ops Tools, Situational

Awareness tools changes that go into production that impacts control room
operations, it includes defects, enhancements and functionality upgrade &
technology upgrade projects.

• Each Training Cycle duration is 6 weeks to train 6 operator crews

PUBLIC 14
 Integrated OTS System Capabilities
• Integrated OTS system has capability to simulate various type
of simulations
 Simulate Grid Events or EMMS Applications failure scenarios (ICCP/SCADA Telemetry failures,
State Estimator, CA, LFC/RLC/SCED, VSAT etc. failures) and Applications functionality
changes (Nodal Protocol changes, System changes, Enhancements etc.)

 Replay Grid and Real-Time Market Events

 Standalone mode without Market Integration

 Black Start Training with Market Participants

 Market Participants Training (Winter Storm Drill, Hurricane Drill)

 Simulation based on Future Models (60, 90 days ahead CIM Models)

• Integrated OTS system has capability to run simulation for 48 hours or

more in integration with all other systems including Market System.

PUBLIC 15
 Operator Training – Summer Readiness
• Developed OTS case based on 2019 March Production EMS Snapshot and Market
data, overlaid summer peak load pattern.

• OTS Market System data (AS Qualification, COP, Offers) was prepared to setup Market
applications to run SCED/HRUC, deploy Offline Non-Spin, Load Resources RRS, ERS
and dispatch switchable generators during EEA event.

• Training team added events and tuned OTS case to simulate following events to train
Operators for summer events
 Energy Emergency Alert (EEA)
 Switchable Generators
 Real-Time Assessment (RTA)
 IROLs and SOLs Exceedances

• All required OTS systems and application were setup to run simulator and simulate
above events during Cycle 3 training cycle (May – June).

PUBLIC 16
 Operator Training – EEA
• EEA Trigger Conditions
 EEA1 - must be declared when PRC < 2300 MW & it is not projected to be recovered above
2300 MW within 30 minutes
 EEA 2 - must be declared when PRC < 1750 MW & & it is not projected to be recovered above
1750 within 30 minutes or may be declared EEA 2 if System frequency below 59.91 for 15
minutes
 EEA3 - must be declared when PRC <= 1375 or must go to EEA3 if system frequency is below
59.91 for 25 minutes or may declare EEA3 if system frequency is below 59.91 for 20 minutes

• Energy Emergency Alert (EEA) Days

 EEA days this year – 08/13/2019 & 08/15/2019
 Last EEA Days – 01/06/2014 & 01/18/2014
• Constant EEA training is required for operators in Operator Training
Simulation environment because
 EEA is rare event
 EEA situational awareness tools, EMS/MMS applications, processes
and procedures changes constantly
 Every operator crew requires to go through real-time EEA events
simulation
o ERS, Offline Non-Spin and Load Resources RRS deployment related MMS
Applications are used mainly during EEA events other than qualification
testing.

PUBLIC 17
 Operator Training – EEA

• Operators followed the EEA procedure to maintain Grid reliability during EEA and
brought back the Grid to normal conditions. Following are some of the key actions
performed by Operators during EEA event using OTS EMS and MMS applications

 Manual RRS Deployment through LFC

 Offline Non-Spin Deployment
 ERS Deployments
 Load Resource RRS deployment
 RUC Commitments through VDI
 Firm Load shed through OTS Events

• All Operator crews were trained on EEA event in OTS environment by end of June that
includes all the situational awareness tools changes and EEA event
processes/procedures.

• Operators were also trained on Market System Emergency Response Service (ERS)
and Non-Spin Display enhancements during EEA event training in OTS environment.

• Control Room Shift Supervisors and Operators were extremely happy on how OTS
training on EEA helped them to handle this summer EEA events effectively.

PUBLIC 18
 Operator Training – EEA Dashboard

PUBLIC 19
 Operator Training – Switchable Generators
• Switchable Generators (SWGRs) changes were brought into Production EMS and MMS
systems end of May this year in preparation to handle peak demand this summer.

• A new telemetered resource status code EMRSWGR is made available to represent

Emergency Switchable Generation resources.

• When Switchable Generation qualified resources are switched from other region and
ready to operate in ERCOT region, QSE changes resource status telemetry from
EMRSWGR to ON, SCED dispatches the units.

• OTS Market System data was prepared to simulate Switchable Generators (SWGRs)
following behavior in OTS environment the way it happens in real-time

 Showing SWGRs current and future operating status, capacity in Market System
new Displays
 SWGRs Resource Status Telemetry changes in EMS
 SCED dispatching SWGR units

• Operators were trained on Switchable Generators operations in OTS environment before

changes were deployed in Production before start of this summer.

• Operators were able perform Switchable Generators Operations effectively during this
summer EEA days.

PUBLIC 20
 Operator Training - RTA Failure Scenarios
• RTA (Real-Time Assessment) improvements that were put in place in EMS after past
incidents helped significantly ERCOT RTA process running through any TSP telemetry
failures.
• Rigorous Shift Engineer and Operator Training was provided during this summer in OTS
environment by simulating RTA failure scenarios that helped them significantly in
understanding the RTA details and performing manual RTA within 30 minutes.

RTA failure scenarios

Loss of an
EMS Loss of Real Loss of RT &
Partial Loss Complete
Application Time (RT) Study (ST)
of ICCP Loss of ICCP
(RTNET/ EMS System EMS systems
RTCA/RTDCP)

Perform RTA
Perform Perform using
Use SCADA/ Manual RTA Manual RTA Use Study EMS Powerworld /
RTNET Using Using system with Offline VSAT
enhancements Enhanced Enhanced last successful using last
to help solve STNET Online STNET Online RTA solution successful RTA
RTNET Powerflow in Powerflow in available solution
RT EMS RT EMS exported in
the applicable
format

PUBLIC 21
 Questions?

PUBLIC 22
 Thank You

PUBLIC 23