Professional Documents
Culture Documents
GEH-6840.pdf Network PDF
GEH-6840.pdf Network PDF
• Small
• Small extended
• Large
• Large extended
The system designs are meant to provide comprehensive, overall system design guidance. They are the basic guidelines for
selecting the components, computers, network switches, their interconnections, and controller connections for building the
System Topology Diagram. System definitions and guidelines for fulfilling the system design are provided.
Engineering Workstations
Historian
- WorkstationST EGD/OPC Server
- WorkstationST Alarm Server
- WorkstationST Alarm Viewer
- CIMPLICITY Edit/View
- ToolboxST Application
- EGD Configuration Server
- SDB Configuration Server
- System Configuration Files
- Control System Toolbox
Edge Switches
HMI
LS2100e
8 Port
TMR EX2100e TMR Mark VIeS TMR Mark VIe
Switch
Small System
• Gas turbines
• Steam turbines
• Turbo compressors
• Combined cycle systems
• Other distributed control systems (DCSs)
This document does not provide:
The NetworkST 3.1 topology design provides reliable communications between control system devices; controllers, HMIs,
Historian, OSM, Remote Services Gateway (RSG), relays, vibration and predictive monitoring equipment, and asset
monitoring system. The system supports TCP and UDP IP protocols.
NetworkST 4.0 topology design extends the NetworkST 3.1 topology by adding routing capability and a firewall that can be
used to separate functions to multiple VLANs and a DMZ. This capability can be used to enhance security by separating
devices into different VLANs based on their function.
Switches communicate over trunk lines that carry Virtual Local Area Networks (VLANs), which define the segmentation of
specific functions on the network:
Note The network switches are preconfigured with ports for UDH, PDH, MDH, and Trunk lines.
The controllers referenced in this document do not use the above networks to communicate with their associated I/O. Network
traffic between a controller and I/O is done on IONet using unmanaged switches.
Note Customers should not connect additional equipment to the PDH. Additional equipment that needs to be added to the
system should be connected to the DMZ provided by the NetworkST 4.0 solution. If communication between these devices
and the PDH is required, network engineering will be required to define router and firewall rules to enable the
communication.
The NetworkST 3.1 topology design improves network redundancy by providing rapid spanning tree capabilities. This allows
redundant network paths but does not allow data to loop, where packets are endlessly forwarded creating a data storm that
blocks other traffic on the network segment.
Note For further details, refer to the WorkstationST Network Monitor Instruction Guide (GEI-100693).
Note For more information, refer to the ControlST How-to Guides (GEH-6808), the section How to Configure Time
Synchronization in the ToolboxST Application.
The system can support two NTP Time sources. The primary time source can be an NTP Server with IRIG-B, GPS inputs if
high resolution time is needed. An EWS or HMI can be the primary time source if low resolution time is sufficient. One of
these should also be configured as the backup time source for the system.
Note Switch pairs are used to provide redundant links to devices on the network.
The Ethernet interfaces with RJ-45 connectors can be configured as 10BASE-T and 100BASE-TX. On some switches the
RJ-45 Ethernet interfaces can be configured as 1000BASE-T ports. (Review the individual part definition for more details.)
The supported Ethernet cabling depends on the interface configuration: 10BASE-T ports supports 2-pair Category 3, 4 and 5
unshielded twisted-pair (UTP) cable, 100BASE-TX ports support 2-pair Category 5 UTP cable, 1000BASE-T ports support
4-pair Category 5 UTP cabling. Fiber-optic cables plug into the appropriate ports on the front panel using small form-factor
pluggable (SFPs) transceivers. Data rate through the fiber-optic ports is 100 or 1000 Mbps, depending on the specific switch
configuration and choice of SFP. Switches are configured by GE; pre-configured switches should be purchased from GE.
Fiber-optic cable provides the best signal quality, completely free of electromagnetic interference (EMI) and radio frequency
interference (RFI). Large point-to-point distances are possible, and since the cable does not carry electrical charges, ground
potential problems are eliminated. Fiber-optic cable is to be used anytime the cable run leaves a building to go to another
building. It is also to be used between separate ground grids.
The NetworkST 3.1 design provides single switches for controls retrofit applications.
1 11 13 23 1 11 13 23
2 12 14 24 2 12 14 24
Unstacked 2960X Edge Switch Connections
The unstacked edge switch configuration has been tested to validate that in the presence of various failure scenarios (failed
trunk connection, single edge switch failure, single root bridge failure) the unaffected switch in the pair will provide
redundant communication in less than one second.
In cases, where redundant control equipment is connected to the switch pair, the failover time can be much faster than one
second. It is recommended that the application developer understand the communication timing requirements for their system
validate that the designed network meets those requirements.
Stacked Edge Switches
The Cisco Catalyst 2960X can be deployed in a stacked edge switch configuration.
While the stacking capability does provide benefits, the failover timing that can occur when one of the stack members fails
may be unacceptable for some applications. Testing of the 2960X as an edge switch has shown that when one switch fails
there is a potential for up to a three second delay for the other switch to take over forwarding traffic. This can cause
communication from the equipment connected to the non-failed switch to become unavailable for up to three seconds.
Depending on the application, this failover timing may or may not be acceptable. The application developer must understand
the communication requirements for the system and decide if the 3 second failover time is acceptable.
If the three second failover timing is unacceptable, the edge switches can be deployed in unstacked pairs.
1 11 13 23 1 11 13 23
2 12 14 24 2 12 14 24
Stacked 2960X Edge Switch Connections
SFP
LC Connector
Engineering Workstations
Historian
- WorkstationST EGD/OPC Server
- WorkstationST Alarm Server
- WorkstationST Alarm Viewer
- CIMPLICITY Edit/View
- ToolboxST Application
- EGD Configuration Server
- SDB Configuration Server
- System Configuration Files
- Control System Toolbox
Edge Switches
HMI
LS2100e
8 Port
TMR EX2100e TMR Mark VIeS TMR Mark VIe
Switch
PDH Primary
UDH Primary
1 11 13 23 1 11 13 23
2 12 14 24 2 12 14 24
UDH Backup
PDH Backup
1 11 13 23 1 11 13 23
2 12 14 24 2 12 14 24
trunk
LS2100e
8 Port
Switch
• Number of controllers
− Large numbers of controllers in dispersed locations increase network design complexity
• Communications requirements
− Redundant communications are required for new units
− Redundant communications are available for retrofit jobs
− Simplex is in the controls retrofit market
• Power requirements
− Switches need reliable power
− Reliable power can be provided by a site uninterruptible power supply (UPS)
− Two power sources (one for each switch) is preferred
− Consider total power blackout in the PEECC for maintenance impact
• Distances between controllers
− Use a single switch pair for controllers clustered together
− Consider additional switch pairs connected by a fiber-optic trunk to the root switch for controllers that are widely
separated
− Use fiber-optic cable when copper distances are exceeded
• Outdoor cable runs
− Must be fiber-optic cable
− NetworkST 3.1 is standardized on single mode FO (10 km maximun)
− Use diverse cable routes
• Control room considerations
− Incorporate switches adequate for the numbers of computers in the design
• Numbers of controller sets supported
− Limited to the number of fiber-optic ports plus the number of copper trunk ports on the switches used in the design
• Dual redundant connections used between the computer and the switch pairs
• Switch pairs to allow for redundant connections to HMIs and control equipment
• Switches that use SFPs to allow different types of media to be used between switches
• IEEE 802.1w Spanning Tree Protocol to manage packet forwarding and avoid packet retransmission among redundant
paths around the field switch circular path
An Engineering Workstation (EWS) must be included in this system design. The EWS configures the HMI, Mark VIe, Mark
VIeS, EX2100e, and LS2100e, and stores the system configuration files. The EWS also serves as a HMI.
Additional HMIs can be included based on system requirements. Install HMIs where users need access to both operator
functions and system configuration tools. This system supports an optional Historian. The network supports up to two OSMs
at each controller and control room network switch pair.
Note For further details, refer to the Human-machine Interface (HMI) Product Line User Guide (GEH-6751).
The root switch in this configuration is a stacked pair of 2960X switches. The edge switches in this architecture can be
unstacked IE2000 switch pairs, unstacked 2960X switch pairs, or stacked 2960X switches.
The NetworkST 3.1 design provides non-stacked switches for controls retrofit applications. Devices connect with 100TX
connecting to RJ-45 ports with unshielded twisted pair (UTP) cabling. Fiber-optic cables plug into the appropriate ports on
the front panel using SFP transceivers.
Small System
X = required
O = optional
Edge Switches
HMI
LS2100e
8 Port
TMR EX2100e TMR Mark VIeS TMR Mark VIe
Switch
• Dual redundant connections used between the computer and the switch pairs
• Switch pairs to allow for redundant connections to HMIs and control equipment
• Switches that use SFPs to allow different types of media to be used between switches
• IEEE 802.1w Spanning Tree Protocol to manage packet forwarding and avoid packet retransmission among redundant
paths around the field switch circular path
An Engineering Workstation (EWS) must be included in this system design. The EWS configures the HMI, Mark VIe, Mark
VIeS, EX2100e, and LS2100e, and stores the system configuration files. The EWS also serves as a HMI.
Additional HMIs can be included based on system requirements. Install HMIs where users need access to both operator
functions and system configuration tools. This system supports an optional Historian. The network supports up to two OSMs
at each controller and control room network switch pair.
Note For further details, refer to the Human-machine Interface (HMI) Product Line User Guide (GEH-6751).
The root switch in this configuration is a 4 stack of 2960X switches. The edge switches in this architecture can be unstacked
IE2000 switch pairs, unstacked 2960X switch pairs, or stacked 2960X switches.
The NetworkST 3.1 design provides single switches for controls retrofit applications. Fiber-optic cables plug into the
appropriate ports on the front panel using SFP transceivers.
Any system with a combination of 10 or more HMIs and EWSs needs to be supported by a System Configuration Server and
a pair of redundant Alarm Servers. The System Configuration Server holds the master configuration files, operator screens,
CMS Server, EGD Configuration Server, SDB Server, and backup NTP source. The HMIs and EWSs are configured to use
the CMS to access the master configuration files stored on the System Configuration Server. The HMIs and EWSs are also
configured to collect alarm data from the Alarm Servers
Core Switch
- 24 ports
HMI
LS2100e
Turbine
TMR EX2100e TMR Mark VIe Control
8 Port
Switch
LS2100e
Turbine
TMR EX2100e TMR Mark VIe Control
8 Port
Switch
Edge
Switches
LS2100e
Turbine
TMR EX2100e TMR Mark VIeS TMR Mark VIe Control
8 Port
Switch
LS2100e
Turbine
TMR EX2100e TMR Mark VIeS TMR Mark VIe Control
8 Port
Switch
Stacking Cable Fiber Cable Cat 5e Cable
Small Extended System - Up to 4 Units
• A pair of redundant UDH and PDH connections from the HMI to the switches.
• Dual redundant connections used between the computer and the switch pairs.
• Switch pairs to allow for redundant connections to HMIs and control equipment.
• Switches that use SFPs to allow several different types of media that can be used to interconnect the switches.
• IEEE 802.1w Spanning Tree Protocol to manage forwarding and avoid packet retransmission among redundant paths
around the field switch circular path.
The control room network switch can support a combination of up to eight EWSs, HMIs, and Historians. A second set of
eight ports for EWSs, HMIs, and Historians use can be added either by adding a second switch pair to the control room
network switches, or by creating a second control room ring. Other HMIs are used to supplement the core set previously
described. Apply HMIs where users need access to both operator functions and system configuration tools, with at least one
HMI installed in the central control room.
An EWS must be included in this system design. The EWS configures the system, HMI, Mark VIe, Mark VIeS, EX2100e,
and LS2100e, and hosts the system configuration files, EGD Configuration Server, and SDB Server.
The root switch in this configuration is a stacked pair of 3850 or 3750X switches. The edge switches in this architecture can
be unstacked IE2000 switch pairs, unstacked 2960X switch pairs, or stacked 2960X switches.
Any system with a combination of 10 or more HMIs and EWSs needs to be supported by a System Configuration Server and
a pair of redundant Alarm Servers. The System Configuration Server holds the master configuration files, operator screens,
CMS Server, EGD Configuration Server, SDB Server, and backup NTP source. The HMIs and EWSs are configured to use
the CMS to access the master configuration files stored on the System Configuration Server. The HMIs and EWSs are also
configured to collect alarm data from the Alarm Servers.
Note For further details, refer to the Human-machine Interface (HMI) Product Line User Guide (GEH-6751).
Large System
System Required Comments
Engineering Workstation X Full-time
O Use where both operator and configuration capability
Human-machine Interface
are needed
Historian O Typical location - control room
System Configuration Server O Use if > 10 (HMIs + EWS)
Alarm Server Pair O Use if > 10 (HMIs + EWS)
Application Gateway O Special communication
X = required, O = optional
Field Switch
- 24 ports Core Switch
- Layer 2 stackable switch -12 Fiber Only ports
- 2 Combo SFP slots - Layer 3 Stackable Switch
Supports
up to 8 Turbine or
BOP Controller
Sets Total
HMI
LS2100e
Turbine
Control
TMR EX2100e TMR Mark VIe
8 Port Switch
LS2100e
Turbine
Control
TMR EX2100e TMR Mark VIe
8 Port Switch
Edge
Switches
LS2100e
Turbine
Control
TMR EX2100e TMR Mark VIeS TMR Mark VIe
8 Port Switch
• A pair of redundant UDH and PDH connections from the HMI to the switches
• Switch pairs to allow for redundant connections to HMIs and control equipment.
• Switches that use SFPs to allow several different types of media that can be used to interconnect the switches
• IEEE 802.1w Spanning Tree Protocol to manage forwarding and avoid packet retransmission among redundant paths
around the field switch circular path
The rules for HMIs used in the large system apply to extended capability system. The extended capability system is for a
larger plant and is essentially built with multiple blocks, where each block is equivalent to the large system design. There are
generally large numbers of HMIs and multiple EWSs supported by a pair of Alarm Servers and a System Configuration
Server in the system.
One or more EWSs are used to configure the control system. System configuration files are stored on the System
Configuration Server. Install HMIs where users need access to both operator functions and system configuration tools. Use
the Application Gateway for special purpose communications to external systems. Apply a pair of Alarm Servers to minimize
alarm traffic to the Mark VIe/Mark VIeS controllers. This system supports optional Historians. The network supports up to
two OSMs at each controller and control room network switch pair.
Note For further details, refer to the Human-machine Interface (HMI) Product Line User Guide (GEH-6751).
The root switch in this configuration is a stacked pair of 3750X or 3850 switches. The edge switches in this architecture can
be unstacked IE2000 switch pairs, unstacked 2960X switch pairs, or stacked 2960X switches.
VPN
+
""
'
)
EAP Remote Access Zone CSN
EAP
!"
#
ESP ESP ESP ESP
!&!
*!
)
MGH
Router
!!&
*!
SYST S T AT SPE ED R PS
LAN BASE 1G UPLIN K Ca t aly st 2 96 0 -X Se ri es
MA S T S T ACK CO NSO LE
1 11 13 23
2 12 14 24
MGMT CONSO LE 25 26 S FP 27 28
C ON S OL E
SYST XPS STAT SPEED DUPLX BLAN K
MDH
FN S-PWR M AST STACK MOD ULE
M ODE C at alyst 37 50 -X Se rie s
Catalyst 3750- XS eries
1 2 3 4 5 6 7 8 9 10 11 12
$
%
!&!
'!(
Layer
2
Switch
CSMS Zone Process Information Zone
OSM RSG RVC
DC2 DC1 AP1
DCS Maintenance
Backup Historian Workstation
Identity Mgmt Security
Domain (Active Directory Change Mgmt
Controller Radius )
PDH
AP2 AP3 Control Zone
Certificate
SIEM
Authority
HMI
(Hardened )
UDH
!"#$
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Catalyst 2960-S Seri es
STAT
DPLX
SPED
STCK
MODE
25 26 27 28
$#
!"#
CO NSO LE
SYST XPS STAT SPEED DUPLX NETWORK
BLANK
CON SOL E
SYST XPS STAT SPEED DUPLX NE TWORK
BLANK FN S-PWR MAST STACK C 3KX-N M -10G MODULE
MODULE
S-PWR MAST STAC K C3K X-N M-10G M ODULE
MODULE MODE
FN
Catalyst 3750-X Ser ies
MODE Catalyst 3750-X Series Catalyst 3750-X Series
Catalyst 3750-X Series
1 2 3 4 5 6 7 8 9 10 11 12 G1 G2/TE1 G3 G4/TE2
1 2 3 4 5 6 7 8 9 10 11 12 G1 G2/ TE1 G3 G4/TE2
SY ST STAT SP EED RP S
SYST S TAT SPEED RPS
LAN BASE 1G UPLINK Cat alyst 2960-X Series
MAS T STACK
LAN BASE 1G UPLINK Catalyst 2960- X Series CONSO LE
1 11 13 23
1 11 13 23
2 12 14 24
2 12 14 24
MGMT CONS OLE 25 26 SFP 27 28
MGMT CONS OLE 25 26 SFP 27 28
SYST
MAST
STAT
STACK
SPEED RPS
CONSOL E
LAN BASE 1G UPLINK
1 11 13 23
Catalyst 2960-X Series
SY ST
MAS T
STAT
STACK
SP EED RP S
CONSO LE
LAN BASE 1G UPLINK
1 11 13 23
Cat alyst 2960-X Series
2 12 14 24
2 12 14 24
MG MT CONSOL E 25 26 SFP 27 28
MGMT CONS OLE 25 26 SFP 27 28
NetworkST 4.0 Layer Over NetworkST 3.1
• Consume EGD data from controllers in the UDH (Data on EGD is called published Data)
• Consume controller Live Data and Capture Buffer Data into the Data Recorder for Trip Log evaluation
• Receive controller alarms that can be displayed in the Alarm Viewer (Alarm Protocol)
• Upload controller configuration
• Display controller unpublished data (SDI data for variables not on EGD)
4.3 DMZ
The De-militarized Zone (DMZ) is the physical or logical sub-network that exposes the GE ICS external-facing services to
potentially untrusted networks and services. The figure NetworkST 4.0 Connections illustrates the location of the DMZ
network. By design, devices located in the DMZ (RSG/OSM) have very limited access to site equipment. The access is
provided by the UTM/Firewall. The GE UTM/Firewall standard configuration allows limited application data communication
to the M&D center, reducing the likelihood of exposure to malicious exploits. Devices in the DMZ are not allowed to join the
HMI domain. The GE standard UTM/Firewall rules only allow WorkstationST devices located in the DMZ and running
ToolboxST software to consume EGD data from controllers in the UDH (published data).
4 GE DMZ
5 GE WAN
6 Enterprise
7–10 Available
RPS A
S
E
MSTR T
STAT
DPLX
SPED
STCK
2X 12X 14 X 24X
MODE
25 26 27 28
Note The customizing of the GE network design to meet customer’s and third-party communication needs, and the
implementation and deployment of routing rules to achieve secure communication channels should be performed by
networking professionals.
The router and UTM/Firewall running configurations provided as part of the standard NetworkST 4.0 product have all
required policy routing rules for GE RSG and OSM communication from the MDH and the DMZ to the GE ICS network.
These configurations can be used as guidance to identify the required policy routing and firewall rule areas and implement the
necessary changes.
The Cisco routers use the Hot Standby Router Protocol (HSRP) for redundancy. Only one router is actively forwarding traffic
at a given time, the other is in standby mode. Upon failure or disconnection of the active router, the standby router takes over
the routing tasks.
• SSH (Secure Shell) connections are used to manage and maintain the switch. This includes retrieving the configuration
for backup purposes, or altering the existing configuration in the switch. It is also used for advanced diagnostics.
• SNMP (Simple Network Monitoring Protocol) connections are used by the WorkstationST Network Monitor program to
provide a report on the network health. (Control System Health also uses this same interface.)
In addition, the switches also use the management interface for requests from the switch:
• Logging messages are created and sent to the system Syslog server.
• Network time requests may be issued to the site time server.
• RADIUS requests are made to the Domain Controllers to authenticate users.
• SSH clients on the PDH connect to the switch using its PDH address.
• SNMP clients on the PDH make requests to the switch using its PDH address.
• The switch sends its logging messages to the Syslog server on the PDH.
• The switch can request time from the time server on the PDH.
• RADIUS requests are made directly to the Domain Controllers on the PDH.
The main disadvantage of this scheme is that any computer on the PDH has access to the switch management interfaces and
therefore presents a potential risk for impacting the switch operation.
• SSH clients will be allowed from the AP1 server in SecurityST systems (AP1 at 172.16.201.103) and from the primary
Engineering Workstation (EWS1_SVR at 172.16.201.22).
• SNMP clients will be allowed to make requests from the special Network Monitor functional IP address (NetMon1 at
172.16.201.60). (Refer to the section Network Monitor Functional IP Address.)
All other access to the management interface will be blocked.
Outbound functions, such as the switch logging to the Syslog server, will be allowed by the routers, but only to the expected
address (such as Syslog only to AP2 at 172.16.201.104). This prevents devices on the Management VLAN from being able to
attack entities on the PDH.
• Reconfigure the routers with the address of the computer running the Network Monitor.
• Reconfigure every switch with the address of the computer running the Network Monitor.
To accomplish the above the access control lists in the routers and the switches are configured with a Functional IP Address.
This is an IP address that is assigned to a computer that is designated to run a particular function, but it is not the primary IP
address of that computer. Instead, the functional IP address is a second IP address that is added to the computer in addition to
its primary address. By using a second functional IP address it allows the function to be landed on any WorkstationST class
computer without having to change the primary address of that computer or any of the access control lists in the routers or the
switches.
To accomplish this, the following steps are taken on the computer that is designated to run the Network Monitor function:
• A second IP address is added to the PDH network adapter of the selected HMI (NetMon1 at 172.16.201.60).
• The Network Monitor software is configured to use the IP address of the Network Monitor function (NetMon1 at
172.16.201.60).
Note Use of a secondary IP address for the Network Monitor function was first introduced in ControlST V06.00.
Using with the above information, the Network Monitor function knows to issue all ICMP Echo and SNMP requests using the
secondary (NetMon1) IP address as the source address. The access control lists in the routers and the switches have been
configured to pass/accept messages from that source address, so the messages will make it to their intended destination. The
replies will go back to the source address, which routes them back to the computer running the Network Monitor function.
By using a Network Monitor Functional IP address the Network Monitor can access the management network:
• Without having to change the main IP address on the computer running the Network Monitor function. This prevents
having to make any changes to the network drawings or the WorkstationST configurations associated with changing the
IP address of a computer.
• Without having to change the access control lists in the router and every switch.
• A PDH management address can temporarily be assigned to the switch and its configuration can be loaded as it was prior
to the Management VLANs implementation. Once the configuration is loaded the switch will revert to the Management
VLAN which will be present on all its trunk ports and no additional change needs to be made.
• An unused switch port on any switch can be assigned to the Management VLAN and a technician computer can be
plugged into that port and given a Management VLAN address. This technician computer (often a laptop) can then be
used for network transfer of the configuration to the switch.
• [Preferred solution] A non-network-based method can be used to transfer the switch configuration to the switch, such as a
USB drive. This prevents any networking or VLAN changes from being required on the switch, it can go directly from
out-of-the-box configuration to the final configuration with no temporary reassignments required. The Cisco Catalyst
3850, 3750X, and 2960X switches and routers in the NetworkST product line can transfer configurations via a USB drive
formatted with the FAT32 file system.
4.9.5 Summary
Placing the management interface for network equipment on its own VLAN and then controlling access to it through access
lists in both the routers and the switches themselves offers a much higher degree of isolation, and therefore security, than
placing the management interface on the Plant Data Highway with no access list support. By using the concept of a
Functional IP Address for the WorkstationST Network Monitor function site-specific configurations can be addressed without
having to modify the access lists in the routers and the switches. Systems with management interfaces on a separate VLAN
are now quite common, and this architecture is less likely to trigger questions and concerns when it comes to site security
audits.
The NetworkST 4.0 High Availability (HA) system topology includes two Cisco 2960X XDH switches, two FortiGate 300D
UTM firewall devices, and two Cisco 3850 External (EXT) switches. The following figure illustrates the relationship between
an external network hosting (Example PC) and an Application Server (AP1) present on the PDH network inside the GE ICS.
The highlighted devices complete the HA system.
The following table is an example of port assignment showing the three common interfaces on port 1, 7 and 8.
Note Parts were created to allow for ordering individual units (replacement parts).
Note For a UTM HA system with a 3-year FortiCare agreement, order one 117T6409PX02A and one 117T6409PX03A. If a
1-year FortiCare agreement is preferred, order one 117T6409PX02B and one 117T6409PX03B.
All unused ports should be administratively disabled before placing the switches in
service.
Caution
The following sections provide further details about the available HA External switches:
Non-stacked 12-port External Switch Solution with Single Mode SFP Network Segregation Configuration
Non-stacked 12-port External Switch Solution with Multi Mode SFP Network Segregation Configuration
Stacked 12-port External Switch Solution with Single Mode SFP Network Segregation Configuration
Stacked C3850 12–port External Switch with Multi Mode SFP Network Segregation Configuration
Non-stacked C3850 24–port External Switch with Single Mode SFP Network Segregation Configuration
Non-stacked C3850 24–port External Switch with Multi Mode SFP Network Segregation Configuration
Stacked C3850 24–port External Switch with Single Mode SFP Network Segregation Configuration
Stacked C3850 24–port External Switch with Multi Mode SFP Network Segregation Configuration
• Electrical connections within a building for distances less than 90 m (295 ft) can be provided by copper CAT 5e or CAT 6
cables.
• The maximum limit for 100Base-T and 1000Base-T Ethernet is defined as 100 m (328 ft), with 10 m (32.8 ft) allocated
for potential patch cable connections at the switch and the network device. (Splitting cable conductors at patch panels can
slightly reduce the signal strength and distance allowed for copper Ethernet connections).
• Copper GbE connections cannot exceed 15 m (49 ft).
• Fiber-optic connections are required between buildings. Fiber-optic cables provide electrical isolation between differing
ground potentials that occur between buildings. This is normally most important with lightning strikes within a distance
of a few miles of a plant with the resulting electrical potential wave reaching different buildings at different times. With
the electrical ground wave reaching one building before the next, generating large electrical potential spikes across
inter-building links. A large spike can destroy the network switch or its port, and smaller spikes can disrupt data
transmission.
• PVC conduit is recommended underground for fiber-optic connections because the bends can be formed with a much
larger radius as opposed to metal conduit. Gradual bends can be implemented to bring the conduits above ground to meet
pull boxes as required to minimize pull stress
• Redundancy should be considered for continuing central control room (CCR) access to the turbine controls. Redundant
HMIs, fiber-optic links, Ethernet switches, and power supplies are recommended.
• The optical power budget for the link should be considered. The total budget refers to the brightness of the light source
divided by the sensitivity of the receiver. These power ratios are measured in dBs to simplify calculations. The difference
between the dB power of the source and the dB power of the receiver represents the total power budget. This must be
compared to the link losses made up of the connector and cable losses.
• Installation of the fiber-optic cable can decrease its performance compared to factory-new cable. Installers might not
make the connectors as well as experts can, resulting in more loss than planned. The LED light source can get dimmer
over time, the connections can get dirty, the cable loss increases with aging, and the receiver can become less sensitive.
There must be a margin between the available power budget and the link loss budget of a minimum of three (3) dB.
Having a six (6) dB margin is more comfortable, helping assure a fiber-optic link that will last the life of the plant.
5.2.1 Standards
1000BaseLX – Single-mode fiber-optic (SMF)
5 km over 9-µm single-mode fiber-optic
5.2.2 Cables
Fiber-optic cable is an effective substitute for copper cable, especially when longer distances are required, or electrical
disturbances are a serious problem. The main advantages of fiber-optic transmission in the power plant environment are:
• Fiber-optic segments can be longer than copper because the signal attenuation per foot is less. Fiber-optics is a good
choice for high-bandwidth transmission over longer distances.
• In high-lightning areas, copper cable can pick up currents, which can damage the communications electronics. Since the
glass fiber does not conduct electricity, it provides immunity to lightning and reduces lightning caused outages.
• Grounding problems are avoided with optical cable. The ground potential can rise when there is a ground fault on
transmission lines caused by currents coming back to the generator neutral point.
• Optical cable can be routed through a switchyard or other electrically noisy area and not pick up any interference. This
can shorten the required runs and simplify the installation.
• Fiber-optic cable with proper jacket materials can be run direct buried, in trays, or in conduit.
• High quality optical fiber cable is light, tough, and easily pulled. With careful installation, it can last the life of the plant.
• The total cost of installation and maintenance of a fiber-optic segment can be less than a copper segment.
• Fiber-optic cables can be run in the same conduit or path as the power cables.
Fiber-optic network connections should always be used when:
• The distance between components exceeds the communications specifications limits of copper transmission.
• The grounding conditions require isolation.
• Outside runs are required.
Note Refer to the Fiber-optic Cable and Patch Panel Selection (GHT-200001).
• To minimize variation of equipment, therefore simplifying the network switches’ Bill of Material (BOM)
• Simplification of network design due to elimination of the 550 m (1804 ft) limit for Gigabit Ethernet with multi-mode
fiber-optic cable.
• SMF cable is the standard for network applications using Gigabit Ethernet.
Two connectors are required for duplex operation of each fiber-optic link. Each link consists of two fibers, one outgoing and
the other incoming, to form a duplex channel. The outgoing fiber is driven by a light emitting diode, and the incoming fiber
illuminates a photo-transistor, which generates the incoming electrical signal.
The fiber is protected with buffering which is the equivalent of insulation on metallic wires. Mechanical stress is bad for
fibers so a strong sheath is used, sometimes with pretension Kevlar® fibers to carry the stress of pulling and vertical runs.
Connectors for a power plant need to be fastened to a robust cable with its own buffering.
5.5.3 Setup
Network switches are set up according to the Site Network Topology drawing. The core fiber-optic switches for the new
simplified network topology are connected to the legacy network root bridge or control room fiber-optic switch by
multi-mode fiber-optic cables.
Replacing a root bridge switch in a running system is a high risk procedure and
should only be performed under circumstances in which the replacement cannot be
delayed until a system outage. The root bridge is the focal point of the network and
significant network disturbances leading to an unplanned outage may occur. It is
recommended that this procedure only be undertaken by experienced network
Warning professionals.
This procedure is dependent on redundant communication links on all of the devices that are connected to the network. Any
devices that do not have redundant communication links could lose communication during this replacement procedure. The
system owner needs to understand which devices do not provide redundant communication links and understand the impacts
of communication loss on their system.
The system owner needs to confirm that there are no lurking communication faults in the system prior to preforming this
procedure. When replacing a 3750X stacked switch when one of the switches in the stack is failed, confirm that
communication can be established with the devices connected to the switch that is still functioning.
Attention
➢ To replace an 3750X root bridge switch stack
Note The 3750X switch configuration is not compatible with the 3850. Any changes that may have been made to the
existing 3750X will need to be made on the new 3850. If network customizations were made, update the 3850 configuration
to include the modifications prior to adding it to the network.
1. Identify an unused switch IP address on the network (on the switch management network).
2. Apply power to the new 3850 and allow it to boot.
3. Using Appendix E: Set Switch IP Address and Hostname, log onto the new 3850 and set the IP address of the new 3850
switch to the unused IP address that was identified in step 3. Set the Hostname to NewSwitch.
4. One at a time, move the trunk links from the failed switch in the 3750X stack to the same switch and port position in the
new 3850 stack. Be sure to leave the trunk links to the functional 3750X switch connected.
5. At this point half of the connections should be to the existing 3750X switch and half of the connections should be to
the new 3850 switch.
6. At a minimum, verify the link presence light on the replacement switch device ports is lit. The light is typically green, but
may be amber to indicate the port is configured at a slower speed. The light should be the same color as the
corresponding port on the functional 3750X stack. For additional assurance, verify communication to equipment
connected to the new switch following the procedure in Appendix B: Validate Communication to Devices.
7. One at a time, move the trunk links from the remaining switch in the 3750X stack to the same switch and port position in
the new 3850 stack.
8. At a minimum, verify the link presence light on the replacement switch device ports is lit. The light is typically green, but
may be amber to indicate the port is configured at a slower speed. The light should be the same color as the
corresponding port of the other members of the 3850 stack. For additional assurance, verify communication to equipment
connected to the new switch following the procedure in Appendix B: Validate Communication to Devices.
9. The 3750X switch stack should now be powered down and removed. Any functional switches in the stack can be retained
to act as spares to address future failures.
10. Using the procedure in Appendix E: Set Switch IP Address and Hostname on the new 3850 stacked switch, set the
management interface IP address and Hostname to match the values of the replaced switch.
6.2 Logging
Login/out data is available via the syslog protocol and can be accessed using the SecurityST SIEM console
6.3 Passwords
Passwords associated with privileged access to the switches will be changed by the customer at time of commissioning.
Note To prevent providing your switch credentials during a man-in-the-middle attack, use the capabilities of the SSH client
to verify the public keys on the switch prior to providing your credentials. Do not provide network credentials to switches that
you do not recognize or trust.
Log On to a Switch
Logging onto a switch establishes the user's identity, which determines the privilege level of the user.
If a switch is connected to a SecurityST* system then the username and password used should be a domain account that is a
member of the Network Administrators group.
If the switch is not connected to a SecurityST system then the switch local username and password should be used.
➢ To log on to a switch
1. Connect a console terminal to the switch, or use an SSH client to connect to the management interface on the switch.
2. Select <ENTER> on the terminal session.
3. When prompted, enter the Username and Password for access to the switch.
Note The GE configurations include using RADIUS servers to validate the user's identity and establish their privilege level.
If no RADIUS servers are present the local switch account(s) will be enabled. It will take up to a minute for the switch to give
up trying to contact the RADIUS servers and use the local account(s).
➢ To verify devices that support network teaming connected directly to the switch being tested
1. From the switch, verify the port light is showing as being connected.
2. In the teamed device (HMI, Historian, Engineering Workstation…) open the Control Panel (View by: Small icons) -
Network and Sharing Center - Change Adapter Settings and verify that the UDH and PDH ports show as being
connected.
USB
2. Attach to the console port of a functional Cisco 2960-S switch, log on, and enable commands.
3. Determine the designation of the USB drive (in this example we will use "usbflash0:")
4. Verify the USB flash drive by entering the following command at the switch prompt:
a. dir usbflash0:
b. The command returns the content of the USB flash drive if any. Example:
5. Enter the following command to copy the running configuration into the startup configuration:
a. copy running-config startup-config
6. Copy the content of the startup-config to a new file in the USB flash drive. Example:
a. copy startup-config usbflash0:/sw25_startup_config.txt The command results are the following:
Note If this step is skipped the switch will hang, reading from the USB port during the next step. If that happens, then
remove the USB drive and power cycle the switch
Appendix D: Load a switch Configuration from a USB Port GEH-6840F Application Guide 85
For public disclosure
8. Generate the switch crypto key by entering the following commands:
a. <hostname>#config t
b. <hostname>(config)# crypto key generate rsa general-keys modulus 2048
c. Example output:
The name for the keys will be: <hostname>.HMI.local
% The key modulus size is 2048 bits
% Generating 2048 bit RSA keys, keys will be non-exportable...
[OK] (elapsed time was 54 seconds)
9. Exit from the config mode prompt, then exit the connection.
a. <hostname>(config)# exit
b. <hostname># exit