Professional Documents
Culture Documents
OM and DME PWS - Task Order
OM and DME PWS - Task Order
1
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Table of Contents
1. INTRODUCTION ......................................................................................................................................... 8
2. BACKGROUND ........................................................................................................................................... 8
3. TECHNOLOGY OVERVIEW .......................................................................................................................... 9
4. CONTRACT OVERVIEW ............................................................................................................................ 12
6
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
APPENDIX E - REFERENCES ................................................................................................................................. 190
APPENDIX F - PBGC LOCATIONS ......................................................................................................................... 193
APPENDIX G - IT SERVICE SUPPORT GUIDELINES ................................................................................................ 196
APPENDIX H - IT INFRASTRUCTURE MAINTENANCE SCHEDULE .......................................................................... 197
APPENDIX I - OIG IT INFRASTRUCTURE SUMMARY ............................................................................................ 198
7
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
1. Introduction
This performance work statement (PWS) is for the delivery of professional services to support the
PBGC’S Information Technology (IT) Infrastructure. The PBGC IT Infrastructure Operations Support
Services (ITIOSS) contract provides a wide range of IT professional services supporting a diverse
technological environment. The list of service domains includes end-user services; data center
operations; voice, video, and network infrastructure operations; IT service management; infrastructure
monitoring and reporting; cybersecurity services; test center operations; development, modernization
and enhancements (DM&E); and disaster recovery/continuity of operations planning and testing.
Program management is also necessary to oversee all contracted tasks for the service areas. Two areas
of specific interest are cloud and mobile computing. In addition to limited task-specific, business-area
driven cloud instances, PBGC has recently adopted cloud on an enterprise-wide basis for electronic mail,
collaboration, office automation, individual user data storage and shared, unstructured data storage as
well as IT service management. Mobile computing initiatives include the expansion of office automation
to PBGC’s iPhones and the introduction of Surface Pro 4 tablets and enterprise wireless capabilities
established in 2018. Cybersecurity is a particularly critical service domain and ITIOSS provides both IT
security operations support as well as IT security analysis, ISSO support and audit support. These
supporting services are implemented as an integral part of the solution throughout the entire
enterprise.
2. Background
The Pension Benefit Guaranty Corporation (PBGC) protects the retirement income of more than 40
million American workers in nearly 24,000 private-sector defined benefit pension plans. A defined
benefit plan provides a specified monthly benefit at retirement, often based on a combination of salary
and years of service. PBGC was created by the Employee Retirement Income Security Act (ERISA) of 1974
to encourage the continuation and maintenance of private-sector defined benefit pension plans, provide
timely and uninterrupted payment of pension benefits, and keep pension insurance premiums at a
minimum. PBGC is not funded by general tax revenues. PBGC collects insurance premiums from
employers that sponsor insured pension plans, earns money from investments and receives funds from
pension plans it takes over. A department within PBGC’s Office of Information Technology, the IT
Infrastructure Operations Department (ITIOD) provides the mission-critical IT foundation for computing
services used and procured by PBGC. PBGC cannot accomplish its mission without automated tools and
business software. All other PBGC departments depend on this one for their IT support. PBGC’s Common
Security Controls under the IT Infrastructure Services General Support Systems (ITISGSS) are also
provided by this department. ITIOD is comprised of approximately 40 PBGC FTE and approximately 150
contractors. ITIOD stakeholders include a dedicated PBGC staff of approximately 2,250 government and
contracted personnel, 26,000 pension plan sponsors who pay premiums into the PGBC and over 1.5
million trusteed participants who, combined, receive over $5 billion dollars in benefits payments.
8
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
3. Technology Overview
The following information is provided as a high-level overview of PBGC’s IT infrastructure environments
and technology. Greater detail is provided in the succeeding delineations of the various service
domains.
The PBGC currently operates and maintains three separate yet highly interrelated environments within
its IT Infrastructure. Additionally, ITIOD provides limited support to the PBGC Office of Inspector
General (OIG), as detailed in Appendix I - OIG IT Infrastructure Summary. These are logically separate
infrastructures dedicated for (1) development, (2) testing, (3) production. Each is described below:
• The Common Development Environment (CDE) provides an area for development and
interoperability testing that is isolated from production and other environments. The CDE
provides the capability to develop major applications and infrastructure solutions, which
support PBGC’s changing business needs. The environment allows PBGC to take advantage
of emerging technologies and to improve the exchange of data with internal and external
customers. The CDE is currently comprised of two separate areas (CDE-T and CDE-I):
The CDE-T is the “Team-Specific” area, used for code development, COTS
configuration and unit testing of development efforts.
The CDE-I is the “Interoperability” area, used for various phases of testing
(functional, system, integration, performance, and deployment) of future
integrated releases. The CDE-I is further broken down into two sub-areas – “As Is”
and “To Be.” The “As Is” area reflects the current production environment and
is used for testing against applications that currently exist in production. This area
may be used for supporting production, emergency fixes, and point releases.
Currently this area also serves as the training environment for some custom
developed PBGC applications. The “To Be” area reflects the future production
environment and is used for testing against the applications that will be deployed
in the next major software release and limited training purposes.
• The Integration and Testing Center (ITC) environment has been established for deployment
verification, development shakedown, system integration verification and, user acceptance
testing (UAT). Vulnerability scanning is also conducted for releases in the ITC prior to
production release. Training can be performed in ITC when needed. Currently, testing (and
training) in the ITC is accessed by physically going to the facility located at 1275 K Street, but
over the next 6-12 months this is expected to become a virtual test center and access to it
will be achieved entirely using remote desktop connectivity
• The Production environment (PROD) hosts business application and infrastructure
operations including replication of software and data to the COOP site with similar
hardware.
The disaster recovery (DR) / COOP environment provides redundancy for the
production environment and facilitates the continuation of business activities in the
event of a disruption to the production environment. Failover is to PBGC’s servers at
Wilmington, DE, which have similar cache boxes and web switches with server
health monitoring and failover.
9
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
The DMZ environment provides the infrastructure for public-facing applications and
services. The DMZ includes its own Active Directory and has a presence at both the
HQW and the COOP sites.
PBGC also has several applications and infrastructure services that are largely
supported externally and hosted off site including cloud and non-cloud managed
services
The application deployment process is initiated in the CDE by the contract development team. Solutions
are then deployed to the ITC, based on a deployment guide prepared by the contract development
team. This ITC deployment is a joint effort between the contract development team and the IT
Infrastructure O&M team. The contract development team is responsible for the communication and
coordination effort associated with the CDE, ITC and production deployments.
PBGC has more than 2,500 physical Windows workstations (desktops, laptops, and tablets) and more
than 500 virtual desktops all of which run Windows 10. PBGC is moving towards portable hardware, e.g.
MS Surface Pro 4, Latitude 5290 2-in-1, etc. PBGC also has approximately 1,000 GFE iPhone 6s devices
managed using InTune and providing users with access to PBGC’s Microsoft Office 365 tenant including
email. PBGC plans to upgrade these phones to the iPhone 8 plus model by the end of the 2018 calendar
year. PBGC has established a significant SharePoint capability and Office 365 (Government Community
Cloud E3 plan) as an enterprise collaboration tool. The majority of PBGC’s unstructured data resides on
Office 365.
In support of all environments, ITIOD leverages a Hewlett-Packard (HP) blade server infrastructure, using
VMWare for most systems. ITIOD takes a “virtualize first” approach whenever possible. Current server
operating systems are a mix of Windows, Redhat Linux, and Solaris (physical Sun servers).
The PBGC applications and databases are a mixture of custom, COTS and highly customized COTS.
Further, they have been designed, deployed and supported by multiple contractor teams, contracted for
by multiple PBGC front-line business areas. PBGC primarily uses Oracle and MS SQL databases and is
considering shifting its primary relational database services to MS SQL Server and other Microsoft Azure
cloud database offerings, over a timeline yet to be determined. Furthermore, PBGC is considering
shifting its many of its application middle-tier services from Oracle WebLogic and IIS to Azure and
Dynamics cloud-based services, also over a timeline yet to be determined.
PBGC maintains Storage Area Network (SAN) infrastructure to provide high volume and high-
performance data storage. PBGC primarily uses Brocade SAN switches connecting Hitachi storage
arrays. PBGC also uses Veritas NetBackup and Oracle Recovery Manager (RMAN) for backup and
recovery services.
ITIOD supports PBGC business functions through a number of core business applications, both custom
and COTS, Microsoft Office 365 E3 including SharePoint online, and numerous supporting applications
for office productivity including word processing, spreadsheet, graphics, email, collaboration, browser,
communications connectivity, etc.
10
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Core business applications presently use a mix of Web-based thin client World Wide Web Consortium
(W3C)-compliant browser-based n-tier applications, client/server and service-based architectures. All
current EA-compliant development is service- oriented with browser-based user interfaces.
PBGC supports secure remote access to networked resources via VPN on GFE devices. For non-GFE or
devices, PBGC also offers an RDP web proxy service using Pulse Secure.
PBGC uses ServiceNow for IT Service Desk interactions, incident management, problem management,
change management, asset management, configuration management and discovery. HP Service
Manager 9 service request and service catalog software is used to manage service and access requests.
PBGC plans to migrate these capabilities to the Service Now platform by the end of September 2019.
SailPoint Identity IQ suite will also be leveraged to fulfill access requests for applications that are not
Active Directory-integrated or federated to the extent possible.
PBGC uses HP Business Service Management (BSM) and HP Business Availability Center (BAC), Sitescope,
Real User Monitor (RUM), and Network Node Manager (NNMi) for monitoring the availability and
performance of IT infrastructure systems and applications.
PBGC uses the Microsoft Project On-line (referred to in PBGC as P3M) and Oracle/Primavera Project,
Program and Portfolio Management (PPM) system as a central repository for tracking and reporting
project data. Enterprise Architecture (EA) information is in the custom- developed EA Repository (EAR).
PBGC has selected Azure as its default IaaS and PaaS CSP and expects to shift more and more services to
this over the next several years.
11
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
4. Contract Overview
In addition to other requirements, this contract will also include following objectives:
1. In addition to operations and maintenance of the existing environment, this contract will include
the infrastructure systems engineering and deployment services to perform development,
modernization and enhancement work.
2. The performance of the contractor will be measured and evaluated based on Service Level
Agreements (SLAs) comprised of a set of service level metrics with clearly defined acceptable
levels of quality (AQL), which will be reviewed and adjusted as required quarterly.
3. The contractor shall be required to continuously identify, recommend, achieve, and report on
measurable results for on-going operations and shall specifically provide before and after
measurements that can quantify achievement of development results.
4. Gradually, under a phased approach and following the ITIOD roadmap, the current capital-
intensive, government-owned, locally hosted infrastructure will be transitioned to alternative
service delivery models.
Option Periods: Nine (9) 12-month performance periods subject to exercise by PBGC
12
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
4.4 Contract Phase Out
The Contractor shall provide transition phase-out support and various activities to transition support to
the Federal Government or a third-party service provider at contract end-of-life. Contract phase-out
activities will include those activities defined in the Contract Phase-out deliverable due 90 days before
the end of any awarded period of the contract.
13
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
5. Business Process Support
This section of the PWS outlines those services and requirements that relate to the ancillary yet critical
business process that support ITIOD’s core mission, the provision of IT infrastructure services and tools.
5.1.1.2 Objectives
The contractor shall consider the following objectives when providing program and process
management services:
1) Ensure all IT infrastructure support services are provided timely, accurately, and in a quality
manner, in full compliance with all service level objectives and metrics
2) Cost of services are controlled, and the total cost of infrastructure asset ownership is lowered
on an annual basis via introduction of operational efficiencies
3) Development, modernization and enhancement projects are executed within initial cost and
schedule baselines, and are fully responsive to documented user and system requirements
4) Provide transparency to customers and end-users on service cost, performance, and satisfaction
14
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
5.1.1.4 Requirements
The Contractor shall perform the following services for the Program and Process Management services
under this solicitation:
Reference Requirement
(PM-PM-xx)
PM-PM-01 Contractor shall be responsible for all technical training of Contractor staff, unless
otherwise directed by the Government. The Contractor shall provide technical
staffing proficient in the tools and technologies utilized, supported, planned, and
targeted under this contract. The contractor shall develop a set of experience and
skills required for each role, position, and function on the contract. The set of
experience and skills required for each role, position, and function on the contract
shall be reviewed and approved/accepted by the government prior to the experience
and skill definition to be considered as official for the services, roles, functions, and
support provided and performed on the contract. Additionally, in those cases where
hardware or software vendors require certified technicians to interact with their
products, the Contractor shall ensure that personnel are appropriately certified.
Appropriate certification documents shall be submitted to COR.
17
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PM-PM-12 PBGC considers effective and productive communications essential to a collaborative
relationship and to the success of the ITIOSS Program. Expectations on technical, cost,
schedule, performance and progress towards established expectations (or variances
from), must be communicated clearly and without ambiguity or delay. To this end,
the Contractor shall establish and maintain effective communication with the
Government. The Contractor will structure its ITIOSS support in a manner that
ensures that the Contractor’s goals and objectives are aligned with those of the
Government and reflect the attributes of a partnership through an open, customer-
oriented effort.
The Contractor shall communicate with the Government during all phases of the
contract and, at a minimum, take steps to understand the Government’s business and
technical issues, provide insight into issues and problems, recommend solutions for
issues, and recommend actions to maintain cost, schedule, quality, and technical
baselines. The Contractor shall have effective communications processes and plans to
identify, prepare, review, incorporate review comments, disseminate, and track
appropriate communication items (i.e., deliverables, work products, correspondence,
etc.) for the ITIOSS Program.
Documentation services are the activities associated with developing, revising, maintaining,
reproducing, and distributing up-to-date information in hard copy and/or electronic form across all
service areas. These documents shall be collectively referenced as the ITIOSS Standards, Procedures, and
Concept of Operations (CONOPS). The Contractor shall at a minimum:
• Provide documentation in agreed format to support activities throughout the life cycle of
services as specified in each service area.
• Follow PBGC’s Record Management policies and procedures.
• Maintain a system to centrally store, organize and distribute documentation.
• Document system specifications and configurations, e.g. interconnection topology,
configurations, and network diagrams).
• Document standard operating procedures (e.g., service desk, datacenter, network, boot,
failover, spool management, batch processing, backup, etc.).
• Document procedures, production and maintenance schedules, and job schedules, according to
PBGC policies and industry requirements.
• Document Meeting Minutes/Agendas: This document will be used to capture meeting agendas,
attendees, meeting highlights, meeting outcomes and meeting action items.
18
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Develop, maintain, and update all documentation on a regular basis
Refer to Appendix B – Deliverables and Appendix C- List of Required Meetings and Reports
5.2 Transition
The sections below describe the overview, objectives, scope, requirements, deliverables and SLAs of the
Transition services that the Contractor shall provide under this solicitation.
5.2.1 Overview
The Contractor shall submit a comprehensive Transition Plan in accordance with the PWS that shall
contain a detailed description of tasks, resources, schedule, assumptions, dependencies, risks and risk
mitigation plans associated with the transition.
Upon Government approval of the Transition Plan, the Contractor will accept and sustain ITIOSS
Services. This task involves the transition of the existing environment to an environment that will
support the core service areas as defined below. This transition phase will incorporate consolidation and
streamlining of IT assets and various other activities that are deemed necessary by the PBGC and the
Awarded Contractor to begin services under the new contract. This transition will involve the effort to
set up the needed infrastructure (if applicable) and installation and implementation of management
tools and agents, staffing plans and other preliminary activities needed to prepare for the start of the
contract. The overarching objective of this phase is a low risk and low impact (to end-users) transition as
the Awarded Contractor assumes responsibilities under the ITIOSS program. Additionally, the Contractor
shall specifically prepare for transition to an external, “rural-sourced” Service Desk akin to PBGC’s
current model or an external Service Desk of another model.
5.2.2 Objectives
The Contractor shall perform all services, tasks, and any other support activities required to transition
from the current version of the ITIOSS contract. The existing services under this contract are vital to
PBGC’s mission and must continue without interruption. The Contractor agrees to exercise best efforts
and cooperation to affect an orderly and efficient transition phase after award.
PBGC has identified the following key service objectives for transition in the table below:
19
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Adaptive Communication Adapt the Contractor’s transition approach to PBGC requirements
through the establishment of an adaptive communication strategy
and plan proper execution.
Institute Partnership Define and establish multi-level review and assessment meetings
Expectations and Collaboration with PBGC leadership to collaborate and ensure clear customer
expectations inclusive of published information, schedules, and
progress reports.
5.2.4 Requirements
A startup transition period not to exceed sixty (60) days shall be required after final contract award for
the Contractor to conduct transition services. During the transition period, the Contractor shall
implement all activities necessary to establish a stable environment, where the Contractor assumes
operational control under PBGC’s Infrastructure Operations Department (ITIOD). Phase-in activities
include the finalization and formal adoption of ITIOSS contract SLAs, coordination and activation of the
business systems supporting the new ITIOSS environment and presentation of a Transition Readiness
Review (TRR) in which specific readiness criteria and status are presented. The Contractor’s transition
readiness includes satisfaction of a number of specific criteria including the establishment of:
1) Detailed transition plan, process and schedules coordinated with PBGC and incumbent
contractors for providing an orderly transition with following objectives:
a. Minimize the impacts on continuity of operations
b. Maintain communication with staff and affected communities
c. Identify key issues
d. Overcome barriers to transition
e. Perform due diligence to ensure that all transition activities are identified, negotiated
and completed during the Transition
f. Establish a transition management team capable of providing overall management and
logistical support of all transition activities
2) Contractor transition management team leaders on-board immediately after the award.
3) Required operational interface agreements (e.g., OLAs, Memoranda of Agreement (MOA)) with
incumbents prepared, reviewed, and signed.
4) Appropriate subcontractor and supplier agreements in place.
5) Qualified personnel identified, processed, and available at specific incremental turnover dates.
6) Applicable contractor management systems and support tools in place.
7) Status reporting requirements established, including applicable performance measurement
reporting requirements
20
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
8) PBGC and contractor agreement on existing systems, plans, procedures, forms, and instructions
to be used after transition.
9) Institutionalized knowledge of PBGC operations and technical requirements.
10) An appropriately sized external, “rural-sourced” Service Desk akin to PBGC’s current model or an
external Service Desk of another model
22
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
5.3.1 Location of Documents and Records
The Contractor shall publish all deliverables, reports, SOPs, Work Instructions and other required
documents on PBGC’s instance of SharePoint on-line according to applicable ITIOD processes and
procedures.
The Contractor shall participate in efforts to address IT security control deficiencies related to
infrastructure services the Contractor provides or systems the Contractor maintains, tracked through a
Plans of Action and Milestones (POA&Ms). The IT security control deficiencies may be identified
through an audit, control assessment, or by ITIOD staff during the course of normal operations or
internal control review. Participation consists of attending meetings, reviewing control requirements,
identifying gaps, identifying steps to address control deficiencies, taking steps to address control
23
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
deficiencies, and producing artifacts demonstrating the proper functioning of an IT security control.
POA&Ms will generally have an assigned federal team lead who will provide the Contractor with any
clarifications required. The overall effort will be led by ITIOD’s Security Program Management Office.
During the FY18 audit cycle, ITIOD received and process more than 240 data and meeting requests from
the auditors. In calendar year 2018, ITIOD worked on a total of 9 POA&Ms, completing and closing 2
POA&Ms and 63 POA&M milestones covering control families such as: Access Control, Configuration
Management, Identification and Authentication, Risk Assessment, System and Services Acquisition.
5.3.5.2 Requirements
The Contractor shall perform the following services for the Audit and POA&M Support services under
this solicitation:
Reference Requirement
(PM-AP-xx)
PM-AP-01 Contractor shall provide timely support for security audit-related data calls, reporting,
and presentations. Contractor will general have 1 week to complete an assigned
request.
PM-AP-02 Contractor shall provide support for POA&Ms and POA&M milestones including
attending meetings, reviewing control requirements, identifying gaps, identifying
steps to address control deficiencies, taking steps to address control deficiencies, and
producing artifacts demonstrating the proper functioning of an IT security control.
PM-AP-03 Contractor shall complete assigned POA&M milestones on time. See SLA section for
details on existing SLA measures, which define acceptable quality levels.
24
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
as the Occupational Safety and Health Administration (OSHA), Environmental Protection Agency (EPA),
Government Accountability Office (GAO), and General Services Administration (GSA). For all
correspondence assistance relating to any audits and all Congressional inquiries, the Contractor shall not
actually draft any response. Input may be provided but the drafting of such responses will be done by
the Government. Additionally, the Contractor shall only communicate with other government agencies
through the COR.
The Contractor shall maintain all processes, procedures, and WIs throughout the life of the Contract
including review, verification and update (if needed) no less than annually. The contractor will adhere to
the governance processes for processes, procedures, and WIs including change notification and federal
approval. The Contractor shall notify the COR at least 30 days in advance of any changes that may affect
contract cost and not implement these changes until receiving approval from the COR.
This section and the sections that follow describe the scope and requirements that the Contractor shall
provide under this solicitation which include:
• End-User Services
• Data Center Services
• Voice, Video, and Network Infrastructure Operations
• IT Service Management (ITSM) and Infrastructure Monitoring and Reporting
• Security Services
• Test Center Operations
• Development, Modernization and Enhancements
• Disaster Recovery/Continuity of Operations Planning (COOP) and Testing
• Cloud Integration and Support
See Appendix J – IT Service and Support 2018 Statistical Summary for IT Service Desk interactions, tier 2
incidents, requests for information (RFIs), service/access requests, requests for change (RFCs), and RFC
tasks processed in calendar year 2018 for these services.
Additionally, PBGC provides limited but foundational support to PBGC’s Office of the Inspector General
(OIG). Complete details of that support can be found in Appendix I - OIG IT Infrastructure Summary.
6.1.2 Requirements
The contractor shall comply with all general requirements outlined in the following table:
Reference Requirement
(General-xx)
26
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
GENERAL-01 Contractor shall be in full compliance with and support the enforcement of all Federal
and PBGC policies and security controls
GENERAL-02 Contractor shall ensure that all work performed meets or exceeds acceptable quality
levels defined and shall provide evidence of having attained those levels of service.
See SLA section for details on existing SLA measures, which define acceptable quality
levels.
GENERAL-03 Contractor shall regularly sample, review, and document Information Technology
Service Management (ITSM) interaction, incident, and change quality levels and take
appropriate action, e.g. training, to rectify any deficiencies in ticket quality and
service delivery including failure to meet target performance metrics
GENERAL-04 Contractor shall ensure knowledge base articles (KBAs) or work instructions (WIs) are
established and maintained for common activities associated with interactions,
incidents, and changes and these are updated/certified no less than annually
including federal approval
GENERAL-05 Contractor shall fully implement, manage, and support all activities regarding incident
management according to PBGC policies. By using ITIL best practices, this should
include, but shall not be limited to:
27
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
GENERAL-09 Contractor shall restrict the access to the IT Infrastructure General Support System to
authorized users only using a least privilege methodology and shall maintain data
integrity and prevent unauthorized use and release of PBGC information in
accordance with PBGC policy and procedures
GENERAL-10 Contractor shall revoke access based on the PBGC security guidelines, separation of
employment, evidence of dormant accounts, and other administrative reasons in
accordance with PBGC policy and procedures
GENERAL-11 Ensure all SLAs are met and reported as directed in this PWS
• Telephone
• On-line ITSM Tool (Service Now) Chat
• On-line ITSM Tool (Service Now) ticket submission
• Email
Leverage and maintain the knowledge base module of the ITSM tool suite (currently Service Now) to
promote self-service, first call resolution, consistent support, and timely service restoration. Perform
basic account administration functions. See Appendix J – IT Service and Support 2018 Statistical
Summary for IT Service Desk interactions, tier 2 incidents, requests for information (RFIs), service/access
requests, requests for change (RFCs), and RFC tasks processed in calendar year 2018 for these services.
See Appendix K – IT Infrastructure Tools List for the software utilized to provides these services.
6.2.1.2 Requirements
The Contractor shall establish an IT Service Desk (as defined by Information Technology Infrastructure
Library (ITIL)), serving as the single customer-facing point of contact. The Service Desk will represent
ITIOD as the service provider to its end-users for all IT service and support including incidents, problem,
28
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
requests, advice, guidance and the rapid restoration of normal services, to meet service levels and
manage customer expectations in accordance with ITIL best practices and principles. The Contractor
shall propose either an external, “rural-sourced” Service Desk similar to PBGC’s current model or an
external Service Desk of another model. Except for a limited, on-site Site Support function, PBGC no
longer sees value in hosting a complete Service Desk on premises. Given past negative experience with
Service Desks staffed in the metropolitan DC area, the Service Desk model proposed will be an
evaluation factor and the evaluation will focus on cost, staff turnover and quality of service.
29
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
The contractor shall provide the IT Service Desk services outlined in the following table:
Reference Requirement
(EU-SD-xx)
EU-SD-01 Contractor shall identify a lead for the Service Desk area. This lead is required to serve
as the primary point of contact for all service desk related issues.
EU-SD-02 Contractor shall participate and support the following processes and functions using
ITIL best practices:
• Incident Management including Service Desk interactions and/or incidents
• Requests for Information (RFIs)
including compliance with PBGC-established processes in these functional areas
EU-SD-03 Contractor shall fully implement, manage, and support all activities regarding incident
management according to PBGC policies. By using ITIL best practices, this should
include, but shall not be limited to:
• Receive, respond, escalate, and resolve all IT Service Desk related calls,
emails, chat sessions, and on-line ticket submissions
• Establish a focal point for all customer communications and send PBGC
approved advisories and maintain the main Service Desk phone message, the
on-line IT Service Desk Portal (GetITAll), “Splash” screens, and intranet
content to communicate significant IT events and information, e.g. schedule
maintenance, unscheduled outages, etc. The Service Desk sends two (2) to
five (5) advisories in a typical week although more may be required if there
are significant outages to report. Updates to “Splash” screens and/or
intranet content occur less frequently; once or twice a month.
30
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
EU-SD-08 Contractor shall perform basic account administration functions in accordance with
PBGC policy and procedures for systems and functions where automation is not
already in place including, but not limited to:
• Active Directory using Quest Active Roles Server:
o Updating user information, e.g. primary workstation, name, display
name, job title, associated contract, etc.
o Account unlocks and emergency/temporary disable/re-enable
o Creation of new distribution groups
o Generation of access reports
o Password Resets (including remote user verification when applicable
per PBGC policy)
o HSPD-12 temporary exemptions
• PBGC’s User Provision Tool (UPT)
o providing and removing access to target systems
o account separation
• Leapfile File Transfer
o Account creation upon request
o Account removal upon request or separation
EU-SD-09 Contractor shall establish and maintain, with updates no less frequently than
annually, IT service and support one or more job aids which provide a brief summary
of instructions or a checklist to ensure users know how to obtain IT Services and
Support, e.g.:
• General User – IT Support Job Aid
• General User – IT Requestor Job Aid
• IT Access/Request Approver Job Aid
• IT Access/Request Fulfiller Job Aid
EU-SD-10 Contractor shall assist customers with identifying the appropriate the IT service
catalog item/role to request and the most efficient way to request it, e.g. manual
submission, bulk submission, requests for control groups, etc., including ensuring
access requests are in compliance with approved governance
EU-SD-11 Contractor shall collaborate with PBGC by contributing, subject to Federal approval,
to the development and maintenance of Operating Level Agreements (OLAs) between
ITIOD Service Desk and all other IT functional areas throughout the life of the contract
Provide tier 2 site support to individual end users for IT issues that cannot be resolved by the Service
Desk, as well as support service requests. Typical site support activities include:
31
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Troubleshoot and repair or replace defective IT equipment (PC, laptop, monitor, cabling, IT
supported local and network printers, phone hand and head sets, mobile phones, etc.) as
needed
• Troubleshoot desktop operating systems including space cleanup, user profile repair or replace,
operating system re-image when needed, etc.
• Troubleshoot and reinstall desktop software including office automation, productivity tools, and
business application software as needed
• Troubleshoot mobile phone issues including device reset and re-enrollment and perform remote
wipes on mobile phones reported lost or stolen
• Relocate IT equipment in response to user relocation requests
• Install IT equipment for new user setups and remove upon user separation including
management of port security as needed/applicable
• Add/install additional equipment in response to user request, e.g. second monitor
• Enroll, configure, and provision mobile phones to end-users and remove upon separation
• Support IT equipment in conference and training rooms and provision additional IT equipment
to support specialized requirements upon request. In calendar year 2018, there were 38
requests for special conference room setups and 25 VTC setup requests.
This support is often provided remotely, using Skype for Business for screen sharing, but sometimes
requires physical desk-side visits to address certain hardware and software problems. See Appendix F -
PBGC Locations, including those requiring site support staff as well as planned changes to PBGC’s
facilities over the life of the contract. See Appendix G - IT Service Support Guidelines for impact,
urgency, and prioritization guidelines associated with IT service and support. See Appendix J – IT Service
and Support 2018 Statistical Summary for IT Service Desk interactions, tier 2 incidents, requests for
information (RFIs), service/access requests, requests for change (RFCs), and RFC tasks processed in
calendar year 2018 for these services. See Appendix K – IT Infrastructure Tools List for the software
utilized to provides these services.
Providing support network printers and multi-function devices is out of scope for this contract. This
support is provided by federal staff and the multi-function device lessor.
PBGC’s environment supported by site support consists of many components detailed in the tables that
follow:
Printer Summary
PBGC has approximately 136 network multi-function, printers, and plotters as well as 290 IT supported
local printers as outlined in the following table:
32
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
HP Color LaserJet Miscellaneous 13 13
HP B&W LaserJet Miscellaneous 18 18
EPSON SC-P20000 2 2
Brother HL-3170CDW 42 42
Brother HL-3170CDW 2 2
Brother DCP-L2550DW 161 161
HP LaserJet P2035 85 85
Total 136 290 426
ITIOD is currently wrapping up the process of standardizing the network printer fleet through our
existing enterprise printer lease contract, currently with Ricoh, and will be phasing out a few more of the
non-Ricoh printers. PBGC has more than 75 local printers, mostly HP LaserJet P2015 or 2055 that are
not supported and will be phased out on move to the new headquarters building.
6.2.2.2 Requirements
The Contractor shall provide a walk-up Site Support capability for PBGC employees and contractors, for
those services requiring in-person response that cannot resolved by the IT Service Desk, as well as
support for service requests. This involves providing the end-users, from the PBGC inventory of
government furnished equipment, with a standard configuration desktop or laptop (depending on job
function, location, and/or end-user preference) and any approved peripherals, standard PBGC–approved
COTS office automation and productivity software (e.g., MS Office), and access to any authorized PBGC
COTS and custom-developed applications and resources as required.
The contractor shall provide the Site Support services outlined in the following table:
Reference Requirement
(EU-SS-xx)
EU-SS-01 Contractor shall identify a lead for the Site Support area. This lead is required to serve
as the primary point of contact for all site support related issues.
EU-SS-02 Contractor shall fully implement, manage, and support all activities regarding site
support incident management according to PBGC policies. By using ITIL best practices,
this should include:
• Receive, respond, escalate, and resolve all Site Support walk-up requests
• Provide support for desktop and laptop hardware and operating systems
including basic network connectivity and peripheral support
• Provide support for IT supported, centrally managed (COTS/GOTS)
applications on end-user workstations, e.g. Microsoft Office, Adobe Reader,
Adobe Pro, CRM, Specturm, CFS, PPS, Comprizon, etc.
• Provide support for IT provided phone handsets and headsets and associated
connectivity
• Provide support for IT provided and managed mobile devices, e.g. smart
phones
• Provide support for IT supported local printers
33
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
See SLA section for details on associated SLA measures.
EU-SS-03 Contractor shall provide a walk-up Site Support capability for PBGC employees and
contractors to obtain IT service for services that can only be handled face-to-face, e.g.
PIV card testing, mapping and PIN resets; mobile phone support, asset pickup/drop-
off, password pickup, facilitating new employee/contractor training conducted in the
same space, etc.
Contractor shall offer this service at the HQ campus. See Appendix F – PBGC
Locations, for sites beyond HQ requiring site support staff now as well as planned
changes to PBGC’s facilities and associated services over the life of the contract.
EU-SS-04 Contractor shall provide services to install, move, add, and change (IMAC) IT
equipment for end-users including desktops, laptops, monitors, cabling, IT supported
local and network printers, IT supported phone hand and headsets, mobile phones,
etc., in accordance with PBGC procedures and using PBGC approved security
configuration baselines and associated images when available.
Contractor shall ensure that all changes resulting from IMAC activity are properly and
accurately recorded in PBGC’s asset management system.
34
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
6.3 Data Center Services
The sections below describe the scope and requirements of the Data Center services that the Contractor
shall provide under this solicitation. Providing stable, reliable, secure, optimally performing, and highly
available systems and service is critical to enable the accomplishment of the agency mission and as such,
are critical to excellent performance under this contract. Thus, Data Center services is a key service area.
Much of how the customer views the success of this contract will be dependent on how well the
Contractor administers and supports PBGC’s data center environment and platforms, and how satisfied
ITIOD staff members are with the IT services provided. Data Center services include the following:
The on-premise IT equipment, e.g. servers, SAN, etc., that supports Data Center services is largely
located at PBGC’s HQW location today. The on-premise IT equipment that supports PBGC’s disaster
recovery capability is currently located at PBGC’s Wilmington (WIL) facility. PBGC plans to move most of
the on-premise IT equipment supporting the Data Center services to co-located data centers over the
next several years. Please refer to Appendix F – PBGC Locations for a tentative timeline for this
transition.
• Monitor, troubleshoot and repair or replace defective IT equipment (stand-alone servers, blade
infrastructure and servers, and associated components) as needed
• Monitor, troubleshoot, and repair Windows Server operating systems including space cleanup,
event log analysis, role and feature reconfiguration and reinstall, operating system re-image
when needed, etc.
• Troubleshoot and reinstall application software including office automation, productivity tools,
infrastructure software, and business application software as needed on Windows servers
• Maintain up-to-date physical and virtual Windows Server operating system images/templates
• Provision and configure new physical and virtual servers as requested including SAN connectivity
35
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Provision and configure new blade infrastructure, e.g. chassis, out of band management
modules, connectivity modules (virtual connect flex fabrics), blade servers and associated blade
profiles, etc.
• Perform initial installation and configuration of operating system roles and features, application
software including office automation, productivity tools, infrastructure software, and business
application software as needed on Windows servers
• Remove/decommission physical and virtual servers as requested
• Perform Windows Active Directory account administration for privileged and service accounts
• Monitor, troubleshoot, and repair Windows Server Active Directory and associated services
• Create and modify Active Directory group policy objects as required to ensure compliance with
Windows operating system baselines as well as to achieve desired operational configuration and
user experience
• Deploy patches monthly to Windows servers using patch deployment tool, e.g. BigFix
• Address operating system and software vulnerabilities detected on Windows servers during
monthly vulnerability scans
• Configure and administer NTFS file systems and Windows file shares
• Configure and administer Microsoft Windows Certificate Authorities and associated services
• Configure, monitor, troubleshoot, and repair high availability and disaster recovery capabilities
pertaining to Windows Server and associated services including, but not limited to Windows
clustering, distributed file system replication (DFS-R), failover procedures, etc.
• Support internal and external IP address management and name resolution services by updating
IPAM DNS records
• Maintain server racks
• Establish and maintain work instructions
• Escalate to and work collaboratively with 3rd party vendors on hardware and software issues
Support for Windows Servers and IT equipment in the PBGC data centers is typically handled using
remote management software, e.g. RDP, vCenter console, SSH, Powershell, ILO, OA, etc., but does
occasionally require physical visits to address hardware and software problems. See Appendix F - PBGC
Locations for PBGC locations, including planned changes to PBGC’s facilities and data centers over the
life of the contract. See Appendix G - IT Service Support Guidelines for impact, urgency, and
prioritization guidelines associated with IT service and support. See Appendix J - IT Service and Support
2018 Statistical Summary for IT Service Desk interactions, tier 2 incidents, requests for information
(RFIs), service/access requests, requests for change (RFCs), and RFC tasks processed in calendar year
2018 for these services. See Appendix K - IT Infrastructure Tools List for the software utilized to provides
these services.
PBGC’s Windows Server environment consists of many components detailed in the tables that follow:
36
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PBGC has 4 separate and independent Windows 2008 R2 Active Directory Forests servicing the various
PBGC computing environments.
PBGC has approximately 629 Windows servers, with more than 75% of them being virtual. The following
table breaks them down by physical/virtual and environment:
37
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Virtual 21 11 4 42 22 100
Physical 37 4 74 1 116
Windows Server 2012 R2
Virtual 120 27 23 126 76 372
Windows Server 2016 Physical 0
Virtual 3 3
Windows Server 2019 Physical 0
Virtual 2 2
Total Count 198 57 27 248 99 629
An upgrade to Windows Server 2016/2019 is in progress and will be largely completed in calendar year
2019 to the extent possible.
PBGC has approximately 152 physical Windows servers, of which almost all are HP Proliant servers and
the majority of which are HP Proliant blade servers. The following table breaks them down by model and
operating system:
38
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
An upgrade from HP Proliant Generation 5, 6 and 7 servers to HP Proliant Generation 10 servers is
planned for calendar year 2019 as part of the Windows Server 2016/2019 upgrade project.
The following table provides the count for EnableIT service requests processed in Calendar Year 2018 for
Windows servers:
6.3.1.2 Requirements
The Contractor shall provide incident response and resolution for issues relating to Windows servers or
requiring administrative access to Windows servers that cannot be resolved by an End-User services
team as well as support service requests and requests for change.
The contractor shall provide the Windows Server administration and support services outlined in the
following table:
Reference Requirement
(DC-WS-xx)
DC-WS-01 Contractor shall identify a lead for the Windows Server area. This lead is required to
serve as the primary point of contact for all Windows Server related issues.
DC-WS -02 Contractor shall provide and maintain a fully functional, optimally performing, secure
Windows Server infrastructure in all PBGC computing environments. This shall
include, but is not limited to:
39
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
according to PBGC policies. By using ITIL best practices, this should include, but shall
not be limited to:
40
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
DC-WS-06 Contractor shall maintain up-to-date physical and virtual Windows server operating
system images and templates for supported Windows Server operating system
versions that are in compliance with PBGC-approved security baselines and that
include PBGC approved security patches within 60 days of production security patch
approval, i.e. updated no less infrequently than every other month
DC-WS-07 Contractor shall administer and support PBGC’s internal certificate authority and
associated certificates according to PBGC policy and procedures
DC-WS-08 Contractor shall administer and support NTFS file systems and Windows file shares
according to PBGC policy and procedures including securing to standards and
performing quota management
DC-WS-09 By the end of January of each year, contractor shall:
• Upgrade each Java Platform (JRE/JDK) instance on Windows servers to the
identified target version established the previous January unless risk accepted
• Identify and communicate the target Java platform version for the following
January
DC-WS-10 Contractor shall perform Windows account administration functions in accordance
with PBGC policy and procedures for systems and functions where automation is not
already in place including, but not limited to:
• Active Directory using Quest Active Roles Server:
o Account unlocks and emergency/temporary disable/re-enable for
privileged and service accounts
o Password Resets (including remote user verification when applicable
per PBGC policy) for privileged and service accounts
o Account removal for privileged and service accounts upon request or
separation
DC-WS-11 Contractor shall maintain server racks to include, but not limited to:
• ensuring front and rear doors can be closed and are closed
• ensuring patch cables are of suitable length, are free from entanglement, are
patched to the appropriate port based on applicable PBGC standards, and are
routed neatly to the patch panel
• ensuring power to redundant power supplies is properly distributed between
PDUs to minimize downtime due to a single component failure
• maintaining Visio diagrams of the contents of each rack with
updates/verification no less frequently than every 6 months
• ensuring equipment not in use is powered down, clearly labeled, and
removed from the rack within 90 days of decommission
DC-WS-12 Contractor shall ensure all Windows accounts supporting the Windows server
environment, e.g. local Windows administrator; Windows service accounts; etc., are
changed periodically in accordance with PBGC policy and procedures and the
passwords are stored for emergency use
DC-WS-13 Contractor shall establish and maintain a periodic task list to monitor and proactively
maintain the Windows servers and the associated services
41
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
6.3.2 Windows Desktop Administration and Support, Software Packaging, and Software
Deployment
6.3.2.1 Scope of Services Supported
Provide tier 2 support for incidents relating to Windows Desktop Administration and Support, Software
Packaging, and Software Deployment or incidents requiring administrative access to Windows desktops
and laptops that cannot be resolved by an End-User services team as well as support service requests.
Typical Windows Desktop Administration and Support, Software Packaging, and Software Deployment
activities include:
• Monitor, troubleshoot, and address issues pertaining to the collective set of Windows enterprise
desktop operating systems including event log analysis, reconfiguration, etc.
• Package and deploy application software including office automation, productivity tools,
infrastructure software, and business application software as needed on Windows desktops,
laptops, and general purpose remote desktop services servers
• Create and distribute shortcuts for IT supported web-based applications
• Maintain up-to-date physical and virtual Windows enterprise desktop operating system
images/templates
• Monitor, troubleshoot, and repair software deployments
• Administer and support software packaging and software deployment tools
• Create and modify Active Directory group policy objects as required to ensure compliance with
Windows enterprise desktop operating system baselines as well as to achieve desired
operational configuration and user experience, e.g. adding trusted sites
• Deploy patches monthly to Windows desktops and laptops utilizing patch deployment tool, e.g.
BigFix
• Address operating system and software vulnerabilities detected on Windows workstations
during monthly vulnerability scans
• Escalate to and work collaboratively with 3rd party vendors on hardware and software issues
• Establish and maintain work instructions
See Appendix G - IT Service Support Guidelines for impact, urgency, and prioritization guidelines
associated with IT service and support. See Appendix J - IT Service and Support 2018 Statistical
Summary for tier 2 incidents, requests for information (RFIs), service/access requests, requests for
change (RFCs), and RFC tasks processed in calendar year 2018 for these services. See Appendix K – IT
Infrastructure Tools List for the software utilized to provides these services.
PBGC’s Windows Desktop Administration and Support, Software Packaging, and Software Deployment
environment consists of many components detailed in the tables that follow:
PBGC has approximately 3,087 Windows enterprise workstations, all (100%) of them running Windows
10 and more than 83% being physical. The following table breaks them down by OS, physical/virtual and
environment:
42
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Operating System Physical/Virtual Production Development Test Total
Count
Windows 10 Physical 2,504 15 51 2,570
Windows 10 Virtual 234 277 6 517
Total Count 2,738 292 57 3,087
PBGC has been moving towards portable computers and reduction of users with two physical devices.
This includes the recent replacement of physical workstations dedicated to application development and
testing with virtual desktops. Virtual desktops have also been deployed to support off-site actuaries
without government furnished equipment (GFE).
PBGC has approximately 2,570 physical Windows workstations, of which about 49% are portable devices
which are concentrated at PBGC’s HQ campus. Currently, PBGC is deploying one of three models based
on user location and user preference: MS Surface Pro 4 (HQ campus default), Dell Latitude E7450 (for
mobile power users who prefer it to SP4), and Dell Precision T1700 (for FBA sites, shared spaces, and
users who don’t want a SP4). These three models make up more than 93% of deployed inventory.
PBGC is currently evaluating the Dell Latitude 5290 2-in-1 as the successor to the SP4. PBGC is
considering deployment of laptops/tablets at its remote sites and eliminating use of desktops in
conference rooms. The following table breaks down all PBGC physical workstations by model and
operating system:
PBGC maintains a pool of physical Windows remote access (terminal) servers at both the headquarters
and disaster recovery sites which can provide PBGC users with access to the majority of PBGC
applications remotely during normal operating conditions (in support of telework), during a pandemic,
or should PBGC have a need to operate out of our disaster recovery site as follows:
43
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
HQW 5
WIL (COOP) 7
These servers are currently running Windows 2008 R2 but are scheduled for hardware upgrades (BL460
G10) and operating system modernization (to Windows 2016 or 2019) in calendar year 2018.
The following table provides the count for EnableIT service requests processed in Calendar Year 2018
6.3.2.2 Requirements
The Contractor shall provide incident response and resolution for issues relating to Windows Desktop
Administration and Support, Software Packaging, and Software Deployment or incidents requiring
administrative access to Windows desktops and laptops that cannot be resolved by an End-User services
team as well as support service requests and requests for change.
The contractor shall provide the Windows desktop imaging and software deployment support services
outlined in the following table:
Reference Requirement
(DC-WD-xx)
DC-WD-01 Contractor shall identify a lead for the Windows Desktop Administration and Support,
Software Packaging, and Software Deployment area. This lead is required to serve as
the primary point of contact for all Windows desktop imaging and software packaging
and deployment related issues.
DC-WD-02 Contractor shall provide and maintain a fully functional, optimally performing, secure
Windows Desktop and Software Packaging and Deployment infrastructure in all PBGC
computing environments. This shall include, but is not limited to:
• Monitor, troubleshoot, and address issues pertaining to the collective set of Enterprise Identity
and Privileged Account Management tools, scripts, and databases
• Configure Enterprise Identity and Privileged Account Management tools, scripts, and databases
to ensure accounts and access are managed in compliance with PBGC policies and procedures
• Configure new application security access roles in enterprise identity management tool(s) to
support request, approval, and automated fulfillment
• Update existing application security access roles in enterprise identity management tool(s) that
support request, approval, and automated fulfillment to address changing requirements as
identified
• Conduct annual certification/validation of existing application security access roles in enterprise
identity management tool(s) that support request, approval, and automated fulfillment
• Conduct annual account and access recertification for accounts and access managed by PBGC’s
suite of enterprise identity management tools
• Review reports and alerts to ensure all access has been properly authorized for accounts and
access managed by PBGC’s suite of enterprise identity management tools and revoke and create
security event upon detection of unauthorized access
• Federate new cloud-based systems with existing on-premise identity stores
• Establish, monitor, and maintain connectivity between multiple PBGC systems, e.g. HR,
procurements, etc.
• Escalate to and work collaboratively with 3rd party vendors on hardware and software issues
• Establish and maintain work instructions
See Appendix G - IT Service Support Guidelines for impact, urgency, and prioritization guidelines
associated with IT service and support. See Appendix J - IT Service and Support 2018 Statistical
46
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Summary for tier 2 incidents, requests for information (RFIs), service/access requests, requests for
change (RFCs), and RFC tasks processed in calendar year 2018 for these services. See Appendix K – IT
Infrastructure Tools List for the software utilized to provides these services.
PBGC’s Enterprise Identity Management environment consists of many components detailed in the
tables that follow:
The following table details the current number of Active Directory user accounts in PBGC’s primary
production user and resource domain (prod.ent.pbgc.gov) broken down by function:
The following table details the current number of user accounts (outside of production AD) in PBGC’s
production environment broken down by platform and type:
Service Account 14 14
Individual - AP User 27 27
Individual - Regular 1 1
User
Service Account 2 40 42
47
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Database - Oracle Individual - Regular 232 232
User
Service Account 35 20 55
PBGC is actively working to standardize its identity and access management on Active Directory and to
reduce dependencies on OAM and OID. Where Active Directory cannot be used, SailPoint will be
configured to manage accounts and access.
PBGC has implemented a Privileged Account Management tool, CyberArk , to provide central access
control for privileged access to PBGC systems. This includes storing privileged credentials and brokering
and recording sessions requiring privileged access. Currently, CyberArk is used to broker privileged
sessions to Windows, RHEL, Unix, Cisco, and Security devices using Service Accounts or Generic –
Recovery accounts instead of Individual – Elevated Accounts. The use of CyberArk to manage privileged
sessions will be expanded to include other platforms during calendar year 2019.
PBGC is in the process of implementing the service request module of Service Now which will be used as
a front-end request platform for automated fulfillment of access requests using SailPoint LifeCycle
Manager. This is expected to be in place by September 2019.
48
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
6.3.3.2 Requirements
The Contractor shall provide incident response and resolution for issues relating to Enterprise Identity
Management administration and support as well as support service requests and requests for change.
The contractor shall provide the enterprise identity management support services outlined in the
following table:
Reference Requirement
(DC-EI-xx)
DC-EI-01 Contractor shall identify a lead for the Enterprise Identity Management area. This
lead is required to serve as the primary point of contact for all enterprise identity
management related issues.
DC-EI-02 Contractor shall provide and maintain a fully functional, optimally performing
enterprise identity management infrastructure to support all PBGC computing
environments. This shall include, but is not limited to:
• Provide support for the collective set of Enterprise Identity and Privileged
Account Management tools, scripts, and databases
49
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
DC-EI-05 Contractor shall fulfill all approved service requests for IT infrastructure resource and
services (in EnableIT) in accordance with PBGC procedures and timelines for the
following:
50
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
6.3.4 Office 365 and Messaging Administration and Support
6.3.4.1 Scope of Services Supported
Provide tier 2 support for incidents relating to PBGC’s Microsoft Office 365 E3 tenant including Intune
Mobile Device Management (MDM) and Advanced Threat Protection (ATP) as well as PBGC’s on-premise
IronPort email hygiene, relay, and data loss prevention appliances and associated services. Provide
fulfillment for service requests. Typical Office 365 and Messaging administration and support activities
include:
• Monitor, troubleshoot, and address issues pertaining to PBGC’s Microsoft Office 365 E3 tenant
including Intune Mobile Device Management (MDM) and Advanced Threat Protection (ATP) as
well as PBGC’s on-premise IronPort email hygiene, relay, and data loss prevention appliances
and associated services, maintenance tools, and scripts
• Configure PBGC’s Microsoft Office 365 E3 tenant including Intune Mobile Device Management
(MDM) and Advanced Threat Protection (ATP) as well as PBGC’s on-premise IronPort email
hygiene, relay, and data loss prevention appliances and associated services to maximize
availability, optimize performance, and ensure compliance with PBGC governance
• Maintain and administer PBGC’s Microsoft Office 365 E3 tenant including Intune Mobile Device
Management (MDM) and Advanced Threat Protection (ATP) including, but not limited to:
o Performing litigation holds and sensitive data collection requests
o Providing reports on message transport utilizing message tracking
o Updating hub transport rules
o Updating email retention policies
o Creating and removing shared mailboxes
o Delegating access to or transferring email or OneDrive contents as required
o Periodically running a script to copy all email content from separated and deprovisoned
users to a shared mailbox to preserve email according to PBGC’s record schedule and
remove license
o Troubleshooting/resolving identity synchronization issues with Azure AD
o Monitoring Microsoft’s Office 365 IP and URL list and coordinate associated PBGC
network updates to ensure connectivity
• Maintain and administer PBGC’s on-premise IronPort email hygiene, relay, and data loss
prevention appliances including, but not limited to:
o Updating blacklists and whitelists including adding and removing email domains and
individual email addresses as required
o Updating authorized relay list including adding and removing IPs
o Performing periodic device configuration back-ups no less than monthly
o Examining and acting on outbound messages quarantined by the DLP engine in
accordance with PBGC policy and procedures
• Enable and support email integration with SharePoint on-line and perform other business
process automation using Think Automation, e.g. automate upload to SharePoint on-line reports
distributed via email
51
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Enable and support email-enabled/integrated applications as well as MDAemon Mail Server for
application-based mail services that cannot leverage Office 365
• Escalate and work collaboratively with 3rd party vendors on issues
• Establish and maintain work instructions
See Appendix G - IT Service Support Guidelines for impact, urgency, and prioritization guidelines
associated with IT service and support. See Appendix J - IT Service and Support 2018 Statistical
Summary for tier 2 incidents, requests for information (RFIs), service/access requests, requests for
change (RFCs), and RFC tasks processed in calendar year 2018 for these services. See Appendix K – IT
Infrastructure Tools List for the software utilized to provides these services.
PBGC’s Office 365 and Messaging administration and support environment consists primarily of the
following:
• Microsoft Office 365 E3 tenant including Intune Mobile Device Management (MDM) and
Advanced Threat Protection (ATP). PBGC has 2 tenants, one for testing which is licensed for 10
users and on for production use which is licensed for 2,400 users. PBGC Active Directory users
and groups are synchronized to Azure AD utilizing Azure Active Directory Synchronization and
user authentication is federated utilizing Active Directory Federation Services (ADFS). PBGC has
approximately 500 distribution groups, more than 14,000 mail-enabled security groups primarily
for SharePoint access management, and 450 shared mailboxes in addition to its regular 2,250
mailbox-enabled users.
• PBGC maintains on-premise IronPort appliance for email hygiene, relay, and data loss
prevention. Mail is routed between the internet and PBGC’s Microsoft Office 365 tenant
through the IronPort mail gateways located at PBGC’s headquarters and disaster recovery data
centers.
• PBGC maintains a single instance of the MDAemon Mail Server for application-based mail
services that cannot leverage Office 365. This solution hosts approximately 10 application IMAP
mailboxes.
6.3.4.2 Requirements
The contractor shall provide the Office 365 and messaging support services outlined in the following
table:
Reference Requirement
(DC-MS-xx)
DC-MS-01 Contractor shall identify a lead for the Office 365 and Messaging area. This lead is
required to serve as the primary point of contact for all Office 365 and messaging
related issues.
DC-MS-02 Contractor shall support and maintain a fully functional, optimally performing on-
premise messaging infrastructure, e.g. mail hygiene appliances, and infrastructure
services that support connectivity and integration with the cloud-based Office 365.
This shall include, but is not limited to:
52
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Identifying and addressing performance bottlenecks
• Using monitoring tools and Microsoft-provided health reports to proactively
plan and manage infrastructure resources to maximize system and service
availability
• Performing preventative and remedial maintenance of components
• Coordinating performance of work by vendors as required and in accordance
with PBGC Security policies, vendor warranties and maintenance contracts
DC-MS-03 Contractor shall fully implement, manage, and support all incident management
activities regarding Office 365 and Messaging services according to PBGC policies. By
using ITIL best practices, this should include, but shall not be limited to:
• Provide support for Microsoft Office 365 E3 tenant including Intune Mobile
Device Management (MDM) and Advanced Threat Protection (ATP)
• Provide support for PBGC’s on-premise IronPort email hygiene, relay, and
data loss prevention appliances
53
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Updating hub transport rules
• Updating email retention policies
• Creating and removing shared mailboxes
• Delegating access to or transferring email or OneDrive contents as required
• Periodically run a script to copy all email content from separated and
deprovisoned users to a shared mailbox to preserve email according to
PBGC’s record schedule and remove license
• Troubleshoot/resolve identity synchronization issues with Azure AD
• Monitor Microsoft’s Office 365 IP and URL list and coordinate associated
PBGC network updates to ensure connectivity
DC-MS-07 Contractor shall maintain and administer PBGC’s on-premise IronPort email hygiene,
relay, and data loss prevention appliances including, but not limited to:
• Updating blacklists and whitelists including adding and removing email
domains and individual email addresses as required
• Updating authorized relay list including adding and removing IPs
• Perform periodic device configuration backups no less than monthly
• Examine and act on outbound messages quarantined by the DLP engine in
accordance with PBGC policy and procedures
DC-MS-08 Contractor shall enable and support email integration with SharePoint on-line and
perform other business process automation using Think Automation
DC-MS-09 Contractor shall enable and support email-enabled/integrated applications as well as
MDAemon Mail Server for application-based mail services that cannot leverage Office
365
DC-MS-10 Contractor shall maintain passwords for Office 365 and messaging privileged
accounts, e.g. tenant accounts, local system emergency recovery accounts, privileged
service accounts, etc. in privileged account management tool and utilize this tool to
perform administrative functions via brokered session or account check out
DC-MS-11 Contractor shall establish and maintain a periodic task list to monitor and proactively
maintain the Microsoft Office 365 E3 tenant including Intune Mobile Device
Management (MDM) and Advanced Threat Protection (ATP) as well as PBGC’s on-
premise IronPort email hygiene, relay, and data loss prevention appliances
54
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
6.3.5 UNIX/LINUX Administration and Support
6.3.5.1 Scope of Services Supported
Provide tier 2 support for incidents relating to UNIX/LINUX servers or requiring administrative access to
Windows servers that cannot be resolved by an End-User services team as well as support service
requests. Typical UNIX/LINUX server administration and support activities include:
• Monitor, troubleshoot and repair or replace defective IT equipment (stand-alone servers, blade
infrastructure and servers, and associated components) as needed
• Monitor, troubleshoot, and repair UNIX/LINUX Server operating systems including space
cleanup, event log analysis, system daemon and package reconfiguration and reinstall, operating
system re-image when needed, etc.
• Troubleshoot and reinstall software including operating system daemons, RPM packages,
infrastructure software and business application software as needed on UNIX/LINUX servers,
e.g. SSSD, SFTP, Splunk, SAMBA, SAS, Oracle Web Logic middleware, Oracle e business suite, etc.
• Maintain up-to-date physical and virtual UNIX/LINUX Server operating system images/templates
• Provision and configure new physical and virtual servers as requested including SAN connectivity
• Perform initial installation and configuration of operating system daemons, RPM packages,
infrastructure software and business application software as needed on UNIX/LINUX servers
• Remove/decommission physical and virtual servers as requested
• Perform Active Directory account administration for privileged and service accounts supporting
the UNIX/LINUX environment using Quest ActiveRoles Server
• Deploy patches monthly to UNIX/LINUX servers utilizing patch deployment tool, e.g. BigFix
• Address operating system and software vulnerabilities detected on UNIX/LINUX servers during
monthly vulnerability scans
• Configure and administer UNIX/LINUX file systems and file shares
• Install and configure SFTP services
• Develop and maintain custom shell scripts to automate routine activities and support file
transfers
• Schedule automated tasks utilizing Cron
• Configure, monitor, troubleshoot, and repair high availability and disaster recovery capabilities
pertaining to UNIX/LINUX Servers and associated services including, but not limited to Veritas
clustering, failover procedures, etc.
• Support internal and external IP address management and name resolution services by updating
IPAM DNS records
• Maintain server racks
• Establish and maintain work instructions
• Escalate and work collaboratively with 3rd party vendors on hardware and software issues
Support for UNIX/LINUX Servers and IT equipment in the PBGC data centers is typically handled using
remote management software and protocols, e.g. vCenter console, SSH, Powershell, ILO, OA, etc., but
does occasionally require physical visits to address certain hardware and software problems. See
55
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix F - PBGC Locations for PBGC locations, including planned changes to PBGC’s facilities and data
centers over the life of the contract. See Appendix G - IT Service Support Guidelines for impact,
urgency, and prioritization guidelines associated with IT service and support. See Appendix J - IT Service
and Support 2018 Statistical Summary for tier 2 incidents, requests for information (RFIs), service/access
requests, requests for change (RFCs), and RFC tasks processed in calendar year 2018 for these services.
See Appendix K – IT Infrastructure Tools List for the software utilized to provides these services.
PBGC’s UNIX/LINUX Server environment consists of many components detailed in the tables that follow:
PBGC has approximately 361 RHEL servers, with more than 85% of them being virtual. The following
table breaks them down by physical/virtual and environment:
PBGC has approximately 44 physical RHEL servers, of which almost all are HP Proliant servers and the
majority of which are HP Proliant blade servers. The following table breaks them down by model and
operating system:
56
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
ProLiant DL360 G6 HP 1 1
ProLiant DL360 G8 HP 3 3
ProLiant DL360 G9 HP 3 3
ProLiant DL380 G5 HP 2 2
ProLiant DL380 G8 HP 1 1
Total Count 29 15 44
An upgrade from HP Proliant Generation 5, 6 and 7 servers to HP Proliant Generation 10 servers is
planned for FY19 as part of the RHEL 7 and Oracle Fusion Middleware upgrades.
PBGC currently has 3 physical domains on 3 physical Oracle Solaris servers. These servers are all running
Oracle Solaris 11 (5.11):
The following table provides the count for EnableIT service requests processed in Calendar Year 2018 for
LINUX/UNIX servers:
6.3.5.2 Requirements
The Contractor shall provide incident response and resolution for issues relating to UNIX/LINUX servers
or requiring administrative access to UNIX/LINUX servers that cannot be resolved by an End-User
services team as well as support service requests and requests for change.
The contractor shall provide the UNIX and LINUX support services outlined in the following table:
Reference Requirement
(DC-UX-xx)
DC-UX-01 Contractor shall identify a lead for the UNIX/LINUX area. This lead is required to serve
as the primary point of contact for all UNIX and LINUX related issues.
57
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
DC-UX-02 Contractor shall provide and maintain a fully functional, optimally performing, secure
UNIX/LINUX Server infrastructure in all PBGC computing environments. This shall
include, but is not limited to:
• Monitor, troubleshoot, and repair Virtualization Platform server operating systems, clusters,
datastores, etc. including space cleanup, event log analysis, feature reconfiguration and
reinstall, operating system re-image when needed, etc.
• Provision and configure new physical servers and load and configure hypervisor software in
compliance with PBC security baselines as requested including SAN connectivity
• Remove/decommission physical servers providing virtualization services as requested
• Deploy patches no less than quarterly to virtualization platform utilizing virtualization platform
patch deployment tool
• Address operating system and software vulnerabilities detected on PBGC’s Virtualization
Platform(s) during monthly vulnerability scans
• Configure and administer virtualization platform data stores
• Migrate physical servers to virtual servers when technically feasible
• Configure, monitor, troubleshoot, and repair high availability and disaster recovery capabilities
pertaining to PBGC’s virtualization platforms and associated services including, but not limited
to VMware clustering, VMware high availability, VMware Distributed Resource Scheduler (DRS),
etc.
• Establish and maintain work instructions
• Escalate and work collaboratively with 3rd party vendors on hardware and software issues
Support for PBGC’s Virtualization Platform(s) and the associated IT equipment in the PBGC data centers
is typically handled using remote management software, e.g. vCenter console, SSH, Powershell, ILO, OA,
60
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
etc., but does occasionally require a physical visit to address certain hardware and software problems.
See Appendix F - PBGC Locations for PBGC locations, including planned changes to PBGC’s facilities and
data centers over the life of the contract. See Appendix G - IT Service Support Guidelines for impact,
urgency, and prioritization guidelines associated with IT service and support. See Appendix J - IT Service
and Support 2018 Statistical Summary for tier 2 incidents, requests for information (RFIs), service/access
requests, requests for change (RFCs), and RFC tasks processed in calendar year 2018 for these services.
See Appendix K – IT Infrastructure Tools List for the software utilized to provides these services.
PBGC currently has 18 VMware clusters supporting more than 750 virtual servers (Windows and RHEL),
several virtual appliances, and more than 500 virtual Windows enterprise desktops. The following table
summarizes their function and notes the number of ESXi hosts comprising each cluster as well as the
number of virtual machines, datastores, and networks it supports:
61
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
production virtual
machines that run
expensive software
licensed by core
processor count to
control costs, e.g.
Oracle Web Center;
slated for
decommission
HQWPROD_RHEL Primary cluster to host 5 74 10 13
production RHEL
virtual servers
HQWPROD_RHEL_12c Primary cluster to host 4 0 5 14
production RHEL
virtual servers
supporting 12c
WebLogic middleware
HQWPROD_VDI Primary cluster to host 6 236 10 8
production VDI for
Windows enterprise
desktops supporting
off-site staff, e.g.
actuaries, IT Service
Desk, etc.
HQWPROD_Windows Primary cluster to host 10 152 20 13
production Windows
virtual servers
HQWTCO_ITCVDI Small cluster to host 2 10 4 16
VDI for Windows
enterprise desktops
supporting user
acceptance testing in
the ITC which will
ultimately replace a
physical lab consisting
of ~75 desktops
HQWTCO_LowCore Small cluster 2 9 4 15
comprised of ESXi
servers with a small
number of processor
cores to host
development and test
virtual machines that
run expensive
62
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
software licensed by
core processor count
to control costs, e.g.
Oracle Web Center
HQWTCO_RHEL Primary cluster to host 5 145 12 15
Development and Test
RHEL virtual servers
HQWTCO_RHEL-12c Primary cluster to host 3 26 12 15
Development and Test
RHEL virtual servers
supporting 12c
WebLogic middleware
HQWTCO_RHEL-BI Primary cluster to host 3 3 6 15
Development and Test
RHEL virtual servers
supporting BI
HQWTCO_VDI Primary cluster to host 9 279 11 15
Development and Test
VDI for Windows
enterprise desktops
primarily supporting
application
development and
testing
HQWTCO_Windows Primary cluster to host 10 296 26 21
Development and Test
Windows virtual
servers
WIL_MGMT Small cluster to host 2 22 4 7
vCenter, its SQL
database server, and a
few other virtual
servers that are
essential in the event
PBGC must operate
from its DR site
WILPROD_RHEL Primary cluster to host 4 41 10 11
RHEL virtual servers
used in the event
PBGC must operate
from its DR site
WILPROD_Windows Primary cluster to host 3 50 10 11
Windows virtual
servers used in the
63
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
event PBGC must
operate from its DR
site
WILPROD_CiscoPhone Cisco proprietary 5 14 16 1
cluster used to host
Unified
Communications and
UCCX servers running
on BE7H-M5-K9
hardware at DR site
Total Counts 82 1,389
PBGC has approximately 79 physical VMware ESXi servers supporting 12 VMware clusters of which are
all are HP Proliant blade servers. The following table breaks them down by model and cluster:
64
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
6.3.6.2 Requirements
The Contractor shall provide incident response and resolution for issues relating to PBGC’s Virtualization
Platform(s) or requiring administrative access to PBGC’s Virtualization Platform(s) that cannot be
resolved by an End-User services team as well as support service requests and requests for change. The
contractor shall provide the Virtualization Platforms administration and support services outlined in the
following table:
Reference Requirement
(DC-VM-xx)
DC-VM-01 Contractor shall identify a lead for the Virtualization area. This lead is required to
serve as the primary point of contact for all virtualization related issues.
DC-VM-02 Contractor shall provide and maintain a fully functional, optimally performing, secure
Virtualization Platform infrastructure in all PBGC computing environments. This shall
include, but is not limited to:
• Provision and configure new physical servers and load and configure
hypervisor software in compliance with PBC security baselines as requested
including SAN connectivity
• Provide installation and configuration of virtualization platform management
tools, e.g. vCenter, Horizon View, etc.
• Provision resources to virtual servers and workstations as required to ensure
optimal operation including CPUs, memory, disk drives/space, etc.
• Apply applicable security patches at least quarterly
65
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Address operating system and software vulnerabilities detected on PBGC’s
Virtualization Platform during monthly vulnerability scans
• Apply necessary configuration changes to ensure compliance with PBGC
security baselines and industry best practices
DC-VM-09 Contractor shall ensure all accounts supporting the Virtualization Platform
environment, e.g. local VMware ESXi administrator; Active Directory service accounts;
etc., are changed periodically in accordance with PBGC policy and procedures and the
passwords are stored for emergency use
DC-VM-10 Contractor shall establish and maintain a periodic task list to monitor and proactively
maintain PBGC’s Virtual Platform(s)
Support for storage and backup equipment in the PBGC data centers is typically handled using remote
management software and protocols, e.g. https, SSH, Powershell, ILO, OA, etc., but does occasionally
require physical visits to address certain hardware and software problems. See Appendix F - PBGC
Locations for PBGC locations, including planned changes to PBGC’s facilities and data centers over the
life of the contract. See Appendix G - IT Service Support Guidelines for impact, urgency, and
prioritization guidelines associated with IT service and support. See Appendix J - IT Service and Support
2018 Statistical Summary for tier 2 incidents, requests for information (RFIs), service/access requests,
requests for change (RFCs), and RFC tasks processed in calendar year 2018 for these services. See
Appendix K – IT Infrastructure Tools List for the software utilized to provides these services.
PBGC’s current storage infrastructure primarily consists of Brocade DCX8510-4 (Primary data center) and
6520 (DR data center) fibre channel switches and Hitachi Virtual Storage (HVS) platform G-900 (HQ) and
G-700 (COOP) series storage arrays. PBGC is currently migrating data and services from Hitachi HUS and
NetApp Fabric-Attached Storage (FAS) arrays to the VSP platform and is largely done. The total usable
capacity of PBGC’s VSP SAN arrays is over one exabyte and considering all PBGC SAN arrays, capacity
currently amounts to more than 2 exabytes broken down as follows as of March 2019:
67
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Production VMware VDI cluster datastores –
24 TBs
• Dev/Test VMware VDI cluster datastores – 36
TBs
• Production MS SQL Server storage – 9 TBs
• Production Windows Server storage – 12 TBs
• Dev/Test Windows Server storage – 0 TBs
• Production LINUX Server storage – 2 TBs
• Dev/Test LINUX Server storage – 0 TBs
Hitachi Virtual 355 109 Supports Secondary (DR/COOP) Data Center
Storage Platform equipment/services:
(VSP) G700 • COOP UNIX/Oracle/flashback – 0 TBs –
migration is in progress
• COOP VMware server cluster datastores – 44
TBs
• COOP MS SQL Server storage – 0 TBs
• COOP Windows Server storage – 38 TBs
• COOP LINUX Server storage – 27 TBs
Hitachi Unified 130 13.5 Supports lower-tier, legacy storage needs at HQ:
Storage (HUS) • Windows server storage – 3 TBs
150 • Legacy Backup Catalog – 10.5 TBs
Hitachi Unified 130 3 Supports lower-tier, legacy storage needs at DR site:
Storage (HUS) • Windows server storage – 3 TBs
150
Hitachi Unified 464 0 Supported higher-tier storage at HQ
Storage (HUS) VM
Hitachi Unified 93 19 Supports higher-tier, legacy storage needs at DR site:
Storage (HUS) VM • DR UNIX/Oracle – 19 TBs
NetApp FAS8040 295 115 Supports higher-tier, legacy storage needs at HQ:
• Production file shares – 38 TBs
• Production IPS file system – 28 TBs
• Production NFS and CIFS shares – 9 TBs
• Dev/Test file shares – 20 TBs
• Dev/Test NFS and CIFS shares – 17 TBs
• NetApp – 3 TBs
Migration to G900 is in progress
NetApp FAS2554 80 63 Supports higher-tier, legacy storage needs at DR site:
• DR file shares replicated from HQ – 34 TBs
• DR IPS file system – 28 TBs
• DR NFS and CIFS shares – 1 TB
Migration to G700 is in progress
68
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Veritas NetBackup software is predominantly used to support backup and restore operations. A master
server controls backup of all production data at HQ and remote locations. All non-production data is
backed up to a pure-disk based Veritas NetBackup 5230 appliance. All production data is backed up to a
3 pure-disk based Veritas NetBackup 5240 appliances (1 masters, 2 media) appliances and replicated to
a DR site to a mirror set of appliances using NetBackup’s Auto Image Replication (AIR). PBGC plans to
move some of its older backup data to Azure storage in the next year. In the interim, PBGC may make
use of its SAN arrays to house this data if it exceeds are capacity. Backup data from the development
environments are retained for 90 days, while the agency’s retention policies for production data require
that data be kept for a maximum of seven years. The agency has a continuing requirement to store tape
backup media on-site at HQ until 2025 to support data restore requests. The legacy backup medium is
LTO tapes which are bar-coded before being loaded into IBM TS3310 tape library, but these are rarely if
ever used.
The following table provides the count for EnableIT service requests processed in Calendar Year 2018
6.3.7.2 Requirements
The contractor shall provide the storage and backup infrastructure support services outlined in the
following table:
Reference Requirement
(DC-SB-xx)
DC-SB-01 Contractor shall identify a lead for the storage and backup area. This lead is required
to serve as the primary point of contact for all storage and backup related issues.
DC-SB-02 Contractor shall provide and maintain a fully functional, optimally performing, secure
Storage and Backup infrastructure in all PBGC computing environments. This shall
include, but is not limited to:
69
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
DC-SB-03 Contractor shall fully implement, manage, and support all incident management
activities related to Storage and Backup infrastructure and associated services
according to PBGC policies and procedures. By using ITIL best practices, this should
include, but shall not be limited to:
• Troubleshoot storage access issues
• Monitor faults, performance, and capacity issues
• Identify, troubleshoot, and resolve backup/recovery issues
• Diagnose, solve and provide root cause analysis for specialized storage and
backup hardware and software related issues
Support for databases in the PBGC data centers is typically handled using remote management software
and protocols, e.g. https, SSH, Powershell, ILO, OA, etc., but does occasionally require physical visits to
address certain hardware and software problems. See Appendix F - PBGC Locations for PBGC locations,
including planned changes to PBGC’s facilities and data centers over the life of the contract. See
Appendix G - IT Service Support Guidelines for impact, urgency, and prioritization guidelines associated
with IT service and support. See Appendix J - IT Service and Support 2018 Statistical Summary for tier 2
incidents, requests for information (RFIs), service/access requests, requests for change (RFCs), and RFC
tasks processed in calendar year 2018 for these services. See Appendix K - IT Infrastructure Tools List for
the software utilized to provides these services.
PBGC’s current enterprise relational database infrastructure is heterogeneous in nature and primarily
consists of Oracle Relational Database Management Systems (RDBMS), and Microsoft SQL Server.
Historically PBGC predominantly used Oracle RDBMS, but more recently has been moving towards
Microsoft SQL Server for its relational database system and PBGC is considering use of PostgreSQL in the
near future. Being able to provide any needed PostgreSQL support is a requirement of this contract.
Data masking is performed utilizing Dataguise.
The following table provides a recent summary of PBGC’s enterprise relational database inventory
spread across approximately 29 Oracle servers and 31 Microsoft SQL servers:
72
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Test: 18 Test: 4,100 GBs
Prod: 21 Prod: 5,028 GBs
COOP: 10 COOP: 3,500 GBs
Total: 97 Total: 28,828 GBs
Microsoft SQL Server 2012 Dev:17 Dev: 148 .07 GBs
Test: 2 Test: 24.93 GBs
Prod:10 Prod: 32.72 GBs
COOP: 2 COOP: 31.60 GBs
Total: 31 Total: 237.32 GBs
Microsoft SQL Server 2014 Dev: 13 Dev: 64.25 GBs
Test: 13 Test: 69.97 GBs
Prod: 12 Prod: 29.04 GBs
COOP: None COOP: 0 GBs
Total: 38 Total: 163.26 GBs
Microsoft SQL Server 2016 Dev: 26 Dev: 126.40 GBs
Test: 10 Test: 30.21 GBs
Prod: 21 Prod: 291.81 GBs
COOP: 1 COOP: 8.39 GB
Total: 58 Total: 492.81GBs
Microsoft SQL Server 2017 Dev: 66 Dev: 1138.31GBs
Test: 17 Test: 203.74 GBs
Prod: 35 Prod: 587.29 GBs
COOP: 2 COOP: 18.55 GBs
Total: 120 Total: 1,947.89 GBs
PostgreSQL Planned; none Planned; none presently
presently
The following table provides the count for EnableIT service requests processed in Calendar Year 2018:
6.3.8.2 Requirements
For each database platform referred to above, the contractor shall provide the database management
support services outlined in the following table:
73
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Reference Requirement
(DC-DB-xx)
DC-DB-01 Contractor shall identify a lead for the Database Management area. This lead is
required to serve as the primary point of contact for all relational database
management related issues.
DC-DB-02 Contractor shall provide and maintain a fully functional, optimally performing, secure
enterprise relational Database infrastructure in all PBGC computing environments.
This shall include, but is not limited to:
• New Database
• Database Fixes (updates)
• Database Refresh
• Database Restore
• New Database Role
75
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
DC-DB-12 Contractor shall copy data between environments as required to support application
development and testing and while doing so, contractor will enforce controls to
protect sensitive data per PBGC policies and procedures including data masking
DC-DB-13 Contractor shall participate in planning for the releases of new COTS/GOTS
applications to ensure that any new product usage or release upgrade takes place
with minimal impact
DC-DB-14 Contractor shall deploy database updates as required for the release of new
COTS/GOTS applications to ensure that any new product usage or release upgrade
takes place with minimal impact
DC-DB-15 Contractor shall monitor and improve capacity utilization by decommissioning unused
databases
DC-DB-16 Contractor shall establish and maintain documentation regarding the enterprise
database management environment including:
• An inventory of all databases broken down by environment, server, and
database version
DC-DB-17 Contractor shall establish and maintain a periodic task list to monitor and proactively
maintain the enterprise relational Database infrastructure and the associated services
DC-DB-18 Contractor shall install, configure and support a PostgreSQL relational database
management system including support for high availability, backup and recovery, and
disaster recovery. Contractor shall establish and maintain work instructions to
support this database platform.
Typical enterprise Web and Application Middleware administration and support activities include:
76
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Address software vulnerabilities detected on enterprise infrastructure during monthly
vulnerability scans on Web and Application Middleware
• Apply necessary configuration changes to ensure compliance with PBGC security baselines and
industry best practices
• Identify and resolve system errors and assist at troubleshooting application program errors
• Deploy and integrate new applications, troubleshoot problems, and install upgrades
• Prepare Web and Application Middleware infrastructure environment(s) for application
deployments
• Develop and/or maintain up-to-date documents showing the inventory and configuration of all
Web and Application Middleware infrastructure
• Performance tuning, capacity planning, and architecture planning
• Provide technical support to application services teams on matters related to the Web and
Application Middleware infrastructure
• Automate Web and Application Middleware infrastructure management and backup tasks
• Configure, monitor, troubleshoot, and repair high availability and disaster recovery capabilities
pertaining to enterprise Web and Application Middleware infrastructure systems and associated
services including, but not limited to clustering, network load balancing, failover procedures,
etc.
• Establish and maintain work instructions
• Escalate and work collaboratively with 3rd party vendors on software issues
Support for web and application middleware running in the PBGC data centers is typically handled using
remote management software and protocols, e.g. https, SSH, Powershell, ILO, OA, etc., but does
occasionally require physical visits to address certain hardware and software problems. See Appendix F
- PBGC Locations for PBGC locations, including planned changes to PBGC’s facilities and data centers
over the life of the contract. See Appendix G - IT Service Support Guidelines for impact, urgency, and
prioritization guidelines associated with IT service and support. See Appendix J - IT Service and Support
2018 Statistical Summary for tier 2 incidents, requests for information (RFIs), service/access requests,
requests for change (RFCs), and RFC tasks processed in calendar year 2018 for these services. See
Appendix K - IT Infrastructure Tools List for the software utilized to provides these services.
PBGC’s current enterprise Web and Application Middleware infrastructure is heterogeneous in nature
and primarily consists of Oracle WebLogic, Oracle SOA suite, Oracle BPEL, Oracle forms and reports,
Microsoft IIS and .Net software, and PBGC’s Oracle E Business Suite applications including CRM and
CFS/PPS Oracle. PBGC is exploring use of Microsoft Azure and Dynamics for many of its business
functions in the future. PBGC may also consider Apache Tomcat for its web and middleware platform.
The following table provides a summary of PBGC’s web application middleware platform instances as of
March 2019:
77
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Oracle WebLogic vers. 11g, 12c 494 165 127 93 879
Microsoft IIS 7.5 14 7 18 3 42
Microsoft IIS 8.5 41 22 42 9 114
Apache 68 24 16 108
Tomcat 5 1 2 3
Tomcat 6 2 4 6
Tomcat 7 12 10 12 4 38
Tomcat 8 16 6 9 1 32
Tomcat (Other) 3 2 5 10
JBoss 3 3
NGINX Web Server 1.12 1 1
Grand Total 648 239 239 110 1236
6.3.9.1 Requirements
For each web and middleware platform noted above, the contractor shall provide the support services
outlined in the following table:
Reference Requirement
(DC-WM-xx)
DC-WM-01 Contractor shall identify a lead for the Web and Application Middleware area. This
lead is required to serve as the primary point of contact for all web and middleware
related issues.
DC-WM-02 Contractor shall provide and maintain a fully functional, optimally performing, secure
enterprise Web and Application Middleware infrastructure in all PBGC computing
environments. This shall include, but is not limited to:
78
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Provide support for IT supported applications and services deployed to
PBGC’s enterprise Web and Application Middleware infrastructure systems
including, but not limited to Oracle WebLogic, Oracle SOA suite, Oracle BPEL,
Oracle forms and reports, Microsoft IIS and .Net software, Apache Tomcat,
PBGC’s Oracle E Business Suite applications including CRM and CFS/PPS
(Financials), etc.
79
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
DC-WM-09 Contractor shall perform application deployments from controlled sources, e.g. PVCS
per PBGC policies and procedures
DC-WM-10 Contractor shall participate in planning for the releases of new COTS/GOTS
applications to ensure that any new product usage or release upgrade takes place
with minimal impact
DC-WM-11 Contractor shall establish and maintain documentation regarding the enterprise Web
and Application Middleware infrastructure environment including:
• An inventory of all Web and Application Middleware infrastructure software
installations/instances broken down by environment, server, and software
version
DC-WM-12 Contractor shall establish and maintain a periodic task list to monitor and proactively
maintain the Web and Application Middleware infrastructure environment and the
associated services
DC-WM-13 Contractor shall install, configure and support an Apache Tomcat web and middle
infrastructure including support for high availability, backup and recovery, and
disaster recovery. Contractor shall establish and maintain work instructions to
support this platform.
The Voice, Video and Network Infrastructure equipment, e.g. phones, routers switches, firewalls, etc.,
are located at PBGC’s HQW Campus as well as Field Benefit Administration FBA) locations. The on-
premise IT equipment that supports PBGC’s disaster recovery capability is currently located at PBGC’s
Wilmington (WIL) facility. PBGC plans to move most of the on-premise IT equipment supporting the
Data Center services to co-located data centers over the next several years. The co-located data centers
and the current and future HQ campuses will be connected via 10 GB Wave backbone replacing the
80
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
existing 1 GB backbone. Please refer to Appendix F - PBGC Locations for a tentative timeline for this
transition.
• Troubleshoot customer network connectivity issues (LAN, WAN, VPN, and Wireless)
• Support internal and external IP address management and name resolution services by creating,
updating, and removing IP address management (IPAM) DNS records
• Establish and maintain work instructions
• Escalate and work collaboratively with 3rd party vendors on hardware and software issues
• Escalate and work collaboratively with external network providers to resolve circuit
degradations and outages
• Maintain current IOS (internetwork Operating System) versions including startup and running
configurations for all network devices
• Manage internal/external hardware firewalls and secure web gateways.
Support for Network Infrastructure equipment is typically handled using remote management software,
e.g. SSH, but does occasionally require physical visits to address certain hardware and software
problems. See Appendix F - PBGC Locations for PBGC locations, including planned changes to PBGC’s
facilities and data centers over the life of the contract. See Appendix G - IT Service Support Guidelines
for impact, urgency, and prioritization guidelines associated with IT service and support. See Appendix J
- IT Service and Support 2018 Statistical Summary for tier 2 incidents, requests for information (RFIs),
service/access requests, requests for change (RFCs), and RFC tasks processed in calendar year 2018 for
these services. See Appendix K - IT Infrastructure Tools List for the software utilized to provides these
services. See Appendix D – PBGC Network Overview Diagram for more information on network
connectivity between sites and with the internet.
PBGC’s Network Infrastructure consists of many components detailed in the tables that follow:
The following table breaks down PBGC’s network infrastructure equipment by model:
81
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Cisco Catalyst 3850 Floor distribution Switches 2
Cisco Catalyst 3750 Floor distribution Switches 5
Cisco Catalyst 6509 Floor distribution Switches 1
Cisco Catalyst 6504 Floor distribution Switches 2
Cisco 4451-X router Vendor Managed Routers for L3/Presidio MPLS 8
WAN
Cisco ARS1002 Vendor Managed Routers for AT&T backbone 3
Cisco ARS1002 Vendor Managed Routers for Verizon TIC 2
Brocade Serverlron ADX Network load balancer 10
1000F
Infoblox Trinzic 1410 Infoblox IPAM Appliance - Grid Master and Grid 5
Master Candidates (DNS, DHCP, IPAM)
Infoblox Trinzic 810 Infoblox IPAM Appliance - DNS resolution and 4
forwarding in the DMZ
Infoblox Reporter 1400 Infoblox IPAM Appliance – Reporting 4
Cisco Wireless Controller Wireless Controller for Access points 3
5520
Cisco Wireless AP3800 Wireless Access points 109
Spectracom NetClock NTP Network Time Appliance 3
9843
PBGC is nearing completion of its effort to replace legacy floor distribution switches, e.g. 3750, 650x, in
the table above with Cisco 9410s and 23 x Cisco 9300s that will support PoE for VoIP. This work is
planned to be completed by May 2019. Network IT infrastructure equipment that supports remote
access or IT security functions is not included in the table above and will be made available in PBGC’s
reading room.
Wireless Summary
PBGC maintains several corporate Wi-Fi networks throughout its facilities (with end-users) administered
under this contract as follows:
Function Details
Corporate WiFi network for GFE computers Provides full access to PBGC’s production
network mimicking wired access. Access to
this network is controlled by Cisco ISE via AD
authentication.
WiFi network for GFE mobile phones Provides temporary access to the internet.
Access to this network is controlled by InTune
profile. Cisco Umbrella services is used to
restrict access to appropriate sites while on
this network.
Guest WiFi network Provides temporary access to the internet. A
PBGC user must contact the Service Desk to
arrange for a temporary guest account for
82
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
their guest. Cisco Umbrella services is used to
restrict access to appropriate sites while on
this network.
Corporate WiFi network for OIG’s GFE Provides full access to PBGC’s OIG production
computers end-user network mimicking wired access.
Access to this network is controlled by Cisco
ISE via AD authentication.
PBGC uses a split DNS configuration in which separate DNS servers are provided for internal and
external networks as a means of security and privacy management. The following table provides a
summary of external DNS which is hosted externally, but administered under this contract:
The following table provides a summary of internal DNS from PBGC’s Infoblox IPAM system which is
supported and administered under this contract:
6.4.1.2 Requirements
The contractor shall provide the network infrastructure support services outlined in the following table:
Reference Requirement
(NIO-NI-xx)
NIO-NI-01 Contractor shall identify a lead for the Network Infrastructure area. This lead is
required to serve as the primary point of contact for all network infrastructure related
issues.
NIO-NI-02 Contractor shall provide and maintain a fully functional, optimally performing, secure
enterprise network infrastructure in all PBGC computing environments. This shall
include, but is not limited to:
83
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Performing capacity planning and management including allocation and
reconfiguration of network infrastructure and associated services as needed
• Using monitoring tools to proactively plan and manage infrastructure
resources to maximize system and service availability
• Performing preventative and remedial maintenance of components
• Coordinating performance of work by vendors as required and in accordance
with PBGC Security policies, vendor warranties and maintenance contracts
NIO-NI-03 Contractor shall fully implement, manage, and support all incident management
activities regarding Network infrastructure according to PBGC policies. By using ITIL
best practices, this should include, but shall not be limited to:
84
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Apply applicable security patches at least monthly and install applicable
hardware firmware updates at least quarterly
• Address operating system and software vulnerabilities detected on network
infrastructure during monthly vulnerability scans
• Apply necessary configuration changes to ensure compliance with PBGC
security baselines and industry best practices
• Process approved modifications to PBGC’s firewall rule set and distributed
access control lists
• Establish and maintain site-to-site VPN connectivity as needed to support
secure network connectivity with other federal agencies and service
providers
• Process approved modifications to PBGC’s web proxy black and white lists
• Provision new subnets or remove those no longer in use
• Update network load balancers to support new or changes to existing
applications
• ensuring front and rear doors can be closed and are closed
• ensuring patch cables are of suitable length, are free from entanglement, are
patched the appropriate port based on applicable PBGC standards, and are
routed neatly to the patch panel
85
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• ensuring power to redundant power supplies is properly distributed between
PDUs to minimize downtime due to a single component failure
• maintaining Visio diagrams of the contents of each rack with
updates/verification no less frequently than every 6 months
• ensuring equipment not in use is powered down, clearly labeled, and
removed from the rack within 90 days of decommission
• ensuring all cables are clearly labeled for purposes of troubleshooting and
inspection
NIO-NI-09 Contractor shall ensure all local accounts supporting the network infrastructure, e.g.
the built-in admin account, are changed periodically in accordance with PBGC policy
and procedures and the passwords are stored for emergency use
NIO-NI-10 Contractor shall establish, maintain and post an inventory of virtual IP addresses to
include, at a minimum the following information regarding network load balancing
and clustering in the environment: VIP address, fully qualified DNS name,
balanced/clustered devices, application/service, date established, RFC#
NIO-NI-11 Contractor shall maintain a list of PBGC subnets in the IPAM system with details to
include subnet function and whether it is managed by PBGC or by a network carrier.
Contractor shall ensure that updates to this list are communicated to the
configuration management team and the vulnerability scanning team.
NIO-NI-12 Contractor shall establish and maintain a periodic task list to monitor and proactively
maintain the network infrastructure and the associated services
• Monitor, troubleshoot and repair or replace defective Telephony infrastructure and video
hardware and software as needed
• Relocate IT phone equipment in response to user relocation requests including updating
connectivity and update database containing relationships between phone numbers/extensions,
users and locations
• Install IT phone equipment for new user setups and remove upon user separation including
updating connectivity and update database containing relationships between phone
numbers/extensions, users and locations as well as CommView call accounting system
• Monitor, troubleshoot and resolve voice traffic routing issues between PBGC’s HQ campus,
Customer Call Center, and FBA/PVA locations
• Monitor telephony call statistics to detect anomalies in system performance and manage
underlying systems supporting call metrics and assist with call reporting issues and run reports
when issues arise
86
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Provision Polycom personal conference bridges for new user setups and remove upon user
separation
• Provide support to users and organizations that have requested voice/video conferencing
assistance
• Deploy patches, firmware updates, and software updates to the Telephony Infrastructure
• Maintain racks that contain telephony equipment
• Coordinate message updates on PBGC’s IVR
• Administer accounts on telephony infrastructure
• Establish and maintain work instructions
• Escalate and work collaboratively with 3rd party vendors on hardware and software issues
Support for Telephony Infrastructure equipment in the PBGC data centers is typically handled utilizing
remote management software, e.g. RDP, SSH, but does occasional require a physical visit to address
hardware and certain software problems. See Appendix F - PBGC Locations for PBGC locations, including
planned changes to PBGC’s facilities and data centers over the life of the contract. See Appendix G - IT
Service Support Guidelines for impact, urgency, and prioritization guidelines associated with IT service
and support. See Appendix J - IT Service and Support 2018 Statistical Summary for tier 2 incidents,
requests for information (RFIs), service/access requests, requests for change (RFCs), and RFC tasks
processed in calendar year 2018 for these services. See Appendix K - IT Infrastructure Tools List for the
software utilized to provides these services.
PBGC’s Telephony Infrastructure consists of many components detailed in the tables that follow:
PBGC is currently in the process of replacing its legacy telephony infrastructure at all PBGC locations
including the FBA sites by the end of June 2019. All legacy PBX and phones will be replaced with the
latest supported Cisco VoIP (Cisco UC and Voicemail) and call center (Cisco UCCX) technology. This effort
will also introduce enterprise FAX and the InformaCast emergency notification system. The Contractor
is expected to support all aspects of the new VoIP based telephony infrastructure. This effort is
expected to eliminate the NEC PBXs, NEC GNAV, NEC Voice Mail systems, NEC Quework, Nortel IVR,
Mutare ENS, and Qfiniti Etalk. The following table breaks down PBGC’s current telephony infrastructure
equipment by model for the existing system and the system being deployed in June 2019:
87
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Cisco Catalyst 3750 VOIP Switches 20 Cisco catalyst
9000 series
Switches
NEC Global Navigator Management information system that records 2 Cisco Finesse
(GNAV) the activity of calls, tracks the performance of
agents and coordinates the scheduling of
personnel
Nortel MPS500 Interactive Voice Response (IVR) 2 Cisco UCCX
NEC QueWorx Provides Customer Contact Center with a range 3 Cisco UCCX
of customer-focused applications
Qfiniti Etalk Call quality monitoring system 1 Cisco Calabrio
Mutare ENS Emergency Notification System 1 Cisco
Informacast
@Comm CommView Call Accounting and Reporting System 1 Latest
Commview
version
88
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
The following table breaks down PBGC’s target telephony infrastructure equipment by model for the
new system being deployed in June 2019:
Cisco 8800 Series Phones Digital and Voice Over IP (VOIP) desk phones 2500
Cisco UCCX Version 11 - 12.X Unified Contact Center Express – Call Center Up to 400
Agent, Interactive Voice Response (IVR) Agents
Cisco Calabrio Version 11 - Integrated solution for call recording, quality Up to 400
12.X assurance, workforce management, analytics Agents
and reporting.
All Cisco UC/UCCX solutions run on two VMware clusters, HQWPROD_CiscoPhone and
WILPROD_CiscoPhone, that are Cisco proprietary used to host Cisco UC comprised of BE7H-M5-K9
servers.
6.4.2.2 Requirements
The contractor shall provide the Telephony infrastructure support services outlined in the following
table:
89
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Reference Requirement
(NIO-TI-xx)
NIO-TI-01 Contractor shall identify a lead for the Telephony Infrastructure area. This lead is
required to serve as the primary point of contact for all Telephony infrastructure
related issues.
NIO-TI-02 Contractor shall provide and maintain a fully functional, optimally performing, secure
enterprise Telephony infrastructure in all PBGC computing environments. This shall
include, but is not limited to:
• Identifying and addressing performance bottlenecks
• Performing capacity planning and management including allocation and
reconfiguration of telephony infrastructure and associated services as needed
• Using monitoring tools to proactively plan and manage infrastructure
resources to maximize system and service availability
• Performing preventative and remedial maintenance of components
• Coordinating performance of work by vendors as required and in accordance
with PBGC Security policies, vendor warranties and maintenance contracts
NIO-TI-03 Contractor shall fully implement, manage, and support all incident management
activities regarding Telephony infrastructure according to PBGC policies. By using ITIL
best practices, this should include, but shall not be limited to:
90
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Apply necessary configuration changes to ensure compliance with PBGC
security baselines and industry best practices
• ensuring front and rear doors can be closed and are closed
• ensuring patch cables are of suitable length, are free from entanglement, are
patched the appropriate port based on applicable PBGC standards, and are
routed neatly to the patch panel
• ensuring power to redundant power supplies is properly distributed between
PDUs to minimize downtime due to a single component failure
• maintaining Visio diagrams of the contents of each rack with
updates/verification no less frequently than every 6 months
• ensuring equipment not in use is powered down, clearly labeled, and
removed from the rack within 90 days of decommission
• ensuring all cables are clearly labeled for purposes of troubleshooting and
inspection
91
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
NIO-TI-09 Contractor shall perform telephony infrastructure account administration functions in
accordance with PBGC policy and procedures for systems and functions where
automation is not already in place
NIO-TI-10 Contractor shall ensure that the configuration of all telephony infrastructure
equipment and associated data is backed to the corporate backup system up no less
than monthly
NIO-NI-11 Contractor shall ensure all local administrative accounts supporting the Telephony
infrastructure, e.g. the built-in admin account, are changed periodically in accordance
with PBGC policy and procedures and the passwords are stored for emergency use
NIO-NI-12 Contractor shall establish and maintain a periodic task list to monitor and proactively
maintain the telephony infrastructure and the associated services
PBGC’s Network Operations Center monitors the following facilities and facilities equipment:
92
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Data Center 2 HQW (1) – The NOC staff are currently located at this location in room
L721A.
WIL (1)
LAN Closets 24 BUC (1) – This facility is located away from HQ, so the NOC does not
conduct a walkthrough of the LAN closet. They use other monitoring tools
to verify equipment status (NNMi, SiteScan and Temp guard for
environmental)
DOR (1)- This facility is located away from HQ, so the NOC does not
conduct a walkthrough of the LAN closet. They use other monitoring tools
to verify equipment status (NNMi, SiteScan and Temp guard for
environmental)
HQB (2) – The NOC conducts a walkthrough of the LAN closet twice
during an 8-hour shift to conduct visual inspection of network and
telephony. The results are captured in the shift pass down log
HQI (1) - The NOC conducts a walkthrough of the LAN closet twice during
an 8-hour shift to conduct visual inspection of network and telephony.
The results are captured in the shift pass down log
HQW (12) - The NOC conducts a walkthrough of the LAN closet twice
during an 8-hour shift to conduct visual inspection of network and
telephony. The results are captured in the shift pass down log
KIN (2) - This facility is located away from HQ, so the NOC does not
conduct a walkthrough of the LAN closet. They use other monitoring tools
to verify equipment status (NNMi, SiteScan and Temp guard for
environmental)
QUI (1) - This facility is located away from HQ, so the NOC does not
conduct a walkthrough of the LAN closet. They use other monitoring tools
to verify equipment status (NNMi, SiteScan and Temp guard for
environmental)
EUC (2) - This facility is located away from HQ, so the NOC does not
conduct a walkthrough of the LAN closet. They use other monitoring tools
to verify equipment status (NNMi, SiteScan and Temp guard for
environmental)
WIL (2) - This facility is located away from HQ, so the NOC does not
conduct a walkthrough of the LAN closet. They use other monitoring tools
to verify equipment status (NNMi, SiteScan and Temp guard for
environmental)
Phone 24 BUC (1)
Closets DOR (1)
EUC (2)
HQB (2)
HQI (1)
HQW (12)
KIN (2)
QUI (1)
93
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
WIL (2)
Data Center 12 HQW (7) – The NOC monitors the status (temperature, humidity and any
AC Units alarms) of these unit by using SiteScan. The NOC conducts a walkthrough
of the Data Center twice during an 8-hour shift to conduct visual
inspection of the AC units and record temperature and humidly reading
from thermometers through out the Data Center.
WIL (5) – The AC units are located on the first and second floor data
centers. The only equipment on the first floor is the Power Distribution
Unit (PDU).
LAN Closets 22 EUC (1) - WSD has the maintenance contract for the AC unit. The NOC will
AC Units monitor the status via sitescan and will coordinate with LAN admin and
notify WSD if service is required after hours or over a weekend.
DOR (2) - WSD has the maintenance contract for the AC unit. The NOC
will monitor the status via sitescan and will coordinate with LAN admin
and notify WSD if service is required after hours or over a weekend.
HQB (2) – ITIOD has the maintenance contract for the AC unit. The NOC
will monitor the status via sitescan and visual inspection twice during the
8 hour shift and will coordinate with the provider if service is required.
HQI (1) - WSD has the maintenance contract for the AC unit. The NOC will
monitor the status via sitescan and will coordinate with LAN admin and
notify WSD if service is required after hours or over a weekend.
HQW (14) - ITIOD has the maintenance contract for the AC unit. The NOC
will monitor the status via sitescan and visual inspection twice during the
8-hour shift and will coordinate with the provider if service is required.
KIN (2) - ITIOD has the maintenance contract for the AC unit. The NOC will
monitor the status via sitescan and visual inspection twice during the 8-
hour shift and will coordinate with the provider if service is required.
Data Center 2 KIN (1) – The property management at Kingstowne has the support
Generators contract for the preventative maintenance of the generators a long with
Work Place Solution (WSD). WSD will schedule the maintenance with the
vendor during normal business hours. In the event of a power outage in
which the generator fails to operate after hours, the NOC has instructions
on who needs to be contacted and will facilitate the call.
WIL (1) - The property management at Wilmington has the support
contract for the preventative maintenance of the generators a long with
Work Place Solution (WSD). WSD will schedule the maintenance with the
vendor during normal business hours. In the event of a power outage in
which the generator fails to operate after hours, the NOC has instructions
on who needs to be contacted and will facilitate the call.
PBGC plans to move most of the on-premise IT equipment supporting the Data Center services to co-
located data centers over the next several years. Please refer to Appendix F – PBGC Locations for a
94
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
tentative timeline for this transition. Also, a Privileged Account Management tool, CyberArk, is being
deployed at PBGC (FY17/FY18) to store privileged credentials and broker and record sessions requiring
privileged access. These initiatives will eliminate the vast majority of required services noted in the
requirements for the Network Operations Center that follows. Once these initiatives are completed,
PBGC no longer sees value in hosting a complete Network Operations Center on premises. The
Contractor shall propose how to best meet any remaining requirements.
6.4.3.2 Requirements
The contractor shall provide Network Operations Center support outlined in the following table:
Reference Requirement
(NIO-NO-xx)
NIO-NO-01 Contractor shall identify a lead for the Network Operation Center. This lead is
required to serve as the primary point of contact for all network operation center
related issues and activities.
NIO-NO-02 Contractor shall provide and maintain a fully functional, optimally performing IT
Infrastructure by providing ongoing monitoring through the Network Operations
Center. This shall include, but is not limited to:
• Problems with the uninterruptible power supply (UPS) and power distribution
units
• Problems with AC units or with data center-wide or area specific temperate
issues
• Problems with fire detection/suppression equipment, heat exchanger, water
pumps, etc.
NIO-NO-06 Contractor shall staff a Network Operations Center (NOC) at the PBGC Headquarters
location to support continuous, uninterrupted (24 hours per day, 365 days per year)
monitoring of PBGC’s IT infrastructure as well as provide limited IT service and
support services including, but not limited to:
• Maintaining computer room security sign in log to include Password and hard
key check-out in accordance with PBGC policies and procedures
• Provide escort as needed for vendors performing maintenance or cleaning of
the data center (floors) or other designated locations
• Conduct a visual inspection of all infrastructure equipment racks every 4
hours to identify equipment with fault lights and escalate to the appropriate
contract staff
• Respond to monitoring tool e-mail notifications to coordinate
troubleshooting and resolution, e.g. HP Site Scope
• Monitor local area (LAN) and wide area network (WAN) availability via
performance monitoring and topology software, e.g. HP Network Node
Monitor (NNMI)
• Facilitate troubleshooting and circuit restoration with external circuit
providers by monitor external circuit provider portals for scheduled and or
emergency maintenance which may impact network availability
• Maintain a shift log to capture routine checks as well as outages or anomalies
that need to be escalated or passed to the other shift
• Provide end-user customer support outside of normal business hours
• Perform repeatable tasks in support of other IT infrastructure service areas,
e.g. imaging workstations, assisting with asset management activities, etc.
96
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
NIO-NO-07 Contractor shall ensure all local accounts supporting the data center facility
infrastructure including HVAC, power, and other environmentals; e.g. the built-in
admin account; are changed periodically in accordance with PBGC policy and
procedures and the passwords are stored for emergency use
NIO-NO-09 Contractor shall establish and maintain a periodic task list for the services provided by
the NOC
The ITSM and the Infrastructure Monitoring and Reporting services include the following:
See Appendix G - IT Service Support Guidelines for impact, urgency, and prioritization guidelines
associated with IT service and support. See Appendix J - IT Service and Support 2018 Statistical
Summary for tier 2 incidents, requests for information (RFIs), service/access requests, requests for
change (RFCs), and RFC tasks processed in calendar year 2018 for these services although these apply to
HP Service Manager 9, PBGC’s previous ITSM tool. See Appendix K – IT Infrastructure Tools List for the
software utilized to provides these services.
PBGC’s ITSM tool and associated services administration and support environment consists primarily of
the following:
98
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• PBGC’s 3 instances of ServiceNow instances; specifically, the ITSM module as well as PBGC’s on-
premise ServiceNow MID server and associated services. PBGC’s 3 instances are licensed for
2,400 users and 750 servers/nodes for discovery. PBGC Active Directory users and select groups
are synchronized to ServiceNow utilizing LDAP and the ServiceNow MID servers and user
authentication is federated utilizing Active Directory Federation Services (ADFS).
6.5.1.2 Requirements
The contractor shall provide the ITSM tool support services outlined in the following table:
Reference Requirement
(SM-SN-xx)
SM-SN-01 Contractor shall identify a lead for the IT Service Management (ITSM) tool support
area. This lead is required to serve as the primary point of contact for all ITSM tool
related issues and enhancement requests.
SM-SN-02 Contractor shall support and maintain a fully functional, optimally performing on-
premise infrastructure supporting ITSM, e.g. ServiceNow MID server, and
infrastructure services that support connectivity and integration with PBGC’s cloud-
based ServiceNow instances. This shall include, but is not limited to:
99
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Apply necessary configuration changes to achieve ITSM objectives including
compliance with PBGC policies and procedures as well as to achieve desired
operational configuration and user experience and align with industry best
practices. This shall include staging update sets to the appropriate OBGC
code repository for promotion to PBGC’s production instance of ServiceNow.
In addition to GetITAccess, PBGC has a SharePoint site called EnableIT to provide for request and
approval of requests for IT infrastructure resource and services such as:
• New Servers
• New Software Packages/Deployments
• New Databases
• Database Fixes (updates)
1
Includes only those “dynamic” roles generated via Active Directory group. There are approximately 10 “custom
developed” service catalog items that have custom forms and workflows to address more complicated service
requests like: new employee setup, employee separation, equipment relocation, direct Oracle access, etc.
Requests pertaining to accounts and access will move to ServiceNow/SailPoint solution. Requests for equipment
relocation will also move to Service Now.
101
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Database Refresh
• Database Restore
• New Database Role
• New Service Catalog Items
• New Storage Allocation
• New ServiceNow CMDB CI
• New Report
• New System or Application Monitoring
• New SharePoint site
• New SharePoint library with unique permissions
• New SharePoint content development
• Existing SharePoint content enhancement
The following table provides the count for EnableIT service requests processed in Calendar Year 2018
6.5.2.2 Requirements
The Contractor shall maintain PBGC’s IT Service catalogs, centralized catalogs of systems, resources, and
services that can be requested by PBGC users, including, but not limited to:
The contractor shall provide support the IT Service Catalog tools and associated support services
outlined in the following table:
Reference Requirement
(SM-SC-xx)
SM-SC-01 Contractor shall identify a lead for IT Service Catalog support. This lead is required to
serve as the primary point of contact for all IT Service Catalog support.
102
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
SM-SC-02 Contractor shall maintain the IT Service Catalogs, centralized catalogs of systems;
resources; and services that can be requested by PBGC users, including, but not
limited to:
• publishing any additional items/roles/IT infrastructure services as
required/requested
• unpublishing/removing any items/roles/IT infrastructure services that are no
longer needed as required/requested
• renaming items/roles/IT infrastructure services as required/requested
• changing primary and alternate approvers for items/roles/IT infrastructure
services as required/requested
• No less than monthly, identifying items/roles/IT infrastructure services
missing required information and obtain and update or escalate to federal
counterpart for assistance in data collection
SM-SC-03 Contractor shall reroute all service requests to the appropriate approver upon
request when in accordance with PBGC policy and procedures and alternatively
communicate to customer when such a reroute is not authorized
SM-SC-04 Contractor shall identify required on-line forms for all actions needed to maintain the
IT Service Catalog, e.g. change ownership, reroute requests, request for updates, etc.
SM-SC-05 Contractor shall ensure access requests are in compliance with approved governance
and serve as liaison for project teams and end-users to IT Customer & Operations
Services (ITCOS) division should questions regarding access control process and
procedures arise including access to Development and Test environments
SM-SC-06 Contractor shall provide automated reminders for IT service requests pending
approval or fulfillment to ensure all such requests are approved, fulfilled, and closed
out properly and on time and shall escalate to the federal service catalog manager
any request that has been pending approval or fulfillment for more than 30 days.
SM-SC-07 Contractor shall assist customers with identifying the appropriate the IT service
catalog item/role to request and the most efficient way to request it, e.g. manual
submission, bulk submission, requests for control groups, etc.
SM-SC-08 Contractor shall fulfill all approved service requests for IT infrastructure resource and
services (in EnableIT) in accordance with PBGC procedures and timelines for the
following:
103
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PBGC currently uses Network Node Manager (NNMi) and SiteScope for monitoring the status/availability
of the PBGC infrastructure. PBGC utilizes Microfocus Application Performance Management including
Synthetic monitoring, and Real User Monitor (RUM) for monitoring the availability and performance of
IT infrastructure systems and applications. Typical IT Infrastructure and Application Availability,
Capacity, and Performance Monitoring administration and support activities include:
• Monitor, troubleshoot, and address issues pertaining to PBGC’s IT Infrastructure and Application
Availability, Capacity, and Performance Monitoring tools; e.g. HP/MicroFocus NNMi, SiteScope,
BSM, BAC, etc.
• Configure PBGC’s IT Infrastructure and Application Availability, Capacity, and Performance
Monitoring tools, including adding and remove devices to the monitoring tools and configuring
application and system performance monitoring
See Appendix F - PBGC Locations for PBGC locations, including planned changes to PBGC’s facilities and
data centers over the life of the contract. See Appendix G - IT Service Support Guidelines for impact,
urgency, and prioritization guidelines associated with IT service and support. See Appendix J - IT Service
and Support 2018 Statistical Summary for tier 2 incidents, requests for information (RFIs), service/access
requests, requests for change (RFCs), and RFC tasks processed in calendar year 2018 for these services.
See Appendix K - IT Infrastructure Tools List for a comprehensive list of the software utilized to provides
these services.
6.5.3.2 Requirements
The contractor shall provide the IT infrastructure and application availability, capacity, and performance
monitoring services outlined in the following table:
Reference Requirement
(SM-AA-xx)
SM-AA-01 Contractor shall identify a lead for IT Infrastructure and Application Availability,
Capacity, and Performance Monitoring. This lead is required to serve as the primary
point of contact for all IT Infrastructure and Application Availability, Capacity, and
Performance Monitoring.
SM-AA-02 Contractor shall provide and maintain a fully functional, optimally performing IT
Infrastructure by providing ongoing monitoring capabilities. This shall include, but is
not limited to:
104
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
SM-AA-04 Contractor shall fulfill all approved service requests for IT infrastructure resource and
services (in EnableIT) in accordance with PBGC procedures and timelines for the
following:
105
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Generate and compile Contract required reports and deliverables
• Generate reports to support SLAs
• Collect requirements for and then develop new reports with federal approval
• Modify existing reports with federal approval
• Configure reporting tools
• Post reports to central location and/or distribute via email
See Appendix B – Deliverables and Appendix C - List of Required Meetings and Reports for more
information on required deliverable and reports that are generated as part of these services. See
Appendix K - IT Infrastructure Tools List for a comprehensive list of the software utilized to provides
these services.
6.5.4.2 Requirements
The contractor shall provide the ITIOD reporting and dashboarding services outlined in the following
table:
Reference Requirement
(SM-RD-xx)
SM-RD-01 Contractor shall identify a lead for ITIOD Reporting and Dashboarding. This lead is
required to serve as the primary point of contact for all ITIOD Reporting and
Dashboarding.
SM-RD-02 Contractor shall provide and maintain a fully functional, optimally performing IT
Infrastructure by providing ongoing reporting and associated capabilities. This shall
include, but is not limited to generation of:
106
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
SM-RD-04 Contractor shall fulfill all approved service requests for IT infrastructure resource and
services (in EnableIT) in accordance with PBGC procedures and timelines for the
following:
• New Report
PBGC utilizes the ITSM module of ServiceNow to track relate Service Desk interactions and incidents to
issues with an ongoing significant adverse impact to PBGC users and/or IT systems. For all such events,
PBGC requires the Contractor to produce an After-Action Report (AAR) within the weeks following the
major incident. Typical Major Incident Management activities include:
• Identify and escalate issues within the program and to federal incident manager
• Assess impact of issues
• Communicate with impacted users and with ITIOD federal stakeholders
• Develop recommended strategies to work around identified issues
• Conduct root cause analysis and resolve issues through PBGC’s change control process
107
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Open tickets with 3rd party vendors supporting IT infrastructure hardware, software and service
issues
• Develop and obtain approval for AAR
• Address any recommendations identified in AAR with federal approval
The following table summarizes the Major Incidents in the 2018 calendar year. PBGC transitioned to
ServiceNow as its operational system for incident management on January 19th. The following table is of
data between January 19th and December 31st, 2018.
108
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
INC0123276 2018-06-22 Master ticket - Emailed user letting him know this issue has been
11:59:45 Sharepoint - getting resolved. Pasting email below.
Access Denied error
message
The issue has been resolved. A notice to the
entire PBGC was just sent out stating:
`````````````````````````````````````````````````````````````````````
``````````````````````````````````
Impact: Users accessing PBGC
Connect/SharePoint sites may have gotten an
Access Denied message for one or more sites.
What's Happening: Technical teams identified
and resolved the issue that was causing
intermittent Access Denied messages when trying
to access some SharePoint sites.
Requested User Action: If you received an Access
Denied message when accessing a SharePoint
site, please close and reopen your browser
before accessing the site again.
`````````````````````````````````````````````````````````````````````
````````````````````````````````````
109
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
INC0123641 2018-06-26 Master Ticket - resolved by workaround instruction:
06:41:06 Outlook - Users are
getting an error Good Afternoon
when accessing
outlook client We have been investigating the outlook issue and
believe we have found a workaround. If you are
still having an issue with outlook, can you please
try these steps below to fix the outlook issue? Let
me know if you have any questions or concerns:
5. Select Install
110
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
INC0132393 2018-08-20 Master Ticket - The databases were restarted and service has
09:39:17 PROD - Three (3) been restored.
production
databases crashed -
COMPROD,
PMVPRD &
OEMPROD
INC0134419 2018-08-31 MASTER TICKET: Services have been restored.
09:05:39 CRM - Users are
reporting the
application is very
slow
INC0134587 2018-09-01 cor0prd1wdfp01 WAN service was restored by rebooting the
12:29:55 host unreachable Switches and L3 NID.
due to
cor0ent1crtr01.ent.
pbgc.gov being
unreachable
INC0134713 2018-09-04 Master Ticket - Technical teams worked with Microsoft, whom
11:24:52 Office365/Sharepoi identified an issue on the vendor side. Microsoft
nt Issue has confirmed that the issue has been resolved
and connectivity restored.
INC0134971 2018-09-05 Link from NMC continues to monitor the circuit after brief
12:42:44 Wilmington to bounced for stability, contacted service provider
Headquarters down. for support. AT&T immediately tested the circuit
and found it was a brief hit – came clear with
test, closing the ticket since the outage was
within the threshold.
INC0134982 2018-09-05 MASTER TICKET - Outlook and Skype thrttole was caused by an
13:32:56 outlook - error update which has since been rolled back by the
message "throttled" vendor Microsoft.
111
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
INC0139195 2018-10-01 MASTER TICKET - Restarting CAS DB resolved the issue.
10:50:35 CAS - users
reporting issues
with CRM, IPS and
Spectrum
INC0145149 2018-11-06 Master ticket - Java The certificate was renewed, the DRS file was
08:10:55 application error - updated and deployed to all workstations in all
Applciation Blocked environments. A BigFix fixlet has been developed
by Deployment Rule to push this file when necessary.
Set
INC0147828 2018-11-21 Master Ticket - Gigamon recommendations have been followed
13:21:56 Network Latency and UDLD has been set to be bypassed on the
and Applications Gigamon, all interfaces are back up
down
INC0149492 2018-12-03 Master Ticket - Determine Symantec Endpoint was scanning
14:26:20 Oracle Access several servers which caused overutilization
Manager Operation which translated to latency on several
Error applications. Security team will investigate on
why the scan was being run as this was not
scheduled in policy during business hours.
112
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
The following diagram depicts the workflow for PBGC’s major incident response process:
113
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
114
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
6.5.5.2 Requirements
The contractor shall provide the services outlined in the following table in supporting major IT
infrastructure incidents:
Reference Requirement
(SM-MI-xx)
SM-MI-01 Contractor shall identify a lead for Major Incident Response. This lead is required to
serve as the primary point of contact for the Major Incident Response process as well
as lead the Contractor’s response to major incidents when available.
SM-MI-02 The Contractor shall ensure that all IT major incidents are tracked and managed in a
consistent manner in accordance with PBGC’s Major Incident Response procedures to
include, but not limited to assessing scope and impact of issues, establishing
workarounds, corresponding with and obtaining guidance from 3rd party support
vendors, addressing root cause of the problem, and communication to impacted
customers and amongst subject matter experts
SM-MI-03 The Contractor shall provide a recovery method to restore service to the user(s)
affected by a major incident as quickly as possible. The Contractor shall perform root
cause analysis and, if it will restore service more quickly than any other approach,
develop and implement a permanent fix during the course of major incident handling.
SM-MI-04 The Contractor shall determine if the work to resolve major incidents is dependent
upon resources outside this contract, such as another vendor or an OEM. If that is
the case, the contractor, upon learning of the dependency, shall:
• Immediately request support from the vendor.
• Notify the PBGC Federal Incident Manager of the dependency including the
ticket number.
• Provide hourly (or time period agreed upon by the federal incident manager)
updates on the status of the ticket with the external vendor or lack thereof.
• Immediately be available to act on any vendor recommendations.
• Receive, respond, escalate, and resolve all IT major incidents in an expedient
manner.
See SLA section for details on existing major incident SLA measure.
SM-MI-05 The Contractor shall assess incidents and make recommendations to the Federal
Incident Manager and the COR regarding declaration of an issue as a “Major
Incident”. Once the Federal Incident Manager determines that the incident is major,
the Contractor shall immediately implement the major incident handling process
SM-MI-06 The Contractor shall at a minimum implement the following Major Incident Handling
procedures:
• Contact the Network Operations Center (NOC), contact the Contractor
Incident Manager, contact the Federal Incident Manager, open an incident
bridge, open an incident ticket, notify the on-call Subject Matter Expert, and
send a SOR or Advisory with Federal approval.
• Work with the Major Incident Management Team to resolve the incident, to
include creating a Request for Change (RFC) where necessary. The Contractor
shall follow the Change Management process for all RFCs.
115
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Update Federal IM and/or Operations Manager every half hour or as
requested
Once the incident is closed, the Contractor shall request Federal approval to close the
incident, send an ending SOR or Advisory noting the incident as closed.
SM-MI-07 The Contractor shall support the Federal Incident Manager in ensuring effective and
efficient resolution of major incidents and ensuring that communications related to
the incident are consistent and concise.
SM-MI-08 The Contractor shall produce an “After Action” Report within 14 days following the
conclusion of a major incident. See Appendix B – Deliverables for more information
on this deliverable.
PBGC utilizes the ITSM module of ServiceNow to track IT infrastructure problems. Typical Problem
Management activities include:
• Identify and track problems associated with activities under the Contractor’s control and
problems that the Government directs the Contractor to monitor
• Develop recommended strategies to resolve or workaround identified problems
• Document correspondence with 3rd party vendors supporting IT infrastructure hardware,
software and service issues
116
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Checkpoint - Checkpoint logs are not being 3- ITIOD- Open
338 Logs are not ingested into Splunk due to Modera OPSSECURIT
being ingested incompatibility issues between te Y-IC
into Splunk PBGC's Checkpoint modules and
Splunk. For any security
investigations in regards to USB
usage on PBGC workstations, the
ITIOD Security Operations team is
limited to the local logs that are only
retained for 120 days within the
Checkpoint management console.
Any security event involving
Checkpoint that occurred prior to
120 days will not be searchable.
PRB0001 Serena - PVCS Support, 4 - Low ITIOD- Open
334 VM 861 I am looking for an (old) solution to OPSCMTOOL
Vulnerabilities removing the PVCS VM security S-IC
vulnerability find for the below
numbers. I believe in the past we
turn off SSL in IIS or turned some
option off. Can you help with the
steps required to remove these
security vulnerability finding. See
attachment for listings.
INC0162834
INC0162892
117
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Veritas case - Plan of action has been devised. 4 - Low ITIOD- Open
331 190305- Waiting for approval to restore the OPSSTORAG
001504 - database from this morning backup. E-IC
unfinished Then inform Veritas.
OpsCenter
database
defragmentatio
n
PRB0001 Microsoft - Issue: Skype for business app. Need 4 - Low ITIOD- Open
330 Able to modify to lock down the application from to OPSWINDO
Phone Number prevent users from changing their WS-IC
settings in phones numbers. Users can currently
Skype change their Phone information on
the app causing the their contact
information to change.
118
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Acunetix - Web We learned that a scan of PBGC.gov 4 - Low ITIOD- Open
323 Application web site that is scheduled to run OPSSECURIT
Scans from 10pm through 6am 4 days a Y-IC
performed week was scanning the portions of
with Acunetix the PBGC.gov web-site daily. That is,
appear to not on the second day of scanning it did
be pauseable not resume from where it ended on
the first day but appeared to start
from the beginning again on the
second day of scanning. We have
reached out to the vendor for
assistance
PRB0001 '-Veritas case validation of old master server SAN 4 - Low ITIOD- Kno
320 190227- storage migration to new Hitachi - OPSSTORAG wn
001634 - INC0162981 E-IC Error
validation of
old master
server SAN
storage
migration to
new Hitachi -
INC0162981
PRB0001 Veritas case Image expiration report shows 4 - Low ITIOD- Pend
319 190227- images whose expiration dates have OPSSTORAG ing
000128 - Image passed/ INC0162978 E-IC Chan
expiration ge
report shows
images whose
expiration
dates have
passed/
INC0162978
PRB0001 Veritas - Veritas case 190226-001178 4 - Low ITIOD- Pend
318 Appliance 5230 OPSSTORAG ing
firmware E-IC Chan
upgrade ge
PRB0001 Veritas - Veritas - 190222-001296 Sev3 4 - Low ITIOD- Kno
317 190222- Vulnerability remediations OPSSTORAG wn
001296 Sev3 E-IC Error
Vulnerability
remediations
119
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Veritas - Sev3 Sev3 Security alert - Nessus plugin 4 - Low ITIOD- Pend
316 Security alert - Veritas case 190226-001178 OPSSTORAG ing
Nessus plugin E-IC Chan
ge
120
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 ServiceNow - SOR tab missing for Incident view in 4 - Low ITIOD- Open
304 SOR tab Development environment after OPSSERVICE
missing from London upgrade. Case has been NOW-IC
the Incident opened with vendor for resolution.
view in
Development
environment
PRB0001 Verritas - Not able to access java console on 4 - Low ITIOD- Open
300 Receiving error old master server- hqw0prod1bak01 OPSSTORAG
when trying to Receving TomCat certifcate error E-IC
access the
NetBackup java
console.
PRB0001 Unable to close Environment - Production. 1- ITIOD- Pend
299 test script as On trying to run and close test Critical OPSTCOTOO ing
HP scripts, with various test status (pass, LS-IC Chan
ALM/Quality no run, block) on trying to close test ge
Center script. HP ALM/QC does not respond.
becomes Vendor ticket no - SD02396238
unresponsive. -
SD02396238
PRB0001 Microsoft - Please open a ticket for the Windows 4 - Low ITIOD- Open
295 Skype Sharing team due to several users OPSWINDO
Issues experiencing issues sharing their WS-IC
screen and being dropped from
Skype for Business meetings.
121
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Oracle - one one oam sever in prod failed , which 4 - Low ITIOD- Open
289 oam server caused multiple applications outage OPSDBA-IC
failed today.
122
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Hitachi - Failed The issue is two ldev devices(4402 4 - Low ITIOD- Kno
288 to install two and 4431) failed to be installed. OPSSTORAG wn
ldev devices on E-IC Error
hqw0cdi0sdb1 One of them on hqw0cdi0sdb12/22.
2/22 and 14 The other on hqw0cdi0sdb14.
I0/HORCM/usr/bin/raidcom add
device_grp -device_grp_name
hqw0cdi0sdb12_5A_6A
hqw0cdi0sdb12_4402 -ldev_i
raidcom: [EX_CMDRJE] An order to
the control/command device was
rejected
It was rejected due to SKEY=0x05,
ASC=0x26, ASCQ=0x00,
SSB=0x2E20,0x0000 on
Serial#(445111)
CAUSE : LDEV is not installed.
[root@hqw0cdt1rcat01 bin]#
/HORCM/usr/bin/raidcom add
device_grp -device_grp_name
hqw0cdi0sdb14_5A_6A
hqw0cdi0sdb14_4431 -ldev_id 4431 -
s 445111 -I0
raidcom: [EX_CMDRJE] An order to
the control/command device was
rejected
It was rejected due to SKEY=0x05,
ASC=0x26, ASCQ=0x00,
SSB=0x2E20,0x0000 on
Serial#(445111)
CAUSE : LDEV is not installed.
PRB0001 Cisco - Vendor Vendor Engagement to Obtain Script 3- ITIOD- Open
281 Engagement to To Address Backlog Issue in IronPort. Modera OPSWINDO
Obtain Script te WS-IC
To Address Related to higher level event which
Backlog Issue caused email latency .
in IronPort
123
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 SharePoint - A vendor ticket (CS-258) has been 3- ITIOD- Open
277 Repeated created with the 3rd party Modera OPSSHAREP
alerts on disk SharePoint online class tagging tool te OINT-IC
space caused Concept Search.
by concept
search Issue: Repeated alerts on disk space
database. caused by concept search database.
Services need to be restarted
constantly. Tagging SharePoint
online items does not exhibit
expected behavior. Need health
check prior to version upgrade.
PRB0001 CyberArk - The CyberArk ticket is case # 3- ITIOD- Open
271 CyberArk 00370893 Modera OPSMEPROJ
connections te ECTS-IC
are dropping Losing connections to several TCO
hosts while working actively on
them. Most recent connection
dropped from
oracle@hqw0cdt0sdb15.
124
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 VM Ware - ESXi the ESXi Host 3- ITIOD- Open
265 Host and VMs "hqw0prd1evpi36.ent.pbgc.gov" and Modera OPSWINDO
Loss of VMs (running on this ESX Host) lost te WS-IC
Connection connection from vCenter
from vCenter.
Ticket to be created with VMware to
identify Root Cause.
PRB0001 SharePoint - Issue: User are having intermittent 4 - Low ITIOD- Pend
262 Users are issues with opening the library, OPSSHAREP ing
getting errors saving, searching and editing excel OINT-IC Chan
in CC Internal spreadsheets in this library. ge
Reports site
Library: CC Internal Reports
URL:
https://pbgcgov.sharepoint.com/tea
ms/PSD/CCD/CCC/CCInternalReports
/
125
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Veritas - The The OpsCenter vxpmdb.db 4 - Low ITIOD- Kno
258 OpsCenter database file was overwritten. Case OPSSTORAG wn
database file opened to Veritas to determine how E-IC Error
was to restore it.
overwritten.
Veritas case #181227-001203 on
how best to restore the Opscenter
database that was deleted due /var
partition getting full.
PRB0001 Hewlett QC freezes with certain commands: 4 - Low ITIOD- Kno
253 Packard - HP INC0151524 - QC Keeps Freezing OPSTCOTOO wn
ALM QC v1120 (Tiffany) LS-IC Error
Freezes - - INC0151588 - QC freezing when
Vendor navigating to requirements section
number: (Teresa)
SD02364889 - INC0151884 - change the status of
a test script from a test set takes
more than 5 mins (Teresa)
126
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Image Change retention policy for backups 4 - Low ITIOD- Kno
252 expiration - - and expire older backup images OPSSTORAG wn
Related to E-IC Error
Change - Related to INC0152139- and RFC
retention #CHG-0012192
policy for
backups and Expiring images as documented in
expire older RFC #CHG-0012192
backup images
1) please implement the image
expirations ASAP, starting with those
on MSDP01 and let me know when
done.
2) Who is going to be on the call with
Veritas besides Bashir? I am going to
be out of pocket for a couple of
hours. Please make sure that we
have the right person on the call
supporting Bashir.
3) Please make sure that we have
Veritas tell us what the right way is
to back up our Oracle 12c data based
on our current environment. Do not
tell them how we want to do it. Ask
them how they recommend doing it,
and summarize the plan for the team
after the call.
PRB0001 RedHat - Recently as a part of patching 4 - Low ITIOD- Open
251 Servers is hqw0prd1rcma61/62, OPSUNIXLIN
unable to boot alx0prd1rnes01 update to new kernel UX-IC
after upgrade version 3.10.0-957.1.3.el7.x86_64,
to new kernel After patching servers not able to
version boot in new kernel. Currently servers
are online on old kernel Version
3.10.0-862.14.4.el7.x86_64.
PRB0001 Microsoft - Users are reporting network drive 4 - Low ITIOD- Kno
249 W:\Drive Mapping issues with W:\Drive which OPSWINDO wn
Mapping issue is impacting Archive access every WS-IC Error
in Wilmington day.
Users have to click on the W:\drive
several times every morning after
they login to reconnect. (see
attached)
127
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Move the 4 - Low ITIOD- Kno
248 policies OPSSTORAG wn
attached from E-IC Error
med01 to
med02
PRB0001 Veritas - VMware full server restore fails. 3- ITIOD- Kno
244 VMware full Veritas case - 181210-001948 Modera OPSSTORAG wn
server restore te E-IC Error
fails
PRB0001 Micro Focus - I am running into an issue with the 4 - Low ITIOD- Pend
241 PVCS VM Client client when trying to login after OPSCMTOOL ing
- Unable to project has been converted from S-IC Chan
Login After VLOGIN to LDAP. User get two errors ge
Ldap Change 'Error reading Config file and cannot
access LDAP Password . See
Attachments for more detail.
LDAPINFOFILE = "L:/CAS-
Suite/pvcsldap.ini"
128
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Symantec - ITIOD Security Operations is working 4 - Low ITIOD- Open
238 Investigate with Symantec support in order to OPSSECURIT
Symantec AV determine if there was any AV Y-IC
scanning on scanning occurring on the weblogic
Weblogic servers on Monday December 3
Servers around 2:30pm EST.
PRB0001 Splunk - Increase in Physical Memory Usage 4 - Low ITIOD- Pend
234 Increase in on 3 indexers upon upgrade of OPSSECURIT ing
Physical Enterprise Security. This is a known Y-IC Chan
Memory Usage issue. Issue has being reported to ge
upon Upgrade Vendor.
of Enterprise
Security
129
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Infoblox - Users Desktop - Users connecting to 3- ITIOD- Open
232 connecting to remote.pbgc.gov getting -"unable to Modera OPSNETWOR
remote.pbgc.g resolve" host when selecting My PC te K-IC
ov getting - link.
"unable to
resolve" host Previously we used the below to
when selecting release the IP if previously connected
My PC link. to WiFi.
Open up command prompt as admin
and type:
ipconfig – will give you the active IP
address on the PC
nslookup "IP Address" – this will
resolve to the host name in DNS
nslookup "host name" – this will
resolve the IP address
130
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Gigamon - We experienced a network issue 3- ITIOD- Open
229 Network which caused severe latency on the Modera OPSPHONE-
Latency and PBGC Network. We isolated the issue te IC
Applications on the interface between AG2 and
down the Gigamon. This interface has been
shut down to resolve the latency
issue. Vendor case will need to be
opened with Gigamon to further
investigate the issue.
PRB0001 Infoblox - Several users when trying to access 3- ITIOD- Kno
228 Issues logging remote.pbgc.gov will get an error. Modera OPSNETWOR wn
into This is an issue with Infoblox and IP te K-IC Error
remote.pbgc.g host names not being released from
ov - IP host Wi-Fi subnet. This has been tested a
names not verified.
being released
from WiFi
subnet
PRB0001 Failed update Issues updating the firmware on a 3- ITIOD- Pend
210 of the component on one of the blade Modera OPSWINDO ing
Vcflexfabric chassis which is causing a te WS-IC Chan
vulnerability finding for Open SSH. ge
PRB0001 Surface Pro - Issue: Users are experiencing several 3- ITIOD- Kno
207 Systems are different issues on their Surface Pros. Modera OPSSITESUP wn
getting an te PORT-IC Error
operating error 1. Blue Screen of death
- BSOD 2. System is "Studdering"
3. Randomly will shut down.
4. system freezes
KB Document(s):MASTER TICKET -
Per Site Support
Verified Contact Information: Yes
Offered Ticket number: Yes
131
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Intact - Asset Following migration of AST43PRD 3- ITIOD- Open
198 Manager CG4 database from M8000 to T5-8, the Modera OPSSERVICE
integration Asset Manager to CG4 integration te MANAGER-IC
scripts not scripts failed. The scenario executes
running as when run manually. It keeps failing
expected. when run as a service. Produces
Stopping error: API error 57094232: '???'
shortly after
restart.
PRB0001 VMware - Jim Edwards is working with VMware 4 - Low ITIOD- Open
191 Setup vMotion Technical support to figure out the OPSWINDO
across configuration that would work with WS-IC
HQW/WIL sites PBGC VMWare current setup.
- ISO getting
help for
RFC0010127
PRB0001 DataGuise - DataGuise software is masking data 4 - Low ITIOD- Open
182 Data Masking with numerous inconsistencies after OPSDBA-IC
Software Issues Oracle 12c patch upgrade.
Data masking timing and status are
different between the software user
interface and the database log.
Documented masking times from the
old version has doubled after the 12c
patch upgrade.
Masking software is overall slow
while performing masking.
132
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Service Issue: In researching the Inactive COR 3 - ITIOD- Open
180 Manager9 - report we found some issues in SM 9 Modera OPSSERVICE
ID's are still . Lis Fortune-Williams has 2 different te MANAGER-IC
valid that ID's Requests are being made udder
should not the wrong ID. Both IDs are showing
be/Duplicate up as active.
IDs under user
name Need to check the LDAP feed or the
way the report is pulled in SM9 to
correctly identify that way a users
account status is conveyed to SM9.
KB Document(s): No KB Referenced
(Ticket Per Mike Skov)
Verified Contact Information: Yes
Offered Ticket number: Yes
PRB0001 Microsoft - When using RDP to connect to VDI 4 - Low ITIOD- Open
169 RDP - The machines and desktop computers, OPSNETWOR
connection has the remote session loses with a K-IC
been lost. message saying "The connection has
been lost. Attempting to
reconnected to your session...
Connection attempt: 1 of 20. See
attached.
PRB0001 Oracle - PROD - PROD - Master ticket - spectrum - 4 - Low ITIOD- Open
159 Master ticket - Users cannot access Spectrum OPSDBA-IC
spectrum - Application.
Users cannot Spectrum application created an
access overwhelming number of
Spectrum transactions that overloaded SDB12,
Application OHS03, OHS04 and crippled the
LDOM server bringing down all
production databases and
applications.
133
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Oracle - ASM Oracle 12c databases on the new 4 - Low ITIOD- Open
144 Instance lost Solaris 11 Server (SDB12) OPSDBA-IC
communication unexpectedly crashed due to a
with Oracle 12c communication issue the ASM
Databases. Instance.
DBA Team opened Ticket with the
vendor to urgently look into the
issue.
PRB0001 Bluecoat - No The issue with PBGC's Blue Coat's 4 - Low ITIOD- Kno
123 Maintenance version SGOS 6.5.10.4. The lack of OPSSECURIT wn
support support for CBC and ECDHE cipher Y-IC Error
contract for suites is preventing PBGC users from
Bluecoat proxy accessing FAITAS and DAU websites
on PBGC network via the Blue Coat
web proxies. Without a maintenance
support contract, PBGC cannot
obtain vendor's technical support nor
upgrade to the latest OS for Blue
Coat.
134
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 Oracle 12c Oracle 12c Platform Interface Issues 4 - Low ITIOD- Open
107 Platform Post Upgrade OPSDBA-IC
Interface Issues
135
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
PRB0001 ServiceNow - Email notification is not triggered 4 - Low ITIOD- Pend
077 Change from ServiceNow to the Change OPSSERVICE ing
Coordinators Coordinators after clicking "Request NOW-IC Chan
do not receive Approval to Deploy" ge
my email
notification
when clicking
"Request
Approval to
Deploy"
PRB0001 HPSM9 - Active ref ticket -sd254463, Q077179-001 4 - Low ITIOD- Open
019 Directory - tel number - 202-326-4600, 7456 OPSSERVICE
Autofullfiller pc number - pc0028759 MANAGER-IC
did not add
accesses user was approved for pd-sh drive
(full)
the autofullfiller did not add it to her
account. user states it was approve a
couple of weeks ago and was never
added to her profile
PRB0001 HP SM9 - Some Line items which are approved and 3- ITIOD- Open
009 sequenced line predecessors are complete are Modera OPSSERVICE
items hung in sometimes not move from requested te MANAGER-IC
'requested' to ordered status. This is especially
status. common for the iPhone fulfillment
sequence. (Sequenced line items not
showing up- SM9)
PRB0001 HPSM9 - GetIT The synopsis is that on occasion, 4 - Low ITIOD- Kno
002 returns an GetIT returns an error when OPSSERVICE wn
error when attempting to submit a request for MANAGER-IC Error
attempting to another user. The work around is to
submit a restart GetIT web services.
request for Problem ticket recreated from
another user existing PM in legacy Service
Manager 9 system. Legacy ticket PM
276).
PBGC transitioned to ServiceNow as its operational system for problem management on January 19th.
The following table is of data between January 19th and December 31st, 2018 and includes problem
tickets “opened” in ServiceNow as a transfer from the legacy system.
136
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Closed Existing Problem 97
6.5.6.2 Requirements
The contractor shall provide the IT problem management services outlined in the following table:
Reference Requirement
(SM-PM-xx)
SM-PM-01 Contractor shall identify a lead for IT Problem Management support. This lead is
required to serve as the primary point of contact for all IT Problem Management
support.
SM-PM-02 The Contractor shall ensure that all IT problems are tracked and managed in a
consistent manner in accordance with PBGC’s Problem Management Policies and
Procedures to include, but not limited to documenting scope and impact of issues,
established workarounds, correspondence with and guidance from 3rd party support
vendors, root cause of the problem, and communication to impacted customers
SM-PM-03 The Contractor shall review all open problems at least weekly and report progress
until the problem is resolved
SM-PM-04 Contractor shall open a problem ticket each time a 3rd party infrastructure hardware
or software vendor is contacted for service
SM-PM-05 The Contractor shall gather and record initial information from the source for the
reported problem, and perform the following actions pertaining to Problem
Management:
• Review incidents for indicators of eligibility for Problem Management
including any that are escalated to a vendor
• Perform incident matching to identify recurring incidents
• Review incidents for linked incidents
• Review alerts from applications and vendors for Known Errors
• Compare potential problems to existing problems
SM-PM-06 The Contractor shall apply criteria related to business impact and urgency to both
prioritize problems against each other, and to assist in decisions related to application
of resources for problem resolution
SM-PM-07 The Contractor shall provide a recovery method to restore service to the user(s)
affected by the incident as quickly as possible. The Contractor shall perform root
cause analysis and develop and implement a permanent fix. The overriding objective
is to restore service to the user as quickly as possible.
SM-PM-08 The Contractor shall create a Known Error Record. This shall involve but not be
limited to:
• Verifying satisfaction with the workaround
• Creating the Known Error Record in the tool
• Transferring the problem description to the Known Error Record
• Relating all relative incidents to the Known Error
• Incrementing incident count against the Known Error
Notifying the Service Desk that a new Known Error Record has been established
137
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
SM-PM-09 The Contractor shall correct the root cause of the problem, including but not limited
to resolve the root cause of the Problem in compliance with IT change management
process; and updating the Problem Record/Known Error Record with the solution.
SM-PM-10 The Contractor shall close out the Problem record to include, but not limited to:
• Documenting Change Record information in the Know Error Database
• Verifying the following fields are properly filled out in the ITSM tool:
o Problem categorization
o Problem prioritization
o Resolution activities
o Root cause analysis
• Updating the ticket as “Closed”
PBGC utilizes Microsoft Project on-line to track IT risks. Typical Risk Management activities include:
• Identify and track risks associated with activities under the Contractor’s control and risks that
the Government directs the Contractor to monitor
• Develop recommended risk mitigation strategies for identified risks
138
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
The ITIOSS Oracle Data/Info 3 4 12 Moderately Transfer/Share The current risk response is
Database High acceptance. Adding an ITIOD Federal
Administration team Approver does not mitigate the risk
provides a "data-fix" since ITIOD personnel typically do not
service to aid the have the necessary domain-specific
application support knowledge to evaluate these changes.
teams by correcting Proper risk mitigation requires
data errors in establishment of an auditable
production application/service owner review and
databases directly, approval process for data fixes, which
rather than through is not available currently. An effort to
officially released implement a service/application owner
application software approval list and management process
user interfaces. is currently underway and can
eventually be leveraged to support
accurate approval routing for data
fixes in the future.
Scrutinizer, which is Reliability Of 1 3 3 Extremely Reduce Currently in the process of finalizing
End of Life (EOL), is Systems Low - Minor Probability Netflow configuration on Cisco Prime.
used to collect Once the configuration has been
network traffic / completed, this risk can be closed since
NetFlow Prime can provide this capability.
information. If the
Risk is not mitigated
or another solution
identified, then
troubleshooting
traffic flow will be
challenging.
Verizon MTIPS Reliability Of 4 2 8 Moderate Reduce No impact thus far. There are multiple
Circuit reduction; Systems Probability options that can be accomplished if
This has the the Circuit Reduction impacts PBGC
potential of affecting business functions. PBGC plans to
all users of PBGC acquire larger circuits during its
internet based transition to EIS as well as acquiring
services including Express Route connectivity to Office
website access such 365.
as PBGC, myPAA,
myPBA, OIG, as well
as teleworking.
PBGC’s installed Reliability Of 2 3 6 Moderate Reduce Have already moved incident and
version of Service Systems Probability change functions to Service Now.
Manager is 9.35. This Working on moving request
version is end-of- management to ServiceNow and
support-life SailPoint. Confirmed vendor support
beginning for the current version that does not
November, 2017. As include bug fixes. Can upgrade to 9.5x
this software is the if a major issue surfaces.
ticketing system
behind the ITIOD
GetIT Access Service
Catalog, an inability
to obtain full Tier 4
support in the event
of a serious outage
could result in longer
service restoration
times and an
extended period
when PBGC
employees were
139
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
unable to request,
approve, and
provision IT Support
services and IT
system access.
Checkpoint logs are Security 5 2 10 Moderate Accept
not being ingested
into Splunk and the
logs are only being
retained locally in
the management
console for 120 days.
Replication of data Data/Info 2 3 6 Moderate Accept Work with the vendor to identify ways
for the SAN of increasing replication throughput or
Migration Project is recover data from backup.
running
slow. Estimated
replication
completion is
approximately 4
weeks. Getting data
on old LUNS will be
quite cumbersome, if
not impossible.
In the absence of a Reliability Of 2 3 6 Moderate Accept A dedicated circuit will be established
deployed solution, Systems to the new ITIOSS TO1 winner.
insufficient data
transport
information and
modeling is available
to guarantee PBGC’s
new Telephony
solution over PBGC’s
existing VPN
infrastructure will be
sufficient to support
continued ITIOD
service desk
operations.
The following table summarizes risks opened and closed in the 2018 calendar year:
6.5.7.2 Requirements
The contractor shall provide the IT infrastructure risk management services outlined in the following
table:
Reference Requirement
(SM-RM-xx)
SM-RM-01 Contractor shall identify a lead for IT Risk Management support. This lead is required
to serve as the primary point of contact for all IT Risk Management support.
140
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
SM-RM-02 The Contractor shall proactively identify, prioritize, and assess the likelihood and
potential impact of IT Risks, and develop and recommend options for mitigating those
risks
SM-RM-03 The Contractor shall ensure that all ITIOD Risks are tracked and managed in a
consistent manner in accordance with PBGC’s Risk Management Policies and
Procedures to include, but not limited to documenting probability, potential impact,
mitigation strategies, and escalation depending upon the rating of the risk
SM-RM-04 The Contractor shall review all open risks at least monthly and report progress until
the risk is resolved
• Serve as CAB secretary outlined in the Change Advisory Board Charter including, but not limited
to:
o Preparing and distributes CAB meeting documentation including agenda and meeting
minutes
o Tracking and following-up on action items from the CAB or change controls needing
required information
• Reconcile proposed changes with the master release schedule and the infrastructure calendar
• Establish and maintain knowledge base articles (KBAs) for the IT Change Management process
See Appendix J - IT Service and Support 2018 Statistical Summary for requests for change (RFCs), and
RFC tasks processed in calendar year 2018 for these services.
6.5.8.2 Requirements
The contractor shall provide the IT infrastructure change management services outlined in the following
table:
Reference Requirement
(SM-CM-xx)
SM-CM-01 Contractor shall identify a lead for the IT Change Management support. This lead is
required to serve as the primary point of contact for all IT Change Management
support related issues
SM-CM-02 Contractor shall comply with and support the PBGC Change Management process
that is based on the ITIL 3.0 framework and associated best practices.
141
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
SM-CM-03 Contractor shall perform the duties of the CAB secretary outlined in the Change
Advisory Board Charter including, but not limited to:
• Prepares and distributes CAB meeting documentation
• Documents, posts and distributes the board meeting minutes to all board
members and attendees
• Prepares CAB agenda based on the PBGC CAB request approved items within
ITSM tool and CAB Chair direction
• Ensures required room is reserved and necessary equipment prepared for
CAB meetings
• Prepares minutes that include discussions and decisions from the CAB,
obtains federal approval of same, and posts to appropriate on-line site
• Distributes CAB action items to the respective Federal RFC Requestors
• Follows-up on each action item and briefs the CAB Chair on their status
SM-CM-04 The Contractor shall conduct reviews of all proposed changes against the Master
Release Schedule and the Infrastructure calendar prior and advise the business
change manager of any conflicts
SM-CM-05 Contractor shall create and maintain knowledge base articles (KBAs) on PBGC’s IT
change management process to answer frequently asked questions and train IT
Service Desk staff to ensure they are prepared to answer basic questions regarding
PBGC’s IT change management process
142
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Monitor deployed assets (CIs) to ensure compliance with PBGC policy and notify on/report to
the government on non-compliance, e.g. mobile phones that haven’t checked in to MDM in 90
days, workstations that haven’t been on the network for 30 days, etc.
• Develop and generate reports on a regular and ad-hoc basis about hardware assets and
software licenses and consumables in the environment or in inventory
• Conduct licensing audits by reviewing software deployments relative to software licenses
• Conduct comprehensive annual inventory and reconciliation
See Appendix J - IT Service and Support 2018 Statistical Summary for requests for change (RFCs), and
RFC tasks processed in calendar year 2018 for these services. See Appendix B – Deliverables and
Appendix C - List of Required Meetings and Reports for more information on required deliverable and
reports that are generated as part of these services. See Appendix K - IT Infrastructure Tools List for a
comprehensive list of the software utilized to provides these services.
PBGC’s Asset Management administration and support is currently performed utilizing ServiceNow’s
asset management module. Inventory tasks are currently automated and mobilized utilizing CG4 bar
code scanners. PBGC has more than 7,000 tracked assets.
PBGC will maintain purchase responsibilities for assets and other related property such as parts and
supplies. PBGC will perform this function in procurement systems which are outside the scope of this
contract. These purchase responsibilities include the payment of invoices and necessary budgeting
activities. PBGC will also control the overarching property and asset management policies. The
contractor shall maintain and update documentation on asset and property management lifecycle
procedures and documents related to the asset system, administrative and operational procedures.
6.5.9.2 Requirements
The Contractor shall maintain accurate records and accountability for all Government Furnished
Property (GFP) including hardware and software assets, related licenses, warranty and maintenance
contracts, and parts and supplies, throughout the item life cycle. The contractor shall provide the IT
asset management services outlined in the following table:
Reference Requirement
(SM-AM-xx)
SM-AM-01 Contractor shall identify a lead for IT Asset Management support. This lead is required
to serve as the primary point of contact for all IT Asset Management support related
issues and activities
SM-AM-02 The contractor shall create each order of asset(s) and property within the asset
management system in preparation for the delivery and receipt of PBGC’s equipment,
parts and supplies, or software purchased
SM-AM-03 Contractor shall prepare, track, and document all IT Assets in accordance with ITIL
best practices and PBGC policy and procedures including preparing assets for use,
shipping assets, and updating location and ownership information in the ITSM tool
143
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
SM-AM-04 Contractor shall record any movement and/or updates to asset location or
assignments in PBGC’s IT Asset Management System within 3 business days of any
change
SM-AM-05 With the approval of COR, Contractor shall coordinate the disposal of Government
property that is no longer needed in accordance with the requirements of PBGC
Directives, NIST Standards, and the General Services Administration (GSA) including
degaussing if applicable
SM-AM-06 The Contractor shall perform a “wall to wall” annual physical inventory. The
Contractor shall conduct random and periodic inventories to verify accuracy and
accountability of asset management procedures and recording in the asset system.
SM-AM-07 The Contractor shall maintain hardware warranty maintenance records and
implement the provisions of the agreements when Government property fails to
perform in accordance with the specifications of the manufacturer
SM-AM-08 The Contractor shall be responsible for tracking “end-of-life” of PBGC hardware and
software assets and provide the COR and other designated PBGC federal staff with a
recommended schedule for the replacement or upgrade of those assets
SM-AM-09 The Contractor shall conduct licensing audits by reviewing software deployments
relative to software licenses at least annually for all software recorded in the ITSM
tool
SM-AM-10 The Contractor shall monitor deployed assets (CIs) to ensure compliance with PBGC
policy and notify on/report to the government on non-compliance
SM-AM-11 The Contractor shall monitor consumables (parts and supplies) and request
replenishment within one business day of reaching the minimum inventory threshold
of any consumable item. The contractor shall send daily reminders to the federal
asset manager, the COR, and the ITA&TRM division manager beginning one week
after reaching the minimum inventory threshold of any consumable item.
SM-AM-12 Contractor shall establish and maintain a periodic task list to maintain the IT Asset
Management system and associated data
• Add new Configuration Items (CIs) that can’t be automatically discovered and added to the
Configuration Management Database (CMDB)
• Configure and maintain the CI model in the CMDB
• Configure rules for mapping discovered CIs to the appropriate CI classification and populating
required attributes
144
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Synchronize and reconcile PBGC’s application list/CIs across disparate systems, e.g. DOJ Cyber
Security Assessment and Management (CSAM); used for storing information about PBGC’s
FISMA systems, ServiceNow CMDB; used to discover and manage PBGC’s Cis, and mAppIT;
PBGC’s centralized application list
• Develop and generate reports on a regular and ad-hoc basis about hardware, software, and CIs
in the environment
• Define relationships between CIs
• Monitor for and notify upon detection of unauthorized hardware and software installation
• Establish and maintain knowledge base articles (KBAs) for the IT Configuration Management
process
See Appendix J - IT Service and Support 2018 Statistical Summary for requests for change (RFCs), and
RFC tasks processed in calendar year 2018 for these services. See Appendix B – Deliverables and
Appendix C - List of Required Meetings and Reports for more information on required deliverable and
reports that are generated as part of these services. See Appendix K - IT Infrastructure Tools List for a
comprehensive list of the software utilized to provides these services.
PBGC’s Configuration Management tools and associated services administration and support
environment consists primarily of the following:
• PBGC currently utilizes the configuration management and discovery modules in ServiceNow
• PBGC’s Configuration Management program integrates the ServiceNow CMDB data with PBGC’s
ServiceNow asset management, it’s centralized application list, mAppIT, and its instance of
CSAM.
The scope of managed infrastructure CIs, known as CI Types, that are tracked by the PBGC infrastructure
CM process are currently (as of March 2019) as follows:
• FISMA System
• PBGC Developed Application
• COTS Application
• Workstation – Windows
• Laptop - Windows
• Mobile phone
• Database – MS SQL Server
• Database - Oracle
• Server - Windows
• Server – RHEL
145
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Server – Solaris
• Server – ESXi
• Storage Array
• Backup Device
• Network Attached Storage
• Printer – Multi-Function Device (MFD)
• Printer – non-MFD (networked)
• Video Conferencing
• Video Streaming
• Telephony Device
• Router
• Network Switch
• Network Load Balancer
• Firewall
• Virtual Private Network (VPN) Concentrator
• Storage Switch
• Network Tap
• Remote Access Device
• Security Device
• Network Management Device
• Environmental Monitoring Device
• Blade Device
• Uninterruptible Power Supply (UPS)
• Virtual Local Area Network (VLAN) / Internet Protocol (IP) Subnet
• IP Address
The degree of CM process that PBGC applies to managing a CI is based on the following factors:
The following table provides the count for EnableIT service requests processed in Calendar Year 2018
146
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
6.5.10.2Requirements
The contractor shall provide the IT configuration management services outlined in the following table:
Reference Requirement
(SM-CF-xx)
SM-CF-01 Contractor shall identify a lead for IT Configuration Management support. This lead is
required to serve as the primary point of contact for all IT Configuration Management
support related issues
SM-CF-02 The Contractor shall provide and be responsible for PBGC Configuration Management
processes that is based on the ITIL 3.0 framework and associated best practices.
Contractor shall be responsible for managing PBGC configuration management plans,
processes, and procedures support, including identification, control, status
accounting, and reviews and audits according to PBGC policy.
SM-CF-03 Contractor shall maintain the CMDB including ensuring relationships between CIs are
defined such that, at a minimum, the following reports can be produced at any point
in time:
• A report of all CIs (dedicated or shared) associated with a FISMA system
• A report of all CIs (dedicated or shared) associated with an application
• A report of impacted applications can be generated in the event of a failure
of any one CI
• A report of all CIs of a specific (or any) CI type with all associated attributes
according to PBGC’s CI model
SM-CF-04 Contractor shall fulfill all approved service requests for IT infrastructure resource and
services (in EnableIT) in accordance with PBGC procedures and timelines for the
following:
• New CMDB CI
• Conduct analysis on suspected phishing attempts and coordinate removal of confirmed phishing
attempts including blocking sender (or sender domain) and any associated URLs
• Conduct analysis on blocked and potentially miscategorized websites to determine if
whitelisting is appropriate and coordinate whitelisting if approved
• Inspect email quarantined as part of DLP or anti-malware measures and release if false-positive
• Review security information and event management (SIEM) events and alerts including potential
data exfiltration, excessive failed logins, potential web intrusion and take additional actions as
necessary
• Define security events and correlations based upon PBGC risk and architectural posture (known
risks, vulnerabilities, deficiencies) and configure SIEM to detect and report and/or alert on these
risks
• Coordinate ingestion of logs and configuration of notifications and/or reports in SIEM for
application events of interest to application stakeholders
148
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Blacklist URLs, domains, email addresses, etc. based on threat intelligence feeds
• Malware analysis, impact assessment, and removal
• Update/refine rule sets of various security tools to minimize false positives
• Monitor endpoint management software, e.g. Symantec End Point Protection, BigFix to ensure it
is current and activated and repair/reinstall as required and ensures all application devices are
having their security logs ingested into the SIEM
• Respond to approved data requests about suspicious activities and potential privacy breaches
Support for IT Security Tools Support and Cybersecurity Incident Response in the PBGC data centers is
typically handled using remote management software and protocols, e.g. https, SSH, Powershell, ILO,
OA, etc., but does occasionally require physical visits to address certain hardware and software
problems. See Appendix F - PBGC Locations for PBGC locations, including planned changes to PBGC’s
facilities and data centers over the life of the contract. See Appendix G - IT Service Support Guidelines
for impact, urgency, and prioritization guidelines associated with IT service and support. See Appendix J
- IT Service and Support 2018 Statistical Summary for tier 2 incidents, requests for information (RFIs),
service/access requests, requests for change (RFCs), and RFC tasks processed in calendar year 2018 for
these services. See Appendix K - IT Infrastructure Tools List for the software utilized to provides these
services.
The tables that following provide additional insight into the components of PBGC’s IT Security Tools
Support and Cybersecurity Incident Response infrastructure as well as the volume of activity expected:
IT infrastructure equipment that supports Cybersecurity will be made available in PBGC’s reading room.
The following table details the critical security events tracked in PBGC’s SIEM in Calendar Year 2018:
149
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Jun-18 87 14 73 False positives orginated from Office 365
activities.
Jul-18 44 9 35 Majority of false positives originated from
Symantec alerts.
Aug-18 54 20 34 False positives originated from NetBackup
devices.
Sep-18 57 24 33 False positives originated from NetBackup
devices.
Oct-18 40 18 22 Cryptomining alerts and phishing attacks.
Nov-18 128 77 51 JSP Webshell Backdoor attempts were dropped by
FirePower IDPS.
Dec-18 446 414 32 JSP Webshell Backdoor attempts were dropped by
FirePower IDPS.
Total 1737 641 1096
There were two (2) security events that were ultimately classified as security incidents and reported to
US CERT in calendar year 2018.
The following table provides the count for SecureIT service requests processed in Calendar Year 2018:
6.6.1.2 Requirements
The contractor shall provide the IT Security Tools and Incident Response support services outlined in the
following table:
Reference Requirement
(CS-IR-xx)
CS-IR-01 Contractor shall identify a lead for IT Security Tools Support and Cybersecurity
Incident Response. This lead is required to serve as the primary point of contact for all
IT Security Tools Support and Cybersecurity Incident Response related issues and
coordinate all ITIOD cybersecurity investigations.
150
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Reference Requirement
(CS-IR-xx)
CS-IR-02 Contractor shall provide and maintain a fully functional, optimally performing, secure
enterprise IT Security Infrastructure in all PBGC computing environments that
enhances capacity to respond to business changes and protects PBGC’s network and
data. This shall include, but is not limited to:
• Identifying and addressing performance bottlenecks
• Performing capacity planning and management including allocation and
reconfiguration of IT Security infrastructure and associated services as
needed
• Using monitoring tools to proactively plan and manage IT Security
infrastructure resources to maximize system and service availability
• Performing preventative and remedial maintenance of components
• Coordinating performance of work by vendors as required and in accordance
with PBGC Security policies, vendor warranties and maintenance contracts
CS-IR-03 Contractor shall fully implement, manage, and support all incident management
activities regarding IT Security infrastructure according to PBGC policies. By using ITIL
best practices, this should include, but shall not be limited to:
• Provide support for IT Security infrastructure hardware and software
• Investigate, troubleshoot, and resolve operational issues due to cyberattacks
such as denial-of-service attacks, DNS poisoning, exfiltration attempts,
unauthorized access, and phishing emails
• Assess, troubleshoot and resolve security issues including coordinating with
vendors to ensure PBGC systems and data are restored to normal operations
• Communicate to end users upon receipt of the security incident ticket and
the resolution of the security investigation
• Brief Federal management of any security tickets that require more than 7
business days to resolve
• Collaborate with other ITIOD platform teams to help restore system
availability due to outages from IT security issues.
151
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Reference Requirement
(CS-IR-xx)
• Apply necessary configuration changes to ensure compliance with PBGC
security baselines and industry best practices
• Submit and process approved SecureIT SharePoint requests for PBGC’s web
proxy blacklists and whitelists
• Update/refine rule sets of various security tools to minimize false positives
• Configure ingestion of security logs and define security events and
correlations based upon PBGC risk and architectural posture (known risks,
vulnerabilities, deficiencies) and configure SIEM to detect and report and/or
alert on these risks
• Develop and implement BigFix fixlets to meet defined requirements
• Create new or refine existing DLP policies to meet defined requirements
152
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Reference Requirement
(CS-IR-xx)
security threats are applicable to PBGC and formulate a defensive strategy to protect
the agency’s data and assets
CS-IR-07 Contractor shall assist Federal staff to respond to any security-related data call from
external government entities including but not limited to DHS, OMB, and Congress
CS-IR-08 Contractor shall assist Federal staff in investigations of potential insider threat or
fraud including but not limited to timesheet fraud and exfiltration of sensitive PBGC
data
CS-IR-09 Contractor shall provide threat and vulnerability management support, e.g., virus
protection, firewalls, IDS/IPS management, DLP, and the coordination of a wide
variety of information regarding threats and vulnerabilities of all PBGC platforms and
services
CS-IR-10 Contractor shall ensure all local accounts supporting the IT Security infrastructure,
e.g. the built-in admin account, are changed periodically in accordance with PBGC
policy and procedures and the passwords are stored for emergency use
CS-IR-11 Contractor shall participate in periodic exercises to demonstrate and assess readiness
for real security incidents. Contractor shall participate in periodic event and incident
handling quality reviews coordinated by PBGC. Contractor shall take appropriate
steps to mature process based on lessons learned and feedback provided.
CS-IR-12 Contractor shall scan non-PBGC media for malware upon request
CS-IR-13 Contractor shall establish and maintain a periodic task list to monitor and proactively
maintain PBGC security tools and appliances.
• Perform monthly, authenticated (when technically feasible) vulnerability scanning of all devices
on PBGC’s network
• Prepare for, attend and actively participate in regular meetings of the Patch and Vulnerability
Management Group (PVMG), the group which tracks vulnerabilities and coordinates their
remediation
• Perform web application, authenticated (when technically feasible) vulnerability scanning of
targeted PBGC systems and produce scanning results which include point-in-time web
application vulnerability details and summary information.
153
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Assess vulnerabilities for scope, research steps required to address, and coordinate needed
actions to address with appropriate Contractor resources. Collaborate and coordinate with
System Owners, Business Owners, and Application Developers as needed.
• Optimize the vulnerability scanning tools to improve the efficiency of the scanning process
Support for IT Vulnerability Scanning and Reactive Vulnerability Management coordination activities in
the PBGC data centers is typically handled using remote management software and protocols, e.g. https,
SSH, Powershell, ILO, OA, etc., but does occasionally require physical visits to address certain hardware
and software problems. See Appendix F - PBGC Locations for PBGC locations, including planned changes
to PBGC’s facilities and data centers over the life of the contract. See Appendix G - IT Service Support
Guidelines for impact, urgency, and prioritization guidelines associated with IT service and support. See
Appendix J - IT Service and Support 2018 Statistical Summary for tier 2 incidents, requests for
information (RFIs), service/access requests, requests for change (RFCs), and RFC tasks processed in
calendar year 2018 for these services. See Appendix K - IT Infrastructure Tools List for the software
utilized to provides these services.
154
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
The following image depicts a high-level summary vulnerabilities detected during a credentialed
vulnerability scan of the PBGC network in December 2018:
6.6.2.2 Requirements
The contractor shall provide the security vulnerability scanning and reactive vulnerability management
services outlined in the following table:
Reference Requirement
(CS-VM-xx)
CS-VM-01 Contractor shall identify a lead for IT Vulnerability Scanning and Reactive Vulnerability
Management. This lead is required to serve as the primary point of contact for all IT
Vulnerability Scanning and Reactive Vulnerability Management related issues.
CS-VM-02 Contractor shall provide and maintain a fully functional, optimally performing IT
Vulnerability Scanning infrastructure covering all PBGC computing environments that
enhances PBGC’s understanding of its cybersecurity risk exposure. Contractor shall
provide reactive vulnerability management services that reduces the cybersecurity
attack surface of PBGC’s network and data as well as its cloud hosted systems. This
shall include, but is not limited to:
• Identifying and addressing performance bottlenecks with regard to
vulnerability scanning
• Performing capacity planning and management including allocation and
reconfiguration of IT Security infrastructure and associated services as
needed
• Using monitoring tools to proactively plan and manage IT Security
infrastructure resources to maximize system and service availability
• Performing preventative and remedial maintenance of components
• Coordinating performance of work by vendors as required and in accordance
with PBGC Security policies, vendor warranties and maintenance contracts
CS-VM-03 Contractor shall perform monthly, authenticated (when technically feasible)
vulnerability scanning of all devices on PBGC’s network and produce scanning results
including point-in-time vulnerability details and summary information including the
155
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
“PBGC Security Posture Summary” report. See Appendix B – Deliverables for more
information on this report. Contractor shall troubleshoot and resolve authentication
issues on devices that support authenticated vulnerability scans.
CS-VM-04 Contractor shall fully implement, manage, and support all change management
activities regarding IT Vulnerability Scanning and Reactive Vulnerability Management
services they provide according to PBGC policies. By using ITIL best practices, this
should include, but shall not be limited to:
• Apply applicable security patches and configuration changes to address
vulnerabilities detected during vulnerability scanning
• Apply necessary configuration changes to ensure compliance with PBGC
security baselines and industry best practices
156
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
6.6.3 IT Security Controls Support
6.6.3.1 Scope of Services Supported
Provide IT Security Controls Support by providing direct support to the IT Infrastructure Services General
Support System (ITISGSS) Information System Security Officer (ISSO) in managing and documenting the
ongoing security posture of the ITISGSS and other ITIOD managed FISMA systems including applicable
NIST 800-53 security controls. Typical IT Security Controls Support activities include:
• Collect information from subject matter experts within ITIOD and develop control
implementation statements
• Update ITIOD FISMA system security documentation (SSP and other) with approved new,
significant changes requiring updates
• Support control assessments
• Provide expertise and assistance in the development of the security policies and procedures and
assist ensuring compliance with those policies and procedures
• Provide assistance in developing and updating the security artifacts, managing and controlling
changes to the ITIOD FISMA systems and assessing the security impact of those changes
• Assist in maintaining POA&Ms and the remediation of identified weaknesses
• Assist in coordination with other FISMA systems to ensure continual compliance with conditions
of inheritance from the GSS
During calendar year 2018, more than 113 IT security controls were assessed of which 97 were assessed
as satisfied/fully implemented and 9 were assessed as partially implemented. In support of this process,
more than 300 artifacts were provided to the security control assessors and several hundred questions
were answered. The ITISGSS has 288 NIST 800-53 rev. 4 applicable security controls of which 215 have
been assessed as satisfied/fully implemented and of which 25 have been assessed as partially
implemented. Approximately 165 of the ITISGSS security controls are offered to and inherited by other
PBGC FSIMA systems. It is worth noting that, in addition to the ITISGSS, ITIOD manages a FISMA system
for ServiceNow and a FISMA child of the ITISGSS for Office 365 cloud services and is working on
establishment of a second FISMA child of the ITISGSS for Microsoft Azure cloud services.
6.6.3.2 Requirements
The contractor shall provide support for the development and ongoing management and maintenance
of the security controls for the ITIOD managed FISMA systems including the IT Infrastructure Services
General Support System (ITISGSS) as outlined in the following table:
Reference Requirement
(CS-SC-xx)
CS-SC-01 Contractor shall identify a lead for IT Security Controls Support. This lead is required
to serve as the primary point of contact for all IT Security Controls Support activities.
CS-SC-02 Contractor shall develop and maintain IT security controls related to and offered by
the ITISGSS to the standards set forth in the NIST Special Publication 800-53 as
157
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
described in PBGC Security Policy. Contractor will consult with ITIOD subject matter
experts and review approved ITIOD work instructions in development of IT security
controls to ensure they accurately reflect the ITISGSS control implementation.
Contractor shall document and communicate any control deficiencies identified
during control development for POA&M consideration.
CS-SC-03 Contractor shall support PBGC IT Governance, Risk and Compliance Activities
(e.g., management of standards, approvals, waivers)
CS-SC-04 Contractor shall provide Continuous Security Monitoring. The Contractor shall
monitor the ITISGSS including all IT infrastructure and functional areas identified in
performance work statement in accordance with agency- defined parameters, for
compliance with PBGC Security Policy (SP) and all System Security Plans (SSPs) for the
ITISGSS
CS-SC-05 Contractor shall conduct detailed security impact analysis for any change that
introduces new (type of) hardware or software, requires modification to a security
baseline, requires a new connection to an external entity, significantly changes a
publicly facing application or DMZ infrastructure. Contractor shall ensure any
appropriate recommendations or information is provided in writing to
service/application owners and change coordinators.
CS-SC-06 Contractor shall support periodic control assessments including supplying requested
artifacts and responding to inquiries; coordinating with ITIOD subject matter experts
as needed
CS-SC-07 Contractor shall update the ITISGSS system security documentation (SSP and other)
with approved new, significant changes requiring updates including updating
boundary description and technical description to reflect current environment and
include inheritance within 30 days of completed RFC
CS-SC-08 Contractor shall assist with FISMA reporting
CS-SC-09 Contractor shall review outputs from POA&Ms to assess completeness and make
recommendations for additional work needed or POA&M closure
CS-SC-10 Contractor shall establish and maintain a periodic task list to monitor and proactively
maintain IT Security Controls
158
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
6.7.1 Scope of Services Supported
Provide tier 2 support for incidents relating to Test Center Operations that cannot be resolved by an
End-User services team as well as support service requests. Typical Test Center Operations
administration and support activities include:
Currently, the Test (ITC) lab is a physical facility located at 1275 K Street, but over the next 12-18 months
this is expected to become a virtual test center. See Appendix G - IT Service Support Guidelines for
impact, urgency, and prioritization guidelines associated with IT service and support. See Appendix J -
IT Service and Support 2018 Statistical Summary for IT Service Desk interactions, tier 2 incidents,
requests for information (RFIs), service/access requests, requests for change (RFCs), and RFC tasks
processed in calendar year 2017 for these services. See Appendix K - IT Infrastructure Tools List for the
software utilized to provides these services.
The tables that follow provide information about the size and transactional volume of PBGC’s Test
Center Operations:
The following table lists each Development, Testing, and Release Management Tool and the number of
associated projects and applications it supports:
The following table lists each Development, Testing, and Release Management Tool and the number of
associated releases it supported in calendar year 2018:
159
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Development, Testing, and Release Management Tool # of Releases in 2018
HP Application Life Cycle Management (HP ALM, HP Sprinter) / HP QC 12.x 1
HP Unified Functional Testing (HP UFT) 12.x 2
HP LoadRunner 12.x 1
Serena Business Manager (SBM) 11.x 4
Serena Version Manager (PVCS VM) 8.x 3
The following table lists each Development, Testing, and Release Management Tool and the estimated
number of associated incidents associated with it in calendar year 2018:
The following table lists each Development, Testing, and Release Management Tool and the number of
associated service requests were fulfilled for it in calendar year 2018:
6.7.2 Requirements
The contractor shall provide the Test Center Operations administration and support services outlined in
the following table:
Reference Requirement
(TC-TC-xx)
TC-TC-01 Contractor shall identify a lead for the Test Center Operations administration and
support area. This lead is required to serve as the primary point of contact for all Test
Center Operations administration and support related issues.
160
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
TC-TC-02 Contractor shall provide and maintain a fully functional, optimally performing, secure
Test Center Operations infrastructure that supports application/solution release
management and deployment in all PBGC computing environments as appropriate.
This shall include, but is not limited to:
161
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
TC-TC-05 Contractor shall provide support to project release teams for application release tools
including, but not limited to:
• Requirements management, test planning and functional testing, and defect
management
• Functional testing automation
• Load testing
• Process management and workflow automation
TC-TC-06 Contractor shall perform account administration functions in accordance with PBGC
policy and procedures for systems and functions where automation is not already in
place including, but not limited to:
• Non-AD-integrated accounts utilized by developers and testers for use of
application/solution development, testing, and release management tools
o Account unlocks and emergency/temporary disable/re-enable
o Password Resets (including remote user verification when applicable
per PBGC policy)
o Account removal upon request or separation
TC-TC-07 Contractor shall ensure all accounts supporting the Test Center Operations
environment including those utilized by/for application/solution development,
testing, and release management tools, e.g. local tools administrator and service
accounts are changed periodically in accordance with PBGC policy and procedures
and the passwords are stored for emergency use
TC-TC-08 Contractor shall establish and maintain a periodic task list to monitor and proactively
maintain the Test Center Operations infrastructure and associated services
162
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
6.8 Development, Modernization and Enhancements
The sections below describe the scope and requirements of the IT infrastructure Development,
Modernization, and Enhancement (DM&E) services that the Contractor shall provide under this
solicitation. Providing current, vendor-supported, optimally performing, highly available, and compliant
systems and service is critical to enable the accomplishment of the agency mission and as such, are
critical to excellent performance under this contract. Thus, Development, Modernization and
Enhancements is a key service area. Much of how the customer views the success of this contract will
be dependent on how well the Contractor performs DM&E, and how satisfied ITIOD staff members are
with the IT services provided. The Contractor is expected to provide DM&E Services for all IT
infrastructure technology noted in this PWS.
PBGC maintains software assurance or subscriptions for software outlined in this PWS that will require
periodic modernization and will obtain and furnish new versions of COTS software to the contractor as
Government Furnished Equipment (GFE). PBGC will also provide Contractor with any IT infrastructure
163
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
hardware or cloud services necessary to perform required modernization. The contractor shall plan,
develop, test, and execute upgrades as required, in compliance with the PBGC IT infrastructure
environment policies and standards as described throughout this PWS.
6.8.2 Requirements
The contractor shall provide ongoing development, modernization and enhancements as outlined in the
following table:
Reference Requirement
(DME-ME-xx)
DME-ME-01 Contractor shall identify a lead for the Development, Modernization and
Enhancement (DM&E) services area. This lead is required to serve as the primary
point of contact for all DM&E related issues.
DME-ME-02 Contractor shall support and maintain a fully functional, optimally performing on-
premise infrastructure as part of DM&E service delivery. This shall include, but is not
limited to:
164
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
DME-ME-05 Contractor shall fully implement, manage, and support all change management
activities for DM&E services according to PBGC policies. By using ITIL best practices,
this should include, but shall not be limited to:
166
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
6.9 Disaster Recovery/Continuity of Operations Planning (COOP) and Testing Coordination
and Execution
The sections below describe the scope and requirements of coordinating the Disaster
Recovery/Continuity of Operations Planning (COOP) and Testing Coordination (FFP) and Execution
(CPAF) services that the Contractor shall provide under this solicitation. Providing stable, reliable,
secure, optimally performing, and highly available systems and service is critical to enable the
accomplishment of the agency mission and as such, are critical to excellent performance under this
contract. Thus, Disaster Recovery/Continuity of Operations Planning (COOP) and Testing is a key service
area.
PBGC maintains a DR data center which hosts IT infrastructure and business applications needed to
support critical and essential PBGC functions in the event of a disaster requiring failover. Data is
replicated to the infrastructure located at PBGC’s DR data center through various replication
technologies including file-level replication, transaction log shipping, database mirroring, storage block
replication, etc. PBGC maintains redundant network connectivity to ensure alternative paths are
available to route network traffic in the event of any single failure. See Appendix D – PBGC Network
Overview Diagram for more information on network connectivity between sites and with the internet.
See Appendix F - PBGC Locations for PBGC locations, including planned changes to PBGC’s facilities and
data centers over the life of the contract.
ITIOD has a SharePoint-based solution called the COOP Tracker which is used to store procedures
needed to failover and failback infrastructure systems and services between PBGC’s primary and DR
data centers. The COOP Tracker has workflow that aids in coordinating the appropriate sequencing of
failover/failback procedures. This system also contains personnel lists, PBGC business area test plans,
and other information needed to perform failover and validate systems in the event failover is required.
During exercises and actual events necessitating failover, the system is also used to record and
communicate status of system/application availability.
PBGC has 42 agency critical functions. ITIOD performs 62 discrete steps as part of its failover
procedures, as of 3/25/2019, broken out as follows:
Team Steps
COOP Crisis Management Team 6
Database Administration and Support 7
UNIX/Linux Administration and Support 7
Site Support 3
Test Center Operations 1
Security Tools Operations 2
Windows Server Administration and Support 23
Storage and Backup Administration and Support 4
Network Infrastructure Support 9
Total 62
167
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
ITIOD performs 56 discrete steps as part of its failback procedures, as of 3/25/2019, broken out as
follows:
Team Steps
COOP Crisis Management Team 7
Database Administration and Support 6
UNIX/Linux Administration and Support 5
Site Support 3
Test Center Operations 0
Security Tools Operations 3
Windows Server Administration and Support 17
Storage and Backup Administration and Support 7
Network Infrastructure Support 8
Total 56
Typical activities in support of coordinating disaster recovery and continuity of operations capabilities
include the following:
• Maintain personnel lists and work instructions as well as test results in PBGC’s COOP Tracker
• Provide training for COOP readiness
• Update the ISCP for the ITISGSS based and the IT infrastructure Disaster Recovery Plan (DRP) on
changes to the environment including new business applications and infrastructure solutions
and perform ad-hoc failover/failback testing for new systems supporting critical or essential
function upon introduction to the environment
• Prepare for and coordinate execution of periodic DR failover/failback tests (typically 2 per year)
• Prepare reports regarding DR failover/failback tests
• Coordinate resolution of issues discovered during periodic DR failover/failback testing
• Provide support for the following COOP related exercises:
168
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
o Executive Table Top Exercise (typically one per year)
o Forward Challenge Exercise (typically one per year)
o Interagency Exercise each year (typically one per year)
• Coordinate responses to technical questions regarding the System Contingency process and the
overall IT COOP environment
6.9.1.2 Requirements
The contractor shall provide ongoing support for ITIOD’s disaster recovery and continuity of operations
capabilities including ongoing planning and testing coordination as outlined in the following table:
Reference Requirement
(DR-DRC-xx)
DR-DRC-01 Contractor shall identify a lead for Disaster Recovery/Continuity of Operations
Planning (COOP) and Testing. This lead is required to serve as the primary point of
contact for all activities related to Disaster Recovery/Continuity of Operations
Planning (COOP) and Testing.
DR-DRC-02 Contractor shall participate in coordinating recovery/failover operations and assist in
restoring the capability of the PBGC to conduct business, as required. PBGC will
manage the response to an actual disaster affecting Government locations and
provide direction to the Contractor regarding recovery/failover actions.
DR-DRC-03 The Contractor shall develop, obtain PBGC approval, maintain and coordinate
updates to the ISCP for the ITISGSS, the IT infrastructure DRP, the OIT COOP Plan, and
supporting documentation including work instructions, architecture diagrams,
application lists, location lists, personnel lists and other resources and information
related to the disaster recovery planning. See Appendix B – Deliverables for more
information on these documents.
DR-DRC-04 Contractor shall participate in Business Impact Analysis (BIA) for new PBGC business
applications and infrastructure solutions and based on any new requirements,
coordinate deployment of any needed infrastructure systems to the disaster recovery
data center and update the ISCP for the ITISGSS and the DRP as necessary
DR-DRC-05 Contractor shall prepare for and coordinate execution of periodic COOP
failover/failback tests (typically 2 per year) including preparation of the COOP
exercise plan, conducting COOP training, serving as part of the Crisis Management
Team during the exercise, providing end-to-end support for all failover; system
testing; and failback activities including documentation of all aspects of the event,
and communicating to exercise participants; ITIOD leadership; and agency leadership
prior to; during; and following each exercise
DR-DRC-06 Contractor shall provide a minimum of two conference bridges, with no dependency
on PBGC IT infrastructure, that can be utilized to coordinate recovery activities and to
coordinate application validation in the event that PBGC’s phone systems are
unavailable during an outage requiring failover to the DR data center
DR-DRC-07 Contractor shall establish corrective actions for each issue detected during annual
COOP exercises and identified in the After Action Report within 60 days. All
corrective actions must be stored in the appropriate PBGC repository, currently the
169
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
COOP Tracker, and have an assignee and due date. Contractor shall conduct meetings
to discuss and coordinate the COOP Exercise After Action issues, status and
resolutions and record and publish meeting minutes.
DR-DRC-08 Contractor shall establish and maintain a periodic task list to monitor and proactively
maintain ITIOD’s disaster recovery and continuity of operations capabilities
Typical activities in support of executing disaster recovery and continuity of operations capabilities
include the following:
• Maintain technical work instructions as well as test results in PBGC’s COOP Tracker
• Execute periodic DR failover/failback tests (typically 2 per year)
• Resolution of issues discovered during periodic DR failover/failback testing
• Provide responses to technical questions regarding the System Contingency process and the
overall IT COOP environment
6.9.2.2 Requirements
The contractor shall provide ongoing support for ITIOD’s disaster recovery and continuity of operations
capabilities including ongoing planning and testing execution as outlined in the following table:
Reference Requirement
(DR-DRE-xx)
DR-DRE-01 Contractor shall execute recovery/failover operations and assist in restoring the
capability of the PBGC to conduct business, as required. PBGC will manage the
response to an actual disaster affecting Government locations and provide direction
to the Contractor regarding recovery/failover actions.
DR-DRE-02 Contractor shall fully implement, manage, and support all change management
activities regarding for disaster recovery/continuity of operations planning (COOP)
and testing according to PBGC policies. By using ITIL best practices, this should
include, but shall not be limited to:
170
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Provide installation and configuration of IT infrastructure equipment and
services at PBGC’s DR data center including basic network and SAN
connectivity or decommission when no longer required
• Execute some or all of PBGC’s ISCP for the ITISGSS in response to actual
events or mandatory annual testing including travel to disaster recovery data
center facility as needed
6.10.1 Overview
Through a combination of research, industry feedback and experience with currently contracted cloud
services, PBGC has developed a set of tenets that will guide the acquisition, implementation and use of
171
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
cloud services. These tenets will evolve as the service offerings mature, PBGC’s needs change and its
familiarity with and understanding of cloud services grows. While the acquisition of cloud services will
not occur through this vehicle, PBGC does expect, over the life of this contract, to transition a significant
portion of its IT infrastructure to cloud services.
A. PBGC does not believe it is feasible to use only one cloud service provider but also does not see
effectiveness and efficiency in adopting a “new cloud for every need” approach. PBGC expects
to engage predominantly with one major cloud service provider (CSP) for infrastructure (Iaas)
and platforms (PaaS) services and has selected Microsoft Azure for this purpose. PBGC also
expects to engage with a limited number of additional CSPs based on particular business
requirements, typically providing software services (SaaS).
B. PBGC does not expect to engage in a simple IT infrastructure “lift and shift” approach to cloud
computing. PBGC expects to engage in cloud computing where the service offered is a fully
mature commodity, as the Corporation has done with electronic mail and office automation and
collaboration applications.
C. From an application development perspective, PBGC expects to employ a
“brownfield/greenfield” approach. As funding permits and needs require, PBGC expects to host
either newly developed applications, or those already fully modernized, in a PaaS “greenfield”
cloud environment. Legacy applications will continue to be hosted in PBGC’s existing IT
infrastructure (or perhaps eventually a simple IaaS “brownfield cloud”) until such time as they
can be replaced or modernized. The idea is to allow PBGC’s business areas to plan for – and
acquire the funding needed to – transition to the cloud based on their business requirements
and timeline.
D. PBGC will work with the Contractor to implement the most effective and cost-efficient
integration framework between PBGC’s brownfield and greenfield environments. Through the
use of an integration framework, PBGC will have the flexibility to transition applications and
infrastructure through the appropriate phases of the IaaS/PaaS/SaaS continuum based on
business requirements, prioritization and funding.
E. Given that PBGC is a financial services agency, providing benefits to millions of American
citizens, cloud IT security – and the ability to verify that security as much as possible – is of
paramount importance to PBGC. All cloud solutions and services must meet and maintain a
FEDRAMP moderate ATO.
F. In general, a cloud service will need to meet the following three requirements: 1) it will cost less
to acquire and operate than its locally-hosted predecessor; 2) it will be as secure or more secure
than its locally-hosted predecessor; and 3) it will meet functional and technical requirements of
the organization including speed and reliability.
6.10.2 Objective
PBGC is not seeking a CSP through this vehicle. Rather, we are seeking a trusted and adaptable partner
who can move through a cloud services life cycle with us. Within the context of this contract vehicle,
"partnership" means an interactive, mutually supportive professional relationship that is open,
172
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
collaborative, agile, and customer-oriented. In addition to meeting the objectives described herein, the
Contractor will be expected to:
• consistently take steps to understand PBGC’s crucial business issues and opportunities.
• identify and propose improvements to frameworks, processes, and services throughout the
performance period of the contract.
• share the risks and responsibilities of joint implementations and initiatives.
• ensure its products and services deliver tangible and meaningful business benefits.
• work collaboratively with other contractors, government agencies, and business partners to
ensure project success.
a) Strategy, Planning and Acquisition Support (SPAS). The Contractor shall consult with PBGC
regarding what type of cloud services may be most appropriate for a particular business
requirement; engage in collaborative planning regarding project scope, timelines and associated
activities; and provide – outside the realm of inherently governmental functions – support for
the acquisition of cloud services. Specific activities in this phase may include, but are not
necessarily limited to: 1) assist with analysis of appropriate cloud service providers vis a vis
PBGC’s business, funding and security requirements; 2) assist in the creation of presentations,
budget plans and projections, strategic roadmaps and project plans to provide
recommendations; and 3) assist with the creation of communications plans for all affected
parties of the migration(s) to ensure end-user adoption, customer satisfaction, successful
organizational process changes, and alignment with PBGC policies, requirements and goals.
b) Implementation and Integration Support (IIS). The Contractor shall work collaboratively with the
CSP, PBGC Federal staff and other contractors as applicable to implement any newly acquired
cloud service and integrate it with the relevant other aspects of PBGC’s IT infrastructure. Specific
activities in this phase may include, but are not necessarily limited to: 1) providing the technical
support necessary to implement or migrate the PBGC target applications and services to the
cloud; 2) providing migration status reporting including milestones and support or implement
specified migration testing plans and related rollback capabilities; and 3) providing support to
ensure that the implemented cloud services are integrated properly with other cloud services
and/or existing on-premise IT infrastructure services managed through this vehicle.
c) Monitoring, Alerting, Logging and Liaison Support (MALLS). The Contractor shall engage in
activities necessary to ensure that the CSP is providing the service(s) acquired, to the required
applicable quality standard(s). These activities will include monitoring the quality of service,
alerting PBGC when service is below standard or security events occur and ensuring that
appropriate logs of the auditing and monitoring activities are created and maintained. The
Liaison activity will entail providing customer service and technical coordination between the CSP
173
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
and PBGC. PBGC anticipates MALLS will be on-going in each technology and service area. Specific
activities in this phase may include but are not necessarily limited to: 1) providing post-
deployment evaluation of the CSP, to ensure compliance with SLAs, and make recommendations
about competition among CSP where cloud provider performance is less than acceptable; 2)
providing recommendations for specified auditable events related to the applications or services;
and 3) providing recommendations for the creation of the most effective compliant risk
assessments, routine vulnerability scanning, system patching and change management
procedures, and the completion of an acceptable contingency plan for the cloud service.
d) Continuous Cloud Services Security Support (CCSSS) Security requirements pervade all three
areas described above. The Contractor, therefore, shall assist with security tasks such as the
review and documentation necessary to produce an Authority to Operate (ATO) or customize an
existing FedRAMP ATO for PBGC’s use. Specific activities in this phase may include but are not
necessarily limited to: 1) providing recommendations for support and cloud services in
compliance and alignment with Federal statutory requirements (e.g. 38 U.S.C. 5725) governing
the protection of Personally Identifiable Information (PII), Federal Risk and Authorization
Management Program (FedRAMP) standardized security assessment, authorization, and
continuous monitoring policies; 2) providing cloud migration recommendations regarding
security and privacy that are consistent with the NIST Special Publication 800-144 – “Guidelines
on Security and Privacy in Public Cloud Computing” or other applicable standards and guidelines;
3) providing recommendations for security for non-standard data transfers both in transit and at
rest resulting from the migration of the applications or services to the cloud; 4) identify any
additional security and privacy standards to which cloud service providers should conform their
services/solutions; and 5) providing recommendations for contract support to verify that
security requirements are documented in the contract between the cloud service provider and
the PBGC.
PBGC anticipates that the Contractor, with two exceptions, will provide these services either with
existing staff or with staff who transition from a particular on-premise technology billet to one
associated with analogous cloud services. For example, an Exchange administrator who was
responsible for on-premise Exchange tasks would have some role to play in the SPAS phase. He/she
would have a primary role in the IISS area. Whether he/she would then transition to the MALLS area
would depend on factors such as number of staff required, the employee’s professional aspirations
and whether the employee’s skill set was applicable to the on-going activities in the particular service
area. It may be that equivalent or roughly equivalent numbers of staff may be required but new
people may need to inhabit those billets.
174
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix A - QASP
Double click on PDF document below to open:
175
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix B - Deliverables
In accordance with FAR 52.246-5, the results of any and all work performed under this contract are
subject to inspection by the COR or the COR’s designated representative. The following list of deliverables
represents that sub-set of those results that will be provided to the COR in 10 business days if it is not
defined by the CDRL, and for which a formal acceptance and approval must be obtained from the COR.
The Contractor is encouraged to propose improvements in both the content and delivery mechanism of
the deliverables.
176
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Note: as appropriate, “assets” refers to both
hardware and software
177
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
8 Quarterly Maintenance Quarterly The checklist that documents a complete plan and
checklist schedule for quarterly patching
178
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
obtain a formal acceptance and approval from the
Federal staff responsible for the services
inspected that month.
11 Monthly Financial Monthly The Contractor shall submit to the COR a Monthly
Report Financial Report no later than 10th business day
of each month. The report shall include
categories that consist of Estimated Hours,
Estimated Cost, Actual Hours, Actual Cost, Hours
Requires Until End of FY (already in Contract LOE),
Cost Requires Until End of FY (Already Obligated
for the Contract), Additional Hours Needed Until
FY and Additional Funding Needed. The
Contractor shall provide these reports for each
Functional Area of Work Requirements specified
in this RFP or Project (as the result of M&E and
Continuous Improvement Program) the
Contractor provides support for PBGC under this
Contract.
12 OIT COOP Plan Annually Establishes a general approach for OIT
departments to adapt that is coordinated with
other PBGC and Federal entities when faced with
a localized or widespread emergency to
accomplish the following priorities:
• Preserve the lives and safety of all
PBGC personnel, contractors and
visitors
• Continue or recommence serving our
customers and stakeholders at the
earliest and safest opportunity
• Resume normal operations when
practicable
This plan shall include the following:
• Identification of all the department’s
critical and essential functions.
• Orders of succession and delegations
of authority to key departmental
positions and responsibilities
• COOP essential records
179
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
• Recovery time objectives for each
function
• Department standard operating
procedures (SOPs) to assist individuals
in the performance of the critical and
essential functions
• Information Technology (IT) assets
(systems and applications) required to
perform all critical and essential
functions
• Notification of emergency and
implementation of COOP procedures
• Human capital information
• Alternate facility site locations and
directions
• Description of COOP essential records
• PBGC Critical, Situationally Critical,
and Essential Functions
• Checklists and worksheets to activate
the COOP Plan and support PBGC
management and leadership
• External contacts necessary to
perform the critical and essential
functions
• Test, Training and Exercise guidance
13 Information System Thirty (30) The Contractor shall develop and submit a
Contingency Plan calendar Information System Contingency Plan (ISCP) for
(ISCP) for the days after the Information Technology Infrastructure
Information the award Services General Support System (ITISGSS)to the
Technology of the Government. The ISCP for the ITISGSS shall be
Infrastructure Services contract, due thirty (30) calendar days after the award of
General Support and will be the contract, and will be updated on a quarterly
System (ITISGSS) updated on basis. The ISCP for the ITISGSS establishes
a quarterly considerations for and procedures to deal with
basis. various contingencies to ensure PBGC’s IT
infrastructure and IT systems supporting PBGC’s
mission and essential functions remain available
including in the event of a disruption requiring
failover to PBGC’s disaster recovery data center or
failback from it. This document describes the crisis
management process utilized to coordinate
activities during contingencies impacting normal
operations. It references the COOP tracker and
associated discrete steps (work instructions) to
180
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
perform failover and failback. The ISCP for the
ITISGSS shall meet the specifications contained in
NIST 800-34 and also include the following:
1. A description of the Contractor’s emergency
management procedures and policy
2. A description of how the Contractor will
account for their employees during an emergency
3. How the Contractor will communicate with
PBGC during emergencies
4. A list of primary and alternate Contractor
points of contact, each with primary and
alternate:
a. Telephone numbers
b. E-mail addresses
14 Disaster Recovery Plan The Contractor shall develop and submit an IT
infrastructure Disaster Recovery Plan (DRP) to the
Government. The DRP for the ITISGSS shall be
due thirty (30) calendar days after the award of
the contract and will be updated on a quarterly
basis. The DRP establishes procedures to recover
PBGC’s IT infrastructure and IT systems supporting
PBGC’s mission and essential functions following a
disruption requiring failover to PBGC’s disaster
recovery data center or failback from it. This
document describes the crisis management
process utilized to coordinate failover and
failback. It references the COOP tracker and
associated discrete steps (work instructions) to
perform failover and failback. The DRP shall
include the following:
1. A description of the Contractor’s emergency
management procedures and policy
2. A description of how the Contractor will
account for their employees during an emergency
3. How the Contractor will communicate with
PBGC during emergencies
4. A list of primary and alternate Contractor
points of contact, each with primary and
alternate:
a. Telephone numbers
b. E-mail addresses
181
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
15 EOSL Report Annually This report shall include technical
recommendations for development and
modernization activities for at least the next 3
years. This report should ensure that PBGC is
considering the most cost and technically effective
approaches to product procurement, replacement
and upgrade. The report shall contain all EoSL
items, the EoSL date, and recommendations for
technology refresh, replacement or removal of
the item and its function from the future PBGC
environment.
16 Periodic Task List Ninety (90) The Contractor shall, within 90 calendar days after
calendar award of the Contract, prepare and submit the
days after tasks on the Periodic Task List (PTL) for all services
award, and work requirements in all functional
updated as areas. The contractor shall establish in the PTL a
needed list of all repeating tasks and apply them to a
thereafter frequency table. The PTL frequency table shall
include tasks to be completed every hour, two (2)
hours, four (4) hours, at shift change, daily,
weekly, monthly, quarterly, semi-annually and
annually. The PTL shall contain all required
repeatable tasks necessary for the continuous
delivery of all IT Services and products. The
Contractor shall maintain the PTL throughout the
life of the Contract. The Contractor shall submit
to the COR changes in PTL not less than 3 days
prior to the desired date of implementation or as
directed by the COR. The Contractor shall not
implement any changes to the PTL until
authorized by the COR. The Contractor shall
notify the Contracting Officer of any changes that
affect contract cost and not implement these
changes until receiving approval from the
Contracting Officer.
17 Contractor Contact and Monthly Each month, the Contractor shall submit the
Staffing Report Contractor Staffing Report according to PBGC
monthly contractor staffing report process
including, as a minimum, the Contract Number,
Contractor Name, Employee Primary User ID,
Employee Last Name, Employee First Name, Work
Location, Office Number, Telephone Numbers,
Electronic Mail Address, Primary Work
Assignment, Start Date, Separation Date, Risk
182
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Level (moderate for regular users, high for AP
accounts) and Developer Status. The Contractor
shall notify the COR of any additions, deletions, or
changes within one working day after the
change(s). The Contractor shall include within the
Staffing Report a summary of all terminated staff
for the past 30 days and a summary of planned
hires for the next 30 days.
18 Major Incident After- 14 days Following the conclusion of a major incident, the
Action Report (AAR) after the Contractor shall submit an after-action report
conclusion including, at a minimum, a description of the
of a Major incident, impact of the major incident, a detailed
Incident timeline of events leading up to the onset of the
issue; steps taken to troubleshoot and/or resolve
the issue; communications regarding the major
incident, root cause analysis, and lessons learned.
19 SIM After Action 14 days Following the conclusion of a security incident,
Report after the Contractor shall submit an after-action report
reporting a including, at a minimum, a description of the
security incident, technical and business impact of the
incident to security incident, a detailed timeline of events
US-CERT leading up to the onset of the issue; steps taken to
respond; recover; and remediate the issue;
communications regarding the security incident,
and lessons learned.
20 PBGC Security Posture Monthly Provides a high-level executive review of
Summary vulnerabilities detected during a credentialed
vulnerability scan of the PBGC network which shall
include, at a minimum:
• Vulnerability totals for current scan and
trends from previous 6 months
• Authentication Success and Failure Rates
• Top 15 most vulnerable subnets
• Top 15 most vulnerable hosts
• Top 15 Vulnerabilities
• Instances of FTP, Telnet, Open SMTP
relays, Default SNMP Community strings
183
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
21 PBGC Vulnerability Monthly This report shall track the status of vulnerabilities
Status Report and compliance monitoring including, as a
minimum:
• Vulnerability Issue
• Vulnerabilities identified as False -
positives
• Vulnerabilities requiring Risk Acceptance
• Vulnerabilities that remain open
• Instances of FTP, Telnet, Open SMTP
relays, Default SNMP Community Strings
• Severity
• Host Name
• IP Address
• Original Discovery Date
• Number of Days Open
• Plan for Remediation
184
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix C - List of Required Meetings and Reports
It is expected that certain number of impromptu meetings and ad hoc reports will be necessary during
the course of this contract. It is not expected that the time and effort required for those items will be
beyond the normal course of business. Below is the list of meetings and reports that PBGC currently
requires in order to perform its mission. Although these meetings and reports do not require formal
acceptance and approval by the COR or the COR’s designated representative, the Contractor shall
conduct or produce them in a fashion consistent with PBGC’s overall expectations of quality and
timeliness.
186
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Daily Problem Tickets Daily List of open and aged problem tickets
Incident Ticket Aging Report Daily List of aging Incident and RFI Tickets by
Assignment group
Service Catalog Request Report Daily List of new Service Catalog Request
Daily Backups Daily Provide status of Daily Backup tasks
Backup Errors Daily Provide status of Daily Backup errors
VCenter Data Store Capacity Report Daily Provide capacity status for VCenter Data Stores
Solaris System Performance Report Monthly Provide System Performance for all solaris servers
2018 Availability Report (Business Hours) Monthly Monthly COR Report
High Level Summary -COR Report
SharePoint System Performance Report Monthly Provides SharePoint System Performance
MyPAA Infrastructure Report Daily MyPAA CPU Disk and Memory Performance
Report
MyPAA_WebURL_Report Daily MyPAA WebURL Transaction Availability and
Performance Reports
DataCenter System Availability Detailed Weekly Provide System Availability Details for PBGC data
Report center
Non-Compliant iPhones Users Monthly Compile report from Intune Server to see which
iPhones have not contacted the Server within 30
days
Master Release Schedule Daily Report produced daily on all open and scheduled
changes
ITISGSS Audit Report Semi- Report provided, with fully information from
annually current ITSM tool, twice a year to the auditors, for
all changes up to a certain date
CAB Voting Member Attendance Report Quarterly Report to provide metrics on voting member
attendance for the previous quarter
Unauthorized Change Report Weekly Report providing the number and description of
any changes that occurred without authorization
via the change management process
Emergency Change Review Report Quarterly Report documenting that all emergency changes
were reviewed at the succeeding weekly CAB
meeting
CAB Agenda Weekly Agenda published prior to that week’s CAB
meeting
CAB Minutes Weekly Provides results of changes approved, action
items and status of forward changes
CAB Forward Schedule Changes Report Weekly Provides 3-week forward view of upcoming
changes and tasks
Business Unit Current and Overdue Changes Daily Report provided for review with specific business
Report units of their current and overdue changes and
tasks
187
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Pending Change Closure Report Weekly Report provided weekly for discussion between
release and change management to assist in
closing changes
RSA Report Daily Report of assigned changes, to manage RSA’s
daily workload
Overdue Changes and Tasks Daily Report for review, with change and task owners,
of their overdue items
New and Removed Hardware/Software Quarterly This report shall include any new or removed
Report hardware asset classes (consisting of vendor,
model/version) or infrastructure software either
added or deleted from the PBGC environment
during the period. This report shall include new
cloud-based IaaS virtual hardware. This report
will include the vendor-provided EoSL data
including dates and recommended
update/replacement items
188
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix D - PBGC Network Overview Diagram
In calendar year 2019, PBGC expects to consolidate the vast majority of its WAN, local phone, and all of
its Internet (MTIPS) circuits under a single GSA EIS contract that includes services for/to the co-located
data centers. Transition to this new vendor will likely run into early calendar year 2020. The 1 GB
backbone currently between HQ, COOP, and Kingestowne will be replaced by a 10 GB wave backbone
connecting the col-located data centers, the HQ, and Knigstowne.
189
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix E - References
Meet all Federal mandates and guidelines regarding IT services and operations. Specifically, PBGC must
meet the requirements mandated in the current rules and regulations listed below.
http://www.whitehouse.gov/news/releas
Homeland Security Presidential Directives HSPD-7
es/2003/12/20031217-5.html
“Critical Infrastructure Identification, Prioritization, and
190
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Reference Documents Source
Protection”
http://www.gao.gov/new.items/d04394g.
GAO IT Investment Management pdf
191
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
http://www.federalelectronicschallenge.
Federal Electronics Challenge (FEC) net/
http://www.archives.gov/records-
Federal Records Act 44 USC Chapters 21, 29, 31 & 33 mgmt/laws/
192
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix F - PBGC Locations
The following table details the PBGC locations as of March 2019, including user counts and
requirements for site support staff:
Site Street Address City State Zip Code User Site Support Comment
Code Count Staff Required
193
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
OFF Off-Site * * * 133 N/A
The following table provides a tentative schedule for adjustments to PBGC locations over the life of
this contract and changes to user counts as well as expected requirements for the O&M contractor:
High-level change Estimated Details
Timeline
Expand EUC site Q3-Q4 FY19 Expand infrastructure and workstation count at EUC site to
support COR and WIL FBA consolidations as well as relocation of
Call Center and Document Management Center services; a total
of 300 staff. Work is already underway and will be complete at
award of this task order.
Close out Coraopolis site Q4 FY19 Current O&M contractor will be responsible for decommission,
collection and shipping of IT equipment back to HQ. User count
will shift to EUC facility. Work is already planned and will be
largely or entirely complete at award of this task order.
Close out Kingstowne site Q3 FY20-Q1 New O&M contractor will be responsible for decommission,
FY21 collection and shipping of IT equipment back to HQ. User count
will shift to EUC facility. Based on current plans, DMC will be out
of Kingstowne by 6/2020 and CCC will be out by 12/2020.
Close out Wilmington site Q3 FY20 New O&M contractor will be responsible for decommission,
collection and shipping of IT equipment back to HQ. User count
will shift to FBA North facility.
194
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Build out new primary data Q4 FY20- Q1 Location TBD (within 60 miles of DC). New O&M contractor will
center at acquired Co-Lo and FY21 be responsible for implementation of a new network core
relocate equipment and services router/switch for connectivity as well as relocation of primary
from HQW to the primary data data center equipment and services from HQW to the primary
center Co-Lo data center DR Co-Lo.
Consolidate and relocate Q4 FY21 – Q4 New O&M contractor will be responsible for initial infrastructure
Headquarters FY22 build-out at location and then subsequent relocation of users at
HQW, HQB, and HQI to new, single GSA leased space at 445 12th
Street, S.W., Washington, D.C.
195
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix G - IT Service Support Guidelines
Priority Table:
196
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix H - IT Infrastructure Maintenance Schedule
The following table details the maintenance windows for production IT infrastructure which have been
agreed to by the PBGC business units:
Additional outages may also be scheduled as needed, but must be negotiated with the PBGC business
units. ITIOD maintains a list of PoCs for each PBGC department with whom such arrangements can be
coordinated.
IT infrastructure maintenance for the PBGC Development and Test environments can be conducted on
the weekends and after normal operating hours (7:00 AM – 5:00 PM, M-F).
197
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix I - OIG IT Infrastructure Summary
Background
The Office of Inspector General (OIG) for Pension Benefit Guaranty Corporation serves as an
independent entity within PBGC. Its mandate is to detect and prevent fraud, waste, abuse, and
violations of law, and to promote economy, efficiency and effectiveness of the PBGC. The OIG strives, as
an agent of positive change, to continually improve PBGC management and program operations by
independently conducting audits, evaluations, and investigations.
IT Infrastructure Summary
The OIG’s local IT infrastructure is typically logically and physically separate from PBGC’s. It is, however,
connected to and integrated with PBGC’s primary production IT Infrastructure for services such as
Internet access and other enterprise-levels needs, e.g. IPAM, email gateways, etc. The OIG generally
purchases and uses the same models, types and versions of IT infrastructure hardware and software as
PBGC does. The OIG largely administers their infrastructure with their own contract staff for day-to-day
activities. They do, however, rely upon the primary IT infrastructure DM&E/O&M contractor to assist
with installation of new technology, modernization of existing technology, and more complicated
technical issues.
PBGC’s Office of the Inspector General (OIG) IT infrastructure environments consists of many
components and tools detailed in the tables that follow:
The following table details the approximate number of Active Directory user accounts in PBGC’s OIG
user and resource domain (oig.ent.pbgc.gov) broken down by function:
PBGC’s OIG has approximately 22 Windows servers, with almost all of them being virtual running on
VMware 6.5. The following table breaks them down by OS:
198
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Total Count 26
An upgrade to Windows Server 2016/2019 is planned for FY19.
The OIG also administers and maintains content for the oig.pbgc.gov web site which is actually hosted in
PBGC’s production environment.
199
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix J - IT Service and Support 2018 Statistical Summary
In 2018, approximately 68% of phone calls to the Service Desk were resolved by the Service Desk. A
further 10% were resolved by Site Support. The remaining 22% were escalated to Tier 2 or Tier 3.
The following table provides the approximate count for tier 1 IT Service Desk interactions processed in
calendar year 2018 broken down by contact type and first call resolution (FCR) for calls made to the
Service Desk. PBGC transitioned to ServiceNow as its operational system for incident management on
January 19th. The process for calculating First Call Resolution was adjusted on March 1st. The following
table is of data between January 19th and December 31st, 2018, except for FCR calculations which are
March 1st through December 31st.
** Received between March 1st and December 31st, but not subject to SLA (typically tickets opened by
someone who is not a member of the Service Desk team).
200
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Incident, RFI, Service Requests, and Change Management Requests
The following table provides the approximate count for tier 2 incidents, requests for information (RFIs),
service/access requests, requests for change (RFCs), and RFC tasks processed in calendar year 2018
(January 19th- December 31, 2018) broken down by service area for work in the scope of this contract:
201
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
The following table provides the approximate count for tier 2 incidents, requests for information (RFIs),
service/access requests, requests for change (RFCs), and RFC tasks processed in calendar year 2018 for
assignment teams outside the scope of this contract. PBGC migrated to ServiceNow for Incident, RFI,
and Change Requests on January 19th; the data shown is here are for incidents and RFCs opened
between January 19th – December 31st, 2018; For Service Request opened between January 1st –
December 31st, 2018; For RFCs with target environment deployment dates in 2018; and for RFC Tasks
scheduled to be completed in 2018:
All service requests processed by ITIOD in 2018 in the table above were fulfilled
automatically/programmatically.
202
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix K - IT Infrastructure Tools List
The following table provides a list of the primary software/tools utilized in providing the services under
this contract:
203
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
engine for access request, approval, and fulfillment as well as
replace SharePoint and other non-IT managed solutions utilized
for annual account and access recertification.
Oracle Access Manager 11g and 12c Used to provide single sign-on authentication, leveraging
Windows native authentication and OID, to WebLogic hosted
applications
Oracle Internet Directory 11g and 12c Supports OAM for application access via single sign-on.
Synchronized with AD utilizing Oracle’s Directory Integration
Platform.
Dell/Quest Recovery Manager for Provides backup and granular recovery capabilities for PBGC’s
Active Directory (RMAD) 8.8 Active Directory
Flexera Admin Studio 11.5 Used for conflict resolution and project management for
application deployment packages
Flexara InstallShield 19sp1 Used for creating .MSI application deployment packages and
.MST transforms
IBM BigFix 9.5 Used for patching and occasional software deployments.
Includes Security and Compliance Analytics (SCA) 1.9 for
compliance reporting.
Oracle Enterprise Manager 11g/13c Centralized administration of Oracle databases and middleware
Oracle Recovery Manager (RMAN) 12c Provides for automated Oracle backup and recovery in
conjunction with Veritas Netbackup
Oracle Data Guard 11g/12c Provides for automated data replication to disaster recovery site
Dataguise 5.1.x Masks sensitive data
Microsoft SQL Admin Studio Centralized administration of Microsoft SQL Server databases
Dell/Quest Toad for Oracle SQL Development Tool. PBGC has Base, Xpert, and DBA
modules.
VMware vCenter 6.5 Centralized administration of vSphere environment
VMware Horizon 7.x Centralized administration of Virtual Desktop Infrastructure
environment
VMware PowerCLI 6.5 R1 Automation of virtual environment using Powershell
Office 365 Admin Portal Government cloud management portal administration
Microsoft Azure Admin Portal Manages InTune and Azure (separate tenants for each)
Sharepoint Designer 2016 Develop Sharepoint content including sites and workflows
Infopath 2013 Develop Sharepoint forms
Concept Search 5.4.x Utilized to manage taxonomy terms and automatically assign the
appropriate metadata tags to content based on clues
established by lines of business. This product also supports
detection and quarantine of improperly posted PII and migrating
records to the appropriate folder in PBGC’s records center.
Tectia SSH 6.3.x Provides for SSH support for secured file transfer
IBM Connect Direct 4.x Used to perform secure file transfers between IBM Mainframes
(off-site) and Unix servers
Hummingbird Exceed 13.x SSH and X-Windows client
204
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Splunk Enterprise Security 7.x Splunk is the enterprise log management and security incident
and event management (SIEM) tool for PBGC. Splunk gives the
visibility into log data from all servers and security devices. The
data within Splunk is used for conducting security investigation
and diagnosing the root cause for operational issues.
HP Business Service Management IT Infrastructure and application availability and performance
(BSM) 9.x monitoring, reporting and alerting
HP SiteScope 11.x IT Infrastructure availability, performance, and capacity
monitoring, reporting and alerting
Think Automation 4.x Enable and support email integration with SharePoint on-line
and perform other business process automation, e.g. automate
upload to SharePoint on-line reports distributed via email
Cisco Prime 3.x Manage and monitor (health, performance, alerts, notifications)
Cisco network devices, run baseline checks, automate archive
configurations, push configurations to multiple devices
simultaneously
Cisco AnyConnect 4.x VPN client
Cisco ISE 2.x Provides network access control (NAC) services
HP Network Node Manager (NNMi) Monitor local area (LAN) and wide area network (WAN)
10.x availability via performance monitoring in an easy-to see
graphical format. It shows the devices relative location and
status.
Plixer Scrutinizer 17.x Provides netflow data as well as network traffic analysis
Pscp ver 0.63 Executable binary scp client program (CLI) to securely
copy/transfer router/switch configurations and upload IOS
images to routers and switches
Tftpd64 Executable binary TFTP client program (CLI) to copy/transfer
router/switch configurations and upload IOS images to routers
and switches that don’t support secure protocols
Wireshark 2.x Captures network traffic to analyze packets for troubleshooting
NEC SV9500 PCPro 2.1.x PBX/ACD Management Software (HQ Campus)
NEC Univerge UM8700 Administration Voicemail management software
8.7.x
NEC SV8300 PCPro 9.0.x PBX/ACD Management Software (FBA sites)
NEC Global Navigator (GNAV) 11.x A NEC tool for reporting and monitoring agent activity for the
ACD phone system
PBGC NEC Phone Database A Microsoft Access database used to document relationships
between phone numbers/extensions, users and locations and
phone ports/cables
Liebert Sitescan 5.2 Provides comprehensive monitoring and control of the PBGC
Data Center and Field Benefit Administration (FBA) facility
support systems
205
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Temp Guard Micro Technologies Data Software used to manage, configure and monitor environmental
Capture 4.2.7 (temperature and water) sensors that cannot be managed with
Liebert Sitescan.
Polycom RMX Manager 8.x Used to setup video conferences and to create and manage
personnel conference bridges.
Microsoft Intune 5.x Portal Cloud-Based mobile device management tool used to reset
passwords, factory reset phones, enroll new users, locate lost
devices, manage business required applications as well manage
mobile device compliancy for approximately 1000 mobile
phones.
Hitachi Navigator HUSVM SAN provisioning
Hitachi Storage Navigator Modular HUS150 SAN provisioning
28.x
Hitachi Command Suite 8.5.x HUSVM monitor and SAN provisioning
Hitachi Command Control Interface Shadow Image and COW
HORCM
Netapp Ontap 9.x NFS, CIFS and SAN provisioning
Netapp OnCommand System Manager NFS, CIFS and SAN provisioning
Netapp Snapshot Snapshot
Netapp SnapMirror Remote replication
Netapp FlexClone Local clone
Brocade Web Tools Manage and monitor Brocade Fibre Channel switches
Veritas Netbackup 8.x Enterprise Backup and restore
Veritas Auto Image Replication(AIR) Replication backup data from HQ to WIL
IBM System Storage TS3310 Tape Manage and monitor IBM tape library
Library
Veritas NetBackup appliance software Will be use on new Netbackup appliance on HQ and WIL for
3.x backup, restore and replication using AIR
HP Application Life Cycle Management Requirements management, test planning and functional
(HP ALM) and (and HP Sprinter) / HP testing, and defect management
Quality Center Enterprise 12.x
HP Unified Functional Testing (HP UFT) Automates functional testing
12.x
HP LoadRunner 12.x Used to test applications, measuring system behavior and
performance under load
Serena Business Manager (SBM) 10.x Process management and workflow automation platform for IT
and DevOps designed to orchestrate and automate processes
and provide transparency across an organization
Serena Version Manager (PVCS VM) Code repository and version control system
8.x
Oracle BI Publisher 11.x Enterprise reporting tool
Tableau 10.x Enterprise business intelligence tool
206
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
DOJ Cyber Security Assessment and Used to document FISMA systems, associated controls, control
Management (CSAM) status, and control inheritance, track POA&Ms, etc.
The list of the primary software/tools utilized in providing the security services under this contract will
be made available in PBGC’s reading room.
207
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Appendix L - IT Infrastructure Program Registry and ITIOD Roadmap
The following is an export from the IT Infrastructure Program Registry which is used to record all IT
infrastructure development, modernization, and enhancement projects and activities greater than 40
hours:
IPv6 Readiness Assessment Procure Cisco Assessment Services for IPv6 Activity 10/20/2015 09/20/2016
and Sequencing Plan
Enterprise Identity Establish new Enterprise Identity Project 01/01/2016 12/15/2019 01/01/2020
Management Management Tool and Services per the results
of the ICAM BNA analysis performed in FY
2015.
Oracle Fusion Middleware Oracle Fusion Middleware Upgrade Project 03/14/2016 09/06/2019 09/30/2019
Upgrade (wls/soa(bpel)
Oracle 12c RDBMS Upgrade Oracle 12c RDBMS Upgrade Project 03/14/2016 09/30/2019 10/31/2019
JRE 8u192 Upgrade Upgrade to JRE version 8 wherever possible, Activity 05/02/2016 07/31/2018 03/29/2019
Servers/Workstations (FY17- and patch legacy versions of JRE if still
FY19) required
Network and Local Printer Standardize and consolidate on a limited Activity 08/25/2016 03/31/2019 03/31/2019
Fleet Standardization number of network and local black and white
and color printer models and establish
associated support model
Headquarters Relocation Headquarters relocation planning and support Project 10/02/2016 12/31/2019 12/31/2019
Planning and Support
Developer Tools Developer Tools Consolidation and Project 10/03/2016 04/19/2019 06/30/2019
Consolidation and Modernization effort to identify a PBGC
Modernization standard for development and release tools
that will support Agile and Continuous
Integration (CI), Data Integration, and
Application Performance Monitoring and
Management (APM). The tools identified
should facilitate use of more modern, cost
effective platforms and technologies including
Cloud and DevOps.
FBA Consolidation Consolidate the existing FBAs into two or Project 06/01/2017 06/15/2019 09/30/2021
three facilities
ITSM uCMDB and Asset Tools Consolidate/Replace ITSM, uCMDB, and asset Project 06/08/2017 02/28/2019 02/28/2019
Replacement (Service Now) tools with new Service Now cloud-based
solution
Telephony Infrastructure This project will replace the legacy telephone Project 06/23/2017 09/30/2019 09/30/2019
Modernization systems at all PBGC locations including the
FBA sites. PBX's and telephone handsets will
be replaced with Omni-channel VoIP capable
systems to support Infrastructure and
Business systems.
208
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Cisco Switch Replacements Replace all EOSL 6500 and 3750 closet Project 06/23/2017 10/31/2018 05/31/2019
(FY18) with Wireless and switches. Includes Cisco ISE, Cisco Prime and
Prime and Cisco ISE Wireless AP Implementation in Scope.
RHEL 7 Upgrade (FY18/19) Upgrade RHEL 6.x (and older) servers to the Project 09/29/2017 09/06/2019 09/30/2019
latest version of RHEL in advance of EoSL.
Data Center Colocation and Acquire and migrate to co-location centers for Project 10/02/2017 08/31/2020 09/30/2020
Consolidated 10GB WAN and primary and disaster recovery IT
TIC implementation infrastructure data center services and
upgrade, consolidate WAN infrastructure to
10 GB Wave, and implement high-speed TIC
SAN Infrastructure Brocade SAN Fabric Modernization and Project 06/15/2018 04/15/2019 05/15/2019
Modernization (FY18/19) Storage Array Consolidation (6 arrays to two),
Storage Expansion and Encryption
Symantec Netbackup Activity 06/20/2018 05/15/2019 05/15/2019
Appliance and Capacity Symantec NetBackup Appliance Capacity
Upgrade (FY18/19) Upgrade including addition of 5 Media Shelves
and 60 FETB licenses as well as enablement of
disk encryption.
HP and Serena TCO Tools HP and Serena TCO Tools Annual Activity 06/30/2018 07/30/2019 08/05/2019
Annual Upgrade (FY18/FY19) Upgrade (FY18/FY19)
Data Loss Prevention (DLP) Establish automated systems/tools at PBGC's Project 08/01/2018 06/15/2019 07/15/2019
(FY-19) perimeter to prevent intentional and
unintentional transfer of PBGC data outside of
PBGC's network.
Mobile Phone Solution Recompete mobile/cellular services contract Activity 08/06/2018 08/30/2019 09/30/2019
Update/Refresh (FY19) and replace/upgrade mobile devices .
Network Perimeter and IPAM Modernize Firewalls, Remote Access Activity 08/15/2018 04/12/2019 04/12/2019
Modernization Appliances, Proxies, and IPAM
devices. Replace external DNS and DNSSEC
Service Provider DataMountain with
Oracle/Dyn DNS.
CDM Task Order 2F (CSM, Implement DHS's CDM Task Order 2F Project 08/29/2018 11/15/2019 11/30/2019
SWAM, VUL, HWAM) (CSM,SWAM,VUL,HWAM).
Windows 2016/2019 Server Upgrade Windows 2012 R2 (and older) Project 09/01/2018 03/16/2020 03/31/2020
Upgrade (FY19) servers to the latest supported MS Windows
Server version (2019 where possible, 2016
where necessary) in advance of EoSL.
Web and DB Vulnerability Establish web vulnerability scanning using Activity 09/01/2018 05/19/2020 05/19/2020
Scanning and Remediation Acunetix and database vulnerability scanning
using AppDetective Pro and adjust patch and
vulnerability management processes, SLAs,
and performance metrics as required to
addressed additional findings
Splunk Business Adoption Support business units to adopt Splunk for Activity 10/01/2018 03/16/2020 03/31/2020
their FISMA systems by generating Splunk
reports, alerts, and/or dashboards to help
address their auditable events.
209
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
Security Tools Annual Perform annual upgrade of security tools to Activity 10/01/2018 09/30/2019 09/30/2019
Upgrade (FY19) the latest major version: Symantec Endpoint
Protection (SEP), Splunk, IBM BigFix, Cisco
FirePower, and Tenable SecurityCenter in
FY19.
Active Directory Upgrade and Upgrade to AD and Cleanup of GPO Activity 10/10/2018 09/30/2019 09/30/2019
Cleanup (FY18)
ITIOSS Contract Re-compete Prepare for and award service contract for the Activity 01/01/2019 11/01/2019 12/31/2019
(FY20) full array of O&M and D,M&E services for
FY20 and beyond.
Annual Asset Inventory (FY19) Conduct Annual Physical Inventory for FY19 Project 03/08/2019 09/01/2019 09/15/2019
for PBGC
Cisco SourceFire IDS/IPS Upgrade Cisco SourceFire IDS/IPS in advance Activity 05/01/2019 05/01/2020 05/15/2020
Upgrade of EoSL
HP and Serena TCO Tools HP and Serena TCO Tools Annual Activity 06/30/2019 11/30/2019 11/30/2019
Annual Upgrade (FY19/FY20) Upgrade (FY19/FY20)
JRE Upgrade Upgrade to current version of JRE on servers Activity 09/18/2019 01/31/2020 01/31/2020
Servers/Workstations (FY20) wherever possible, and patch legacy versions
of JRE if still required.
Security Tools Annual Perform annual upgrade of security tools to Activity 10/01/2019 09/30/2020 09/30/2020
Upgrade (FY20) the latest major version: Symantec Endpoint
Protection (SEP), Splunk, IBM BigFix, Cisco
FirePower, and Tenable SecurityCenter in
FY2020.
Headquarters Relocation Build out GSA leased facility and relocate staff Project 01/01/2020 07/01/2021 08/01/2021
and end-user IT equipment there
Annual Asset Inventory (FY20) Conduct Annual Physical Inventory for FY20 Project 03/08/2020 09/01/2020 09/15/2020
for PBGC
HP and Serena TCO Tools HP and Serena TCO Tools Annual Activity 06/30/2020 11/30/2020 11/30/2020
Annual Upgrade (FY20/FY21) Upgrade (FY20/FY21)
Upgrade Cisco Nexus Core Upgrade Nexus 7000 and 7010 Cisco Nexus Project 07/01/2020 03/30/2021 03/30/2021
Routers Routers in the datacenter.
JRE Upgrade Upgrade to current version of JRE on servers Activity 09/18/2020 01/31/2021 01/31/2021
Servers/Workstations (FY21) wherever possible, and patch legacy versions
of JRE if still required.
Security Tools Annual Perform annual upgrade of security tools to Activity 10/01/2020 09/30/2021 09/30/2021
Upgrade (FY21) the latest major version: Symantec Endpoint
Protection (SEP), Splunk, IBM BigFix, Cisco
FirePower, and Tenable SecurityCenter in
FY2021.
Annual Asset Inventory (FY21) Conduct Annual Physical Inventory for FY21 Project 03/08/2021 09/01/2021 09/15/2021
for PBGC
HP Server Modernization Modernize the HP server infrastructure which Activity 01/01/2022 09/15/2022 11/01/2022
(FY19/20) includes all blade infrastructure and servers,
and standalone servers.
Annual Asset Inventory (FY22) Conduct Annual Physical Inventory for FY22 Project 03/08/2022 09/01/2022 09/15/2022
for PBGC
210
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order
The following is the ITIOD Roadmap, a visual representation of the IT Infrastructure Program Registry for
items of significant end-user or PBGC business impact (as of March 2019):
211
Pension Benefit Guaranty Corporation
IT Infrastructure Operations Support Services (ITIOSS)
O&M and DM&E Task Order