2005, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org).

Reprinted by permission from ASHRAE Transactions, Volume 111, Part 2. The material may not be copied nor distributed in either paper or digital form without ASHRAEs permission.

DE-05-8-3

An Owners Perspective on Commissioning of Critical Facilities

Terry L. Rodgers
Member ASHRAE

ABSTRACT This paper provides an overview of various commissioning issues associated with the design, construction, and testing of mission critical facilities typical of data center and telecom industries from the owners perspective. It first discusses commissioning and the differences between traditional startup and testing and formal commissioning. It then discusses the various steps associated with commissioning typical office buildings. The paper extends this discussion to address integrated systems testing of critical infrastructures to provide uninterrupted services to support mission critical operations. This is followed by a brief discussion on the limitations and potential conflicts of commissioning and cost considerations from an owners perspective. INTRODUCTION Building commissioning has been practiced for many years and much has been written regarding the processes and procedures associated with bringing a new facility into operation. The commissioning of mission critical facilities (MCF) encompasses the same philosophy and takes it further and to a greater depth. The primary beneficiary of the commissioning process is the owner. It is important that the owner play an active role in ensuring that the methodology for commissioning their facility is tailored to their specific needs. From the owners perspective, MCF commissioning is more of a process than a procedure and is as much a strategy as it is a program. ASHRAE Guideline 1-1996, The HVAC Commissioning Process, defines commissioning as the process of ensuring that systems are designed, installed, functionally tested, and capable of being operated and maintained to perform in
Terry L. Rodgers is senior engineer at Fannie Mae, Washington, DC.

conformity with the design intent and goes on to state that commissioning begins with planning and includes design, construction, start-up, acceptance and training, and can be applied throughout the life of the building (ASHRAE 1996). The Model Commissioning Plan and Guide Specifications, version 2.05, published by the Portland Energy Conservation, Inc. (PECI) (initially sponsored by the US Department of Energy) defines commissioning as
a systematic process of ensuring that building systems perform interactively according to the design intent and the owners operational needs

and that this is achieved

beginning in the design phase by documenting the design intent and continuing through construction, acceptance, and the warranty period with actual verification of performance, operation and maintenance (O&M) documentation verification and the training of operating personnel (PECI 1998).

Ultimately, it is the owners responsibility to define and communicate the facilitys intent and performance requirements to the project team and to ensure the project delivers what is expected. The owners program document, basis-ofdesign document, and the project commissioning plan are the owners best means to accomplish this. The owners program document captures the facilitys intent (mission) and performance requirements. ASHRAE Guideline 1-1996 defines the owners program as the document that outlines the owners overall vision for the facility and expectations on how it will be used and operated. The basis-of-design document captures the relevant physical aspects of the facility to achieve the performance requirements in support of the mission. The commissioning plan then defines the verification and testing

618

2005 ASHRAE.

process to ensure the project delivers what is expected, including training and documentation. The owner should scrutinize every aspect of developing the design intent, basis-of-design, and commissioning plan documents, as these establish the foundations upon which the project contracts, roles, and responsibilities are built. There is an increasing interest by owners in achieving LEED certification for their facilities. There is great potential for synergy in combining LEED certification efforts with commissioning programs. Both processes document and validate program and design intents against defined performance criteria. Though typically associated with new construction projects, commissioning can also be performed on existing facilities following expansions, renovations, or maintenance activities. This is sometimes referred to as recommissioning. Owners should consider incorporating aspects of commissioning into their normal operations and maintenance programs as a form of continuous commissioning to ensure that the installed systems and facility operations function and perform to reliability and availability expectations. COMMISSIONING LEVELS Commissioning spans the spectrum from the testing of individual components prior to leaving their place of manufacture, to simulating catastrophic events upon entire facilities, and ensuring a myriad of integrated systems perform such that expected responses result in a predicted outcome. The owner should ensure the project establishes a commissioning team that ideally is led by an independent, third-party commissioning agent. The commissioning team should include owners representatives who have expertise in the operation, maintenance, and management of similar facilities. This will ensure the perspective of long-term facility performance is represented. Years after the project is completed and final invoices processed, the owner will rely on the O&M staff to deliver the performance per the design intent. A very important aspect of formal commissioning is in compiling, organizing, and recording critical component, equipment, system, and facility information and data in centralized locations to be maintained for the life of the facility. System manuals that compile information regarding system performance ratings (psi, cfm, rpm, etc.), sequences of operations for normal, standby, emergency, or other defined modes of operation, maintenance requirements and procedures (spare part lists, maintenance frequencies, critical spares, etc.), and even installation data (vendor contact info, warranty info, etc.) help ensure that operational considerations are not lost during the construction phase. Development of these manuals should begin during the design phase and should be maintained for the life of the facility. New software systems called building information management systems (BIMS) are being developed specifically for capturing all relevant details of a building or facility that can cross-reference CAD drawings, soft copy documents, locations, spare parts lists, vendors, operating and maintenance procedures, etc. The
ASHRAE Transactions: Symposia

GSAs Federal Facilities Committee on Emerging Technologies has partnered with Stanford University to promote and pilot BIMS technologies and methodologies where facilities are constructed virtually as part of the design process prior to physical construction. Commissioning activities have been characterized into five broad categories, or levels (Table 1). Level 1 commissioning refers to the testing of products prior to leaving their place of manufacture, and they are sometimes called factory acceptance tests. For critical equipment, the owner should review the scope and methodology of the proposed test to ensure it encompasses the maximum assurance possible relative to the intended use when installed. The owner should consider all Level 1 commissioning requirements prior to or at least concurrent with the product selection process to ensure additional costs are not incurred after the sales agreement is in place. Most savvy manufacturers and/or suppliers are willing to modify or expand the scope of tests to meet project specific requirements, provided it is within their existing capabilities. An example would be to request that motors and other rotating components meet specific vibration acceptance criteria. This can detect latent bearing defects or minor misalignment conditions that can then be easily resolved prior to shipping. If the manufacturer needs to hire an alignment and balancing specialist to perform these tests, the costs can be negotiated into the purchase costs. The owner should require witnessing of critical equipment tests by an owners representative. This has several benefits. Obviously witnessing ensures the tests are performed as required, but there is also an opportunity to learn about the product. As good as product cutsheets and catalogs can be, actually seeing, listening, and observing the equipment under operating conditions can provide insight otherwise not provided. Questions regarding design, operation, maintenance, product support, and customer support can be addressed at the source, and meeting the manufacturers engineering and management staff can result in valuable contacts for future information and advice. Level 2 commissioning refers to field component verification. Products are inspected and verified upon receipt by the owner (or owners representative) to be as specified and without damage when delivered to the point of use. Again, the scope of this phase of commissioning should be defined and expressed in the commissioning plan. For large quantities of noncritical, inexpensive, and short lead-time items, random visual inspection for obvious damage, correct quantity, and specified nameplate data may be sufficient. For critical, expensive, or long-lead time items, a much more thorough and comprehensive verification should be considered. Procurement specifications should be reviewed and validated against design documents to ensure the receiving staff understands exactly what to expect and who to notify when issues arise. The receiving process should be defined and communicated to the receiving staff, and adequate and appropriate on-site transportation and storage should be in place to ensure the product is secured and protected until ready for installation. Even if
619

Table 1.
Level 1 Name Factory Acceptance Tests

Categories of Formal Commissioning Activities

Description Cost Implications Additional costs to the manufacturer are passed on to the customer in the purchase price. These should be negotiated in advance and included in the purchase order. Owner should also consider travel and living costs when witnessing is required. Costs are borne by the owner, directly or through the receiving contractor. Cost avoidance justifies thorough verification efforts. Owners should specify a clear level of effort and deliverables for each participating contractor and include this in their scope-of-work and pricing agreements. It pays to inspect what you expect. Costs are proportional to the defined level of effort and are typically more than traditional start-up and testing. Owners should budget additional time and money to resolve performance issues revealed during testing. Costs are influenced by the size, complexity, and interrelatedness of the facility.

Product testing and verification to ensure compliance with manufacturers advertised specifications, ratings, and characteristics. These can be augmented to include any owner-defined functional, performance or aesthetic requirements agreed to in the purchase agreement. This can include component certifications or documentation. Sometimes referred to as Factory Witness Tests when the tests must be performed in the presence of an owners representative. Inspection, verification, or tests performed on the products upon delivery to the site to ensure the products delivered match those purchased and tested during the Factory Acceptance Tests and have not been damaged or altered during shipment.

Field Component Verification

System Field inspections and certifications that components are assemConstruction Veri- bled into systems as required by plans and specifications. fication Includes verifying compliance with specified means and methods, accessibility, maintainability, and manufacturers installation requirements and directives. Site Acceptance Testing Demonstration that related components, equipment, and ancillaries of a defined system operate and function to acceptance criteria. This should include normal, maintenance, and emergency modes of operation, verifying settings, safeties, and capacities, and performance of associated monitoring and control functions. The functional performance testing of interrelated components and systems to respond as intended to expected and unexpected anomalies.

Integrated Systems Testing

contractors accept responsibility for ordering, shipping, delivery, or receiving problems, the project schedule can still be impacted, which ultimately costs the owner time and money. After all, the owner needs the facility to perform the mission, so project delays have intangible lost business costs beyond those directly attributed to the design and construction project. Level 3 commissioning refers to field inspections and certifications that components are assembled into systems as required by plans and specifications and may be referred to as system construction verification. These formal progress inspections should be coordinated and documented by the construction manager, led by the architectural and engineering firm(s), and require participation by the owner, contractors, and equipment suppliers. This phase can also include test-fits, mockups, and other work layout. The owner should have identified operations and maintenance (O&M) staffing requirements by this time and ideally put the assigned staff on site at the facility. This allows the O&M staff to become familiar with layouts and configurations and to provide input on maintenance access, stocking and inventory control of spares, review of installation and O&M manuals, etc., and the opportunity to question and learn from the design team, construction and installing contractors, and the commissioning agent on how the facility is expected to operate. Level 1 through Level 3 commissioning is to a large extent focused on the component, assembly, and equipment aspect
620

and ensuring they are procured, received, and installed in accordance with the design documents. Level 4 commissioning is commonly called site acceptance testing and refers to demonstrating that related components, equipment, and ancillaries that make up a defined system operate and function to rated, specified, and/or advertised performance criteria. From the owners perspective, this is where the contractual roles and responsibilities become most important. Level 4 commissioning requires all affected parties to understand and agree on how these assembled and installed components are to function and perform as a system. Any vagaries in scope and responsibilities become apparent as discrepancies and issues are frequently followed by finger pointing and debates about who pays for the resolution. A well thought out commissioning plan during the programming phase can minimize this by ensuring the design intent and basis-of-design specifications result in contract documents with clear delineation of roles and responsibilities. This includes defining participation in the commissioning planning and activities, boundaries between trades, coordination of activities, and assignment of responsibilities for meeting performance requirements. Since all parties work either directly or, in the case of subcontracts, indirectly for the owner, the owner has the most control over the success of this phase of commissioning by
ASHRAE Transactions: Symposia

ensuring clear and detailed contracts are in place without gaps between roles and responsibilities. Level 5 commissioning is called integrated systems tests where redundant and backup components, systems, and groups of interrelated systems are tested for their response to expected and unexpected anomalies. The challenge of Level 5 commissioning is to ensure that all methods of failure are considered in advance and appropriately tested to demonstrate that the response meets the original programming intent and expectations. The building automation system (BAS) can play a vital role as a commissioning tool, especially during Level 4 and Level 5 tests. Obviously the BAS system itself must first be subjected to a rigorous commissioning process (to ensure valid and accurate data collection) prior to use in commissioning other systems. This creates a project critical-path challenge in that the BAS system typically lags the other systems design and installation phases. You have to have selected and designed the critical infrastructure before you can complete the BAS design, and you have to have the infrastructure and related equipment installed before you can complete the BAS installation. Yet you need to start up and commission the BAS system first before you can utilize it to support commissioning of the monitored and controlled systems. TYPICAL OFFICE BUILDING COMMISSIONING All building (and facility) construction projects have required aspects of the Level 1 through Level 4 quality control processes described above to provide the owner assurance that the project was completed according to the design. Commissioning refers to the formalization of each of these quality control processes into a phased quality assurance program with supporting documentation and accountability, ideally by an objective third-party entity. At the component level most manufacturers perform quality control inspections or tests inherent to their production processes (Level 1). A component in this context can vary from individual parts to fully assembled and ready to use equipment. It follows that the associated quality control programs vary from random spot-checking of widgets to sophisticated and comprehensive functional performance tests. Most products are received upon delivery in that they are not only taken possession of but are inspected for correct quantity, size, or rating and are visibly undamaged (Level 2). The receiving department checks the packing sheets against the order and calls the end-user. The end-user verifies the shipment matches the design and construction needs. Again, the scope and formality of the quality control (receiving) process can vary greatly. During a major building automation system installation, the vendor-furnished operator workstations were to be factory configured to exacting specifications defined by the contract documents. The PC configurations were not adequately verified prior to shipment nor upon receipt and installation.
ASHRAE Transactions: Symposia

Months after the system had completed acceptance testing and had been in operation, network errors and lock-ups began occurring. After significant effort in troubleshooting and analyzing the anomalies, the root cause was that the PCs were configured with dynamic instead of static IP addresses. As a result, every time a workstation was rebooted, the network recognized it as a new device and left the old IP address active. The result was the network thought there was an evergrowing list of devices to communicate with and that the majority of the addresses were not responding. This example demonstrates a very important aspect of Level 1 and Level 2 commissioning: component level issues are not necessarily identified during system level tests and can result in latent anomalies waiting to happen. Progress inspections are typical on most construction projects where the contractor, architect, engineer, construction manager, and/or the customer provide quality assurance and identify and resolve issues as they arise (Level 3). Level 3 commissioning benefits both owners and contractors. Formal commissioning plans can be very valuable even on small upgrades and retrofit projects. For example, during a centrifugal chiller replacement the installing mechanical subcontractor: set the new chiller such that a vibration isolator was compressed to metal-to-metal contact, electrical conduit used mechanical instead of the specified compression fittings, and one of the four main water pipes did not include a flexible connector.

The work was completed over a weekend without an owners representative present. As a result, the contractor was reluctant to correct the errors and tried to offer credit back instead, and the owner had to negotiate an acceptable resolution. Ultimately: the chiller vibration isolator was replaced and the chiller leveled, the missing flex-connector was installed, and since the conduit fitting met code and would have required a complete rework of the electrical installation at great cost to the contractor, the owner reluctantly agreed to a modest credit back to cover the difference between the labor and materials to install compression fittings vs. the mechanical fittings.

A formal commissioning team and plan would have ensured these specifications were communicated to all parties, inspected for compliance during receiving and construction, that the owner received what was expected, and that associated schedule delays were avoided. As systems are completed, or when the facility is deemed ready for operations (sometimes called substantially complete), the systems or facility undergo the traditional start-up and testing where the owner is shown by demon621

stration that the system or facility is ready for operations and/ or occupancy (level 4). It is not uncommon with traditional start-up and testing for the installing contractor and/or manufacturers representative to perform some pre-operational checks, energize the system, take some readings and measurements to compare with rated (nameplate) values, and so long as nothing trips or exceeds nameplate the tests are considered passed. An independent air and water balancer typically attempts to adjust hydronic and air flows to within plus or minus ten percent of design conditions, and the systems are turned over to the owner for acceptance. The same may occur for electrical systems, with a specialist setting breaker trip settings, etc. Level 4 commissioning activities differ from traditional start-up and testing mostly in the level of detail, comprehensiveness, and formality associated with the systems tests. Safeties are verified and tested by simulating or forcing failure modes to occur and ensuring the system responds appropriately. Breakers are subjected to injection tests that prove that when trip setpoints are reached, the breaker responds (opens) appropriately. Some critical breakers may even be required to prove they open in less than a required maximum time. HVAC systems would be required to respond to changes in heat loading while maintaining environmental conditions within a predefined tolerance. Chiller operations and safety shutdowns should be tested, including when trip settings are exceeded (water flow, gas pressure, motor amps, etc.) and observing that the chiller plant responds as designed. This could include automatic activation of emergency chilled water storage systems and the start-up of redundant chillers, all within preset times and with chilled water supply flows and temperatures being maintained within specified tolerances. COMMISSIONING MISSION CRITICAL FACILITIES Mission critical facilities typically include more demanding performance requirements regarding how they are to respond to expected and unexpected anomalies without impacting critical operations. Systems usually include redundant components and utility feeds and excess capacity or backup systems or equipment that, during emergency modes, can be automatically or manually activated to protect the mission. These redundant or backup components, systems, or groups of interrelated systems are tested during Level 5 integrated systems tests. The challenge of Level 5 commissioning is to ensure that all methods of failure are considered in advance and appropriately tested to demonstrate the facility meets the original programming intent and functional performance expectations. It is Level 5 commissioning that sets mission critical facilities apart from typical office building type commissioning programs. A data center has a data processing and information storage mission. The mission criticality establishes the expected uptime requirements (99.99%, six-sigma, 24 7, etc.) This, in turn, drives the programming and design criteria for the facility (hardened, the Uptime Institute
622

Tier classification, N+1, etc.) (Turner and Brill 2001). Before an effective Level 5 commissioning plan can be developed it is fundamental that acceptance criteria be established that are founded in ensuring the facility can perform reliably to protect the mission from expected internal and external anomalies. For Level 5 commissioning to be successful, it should begin with the project programming phase where the effort to be expended on quality assurance is weighed against the criticality of the mission, the complexity of the facility, and the resources available. Inevitably, as the project progresses and pressures mount to control costs and improve the schedule, the commitment to commissioning will be tested. Level 5 integrated systems testing is where the science of commissioning approaches art in that creativity, insight, and innovation can determine the level of success. The quality assurance sought and expected from Level 5 commissioning begins with defining the mission, the criticality, and the facility. The purpose of the operation defines the mission. The performance goals (availability and reliability) establish the criticality. And the facility is the physical aspect of where the operation is to occur. It is the owners responsibility to define the mission and its criticality, and then to assemble the project team to define the facility. Level 5 commissioning is also the most important aspect from the owners perspective in that this is where the facility is shown to be capable of meeting the design intent and program requirements. As such, the owner should, as Steven Covey teaches in The Seven Habits of Highly Effective People, begin with the end in mind (Covey 1989). From the programming and design phase and throughout the project, the integrated systems tests should be continually reviewed and analyzed to ensure the tests are comprehensive in verifying the performance expected. It is simple to state a hydronic system will be tested to ensure continuous flow and then script a test that fails an operating pump to ensure a redundant standby pump automatically starts. But consider how many ways a pump can fail with regard to how the test is scripted. Will the pump fail by an electrical fault, bearing failure, broken coupling, or failed-open relief valve? Will the pump failure be sensed by high current, loss of voltage, open or tripped breaker, flow-meter, or low pressure? Is the sensor normally energized or de-energized?

All of this needs to be considered such that a holistic approach is made to ensure all failures are encompassed by the commissioning test for the owner to get real assurance that the facility meets expectations.
ASHRAE Transactions: Symposia

Examples of Level 5 commissioning scripts would include loss of normal power tests coupled with UPS and emergency generator operations. UPS (uninterruptible power supply) systems may be load-tested from incoming utility power down to the remote power panels (RPPs), including the UPS modules and power distribution units (PDUs). Not only would the system be subjected to full loading via load banks, but the system would be tested to respond to various failure modes such as loss of incoming power (battery run for static UPSs), module failures (step load increases for multiple module configurations), and other similar performance tests. The heat rejected by the load banks can provide cooling loads for the associated HVAC and equipment cooling systems. The BAS monitoring, control, and alert systems are verified as well. A good strategy is to overlay as many interrelated systems tests as possible. This is why a team approach is critical to Level 5 commissioning. It requires the combined insight and expertise of professional design engineers to represent the design intent, the manufacturer to represent the specific failure modes and responses at the component and equipment level, the installing contractors to represent the interfacing of varied and distinct components, and the owners O&M staff to represent the longterm considerations such as major repairs, expandability, and operational training and documentation. This is also why Level 5 commissioning needs to begin during the programming and design stage. Many times during the development of acceptance tests where failure modes and scenarios are considered, design weaknesses are revealed and identified. By writing the commissioning scripts concurrent with design and even early construction, the project has more flexibility to incorporate design change with minimal budget and schedule impacts. LIMITS AND CONFLICTS IN COMMISSIONING Ideally commissioning is a win-win process for the project team that comprehensively tests and demonstrates that the facility is delivered as defined by the owners program and basis-of-design documents. In practice there are limitations to the comprehensiveness of the tests to encompass all conceivable means of failure, and there is potential for conflict among participants in performing the commissioning and especially in resolving the identified discrepancies. The best means to minimize and avoid conflicts is for all project stakeholders to maintain an active and meaningful participation during the entire commissioning process from programming through final acceptance testing. All parties must understand and buy in to both the owners program and the basis-of-design documents and participate in developing the associated commissioning plan. Communication becomes key to success. In one data center upgrade and renovation project, the owners program required the facility be capable of supporting the expected business needs for the next six years. The basisof-design document clearly stated the final average watts per
ASHRAE Transactions: Symposia

square foot capacity for the raised floor area. This was followed by a discussion of how the value was based on the total nameplate rating of expected equipment loads then derated by a factor that represented the actual configuration of similar equipment currently in use. The analysis was further broken down into year 1 load predictions, year 3 load predictions, and year 6 load predictions. The supporting heat-load calculations showed a final value that was approximately 80 percent of the nameplate rating. The facility upgrades were designed and constructed to support the de-rated projections to support the year 6 loads. Eventually, the owner identified the discrepancy after analyzing localized hot spots following some computer equipment installs. The owner also realized that the raised floor perforated tiles were installed and configured to support a fully populated data center at the year 6 projected loads versus the year 1 loads. The owner, at his own expense, eventually had to relocate several computer room air conditioners (CRACs) and reconfigure the perforated tiles to resolve the localized hot spots. Even though the owner had representatives involved in the entire computer load estimating and projection process and the projected final watts per square foot values met the owners program requirements, the misunderstandings caused stress and hard feelings between the owners facilities department, IT department, IT technology consultant, and the engineering design firm. An example of commissioning that failed to encompass all failure scenarios follows. All CRAC units at a site were required to have dual power feeds to separate motor starters with an integral automatic transfer switch (ATS). Phase-monitoring would sense the loss of normal power and transfer power to the alternate source. This was tested during commissioning by failing the normal power and ensuring automatic transfer of power and automatic restarting of the CRAC units on the alternate power source. Sometime after the site was in full operation a portion of the site electrical service lost one of the three phases of incoming power. The CRAC units were only sensing power on one phase (which happened to not have been the impacted phase). Since the ATSs failed to sense a loss of power, they maintained operation with only two of three phases available. The main switchgear eventually tripped the impacted service, but several CRAC starter coils were damaged and all affected CRAC units failed to automatically transfer power. Another example of a Level 5 commissioning oversight occurred when a facility underwent a building automation system anomaly that required the rebooting of the control system software. When the system rebooted it initially loaded in default values, one of which was system disabled for the critical chiller plant supporting the data center loads. Not only did the entire critical chiller plant shut down, it did so without generating any central plant alarms (since the system status matched the command state). Quick thinking by the chief engineer resulted in identifying the failure and commanding the point back to system enabled state, averting an embarrassing facility impact. As it turned out, the commissioning plan had not included testing of a complete shutdown and reboot of the BAS system.
623

COMMISSIONING COSTS Level 1 activities typically occur at the manufacturers expense and except where witnessing by the owner is required, or where testing is specified beyond the standard manufacturers quality assurance/quality control (QA/QC) process, the costs are included in the equipment pricing. Level 2 activities, or receiving type inspections, require some additional paperwork to formalize what are typical receiving processes and procedures and carry relatively small additional costs. Level 3 activities, where construction managers, architects, and/or engineers perform progress inspections to ensure means and methods are performed according to design specifications and drawings and with acceptable quality, are also typical costs incurred and included in construction projects. Formal commissioning plans tend to increase the level of scrutiny, documentation, and accountability of these efforts and so incremental increases in both time and costs should be expected. Obviously, Level 4 commissioning requires significantly more planning, more detailed procedures, more time and manpower, and, of course, higher overall costs, than traditional start-up and testing activities. The assurance resulting from Level 4 commissioning that critical systems will perform safely and as intended is necessary and essentially a prerequisite to attempting Level 5, integrated systems level testing. As with any worthwhile testing of new systems, failures can occur, resulting in delays and remediation (varying from adjustments, to repairs, to even redesigns and modifications). If this were to occur during Level 5 commissioning, not only would a greater number of people and, therefore, manpower (and associated costs) be impacted, but the failure of one system can conceivably cascade into even more severe failures of dependent systems with corresponding impacts to project schedules and costs. The project budget and schedule should include contingencies (time and money) during the Level 4 commissioning to address and resolve discrepancies and unexpected re-work. For Level 5 commissioning to produce positive results, it first requires a high degree of Level 1 through Level 4 commissioning as a foundation. Just as it is better to identify faulty components prior to shipping, to identify damaged deliveries prior to installation, and to identify installation errors prior to system start-up, it is better to identify system-level issues prior to attempting integrated systems testing. Also, as the costs of commissioning efforts tend to increase (especially from the owners perspective) as the level increases. Level 5 commissioning can be quite expensive in both time and money. For a large tier-3 or tier-4 data center (as defined by the Uptime Institute), performing mission critical operations requiring continuous availability and Class 1 environmental conditions (as defined by ASHRAE [2004]), commissioning can amount to 1% or more of the overall design and construction costs.
624

THE OWNERS PERSPECTIVE Based on the above sections, several general recommendations can be made about owner involvement that will help to provide for an effective commissioning process. Table 2 summarizes these recommendations. SUMMARY Most facilities undergo some degree of commissioning as part of the owners acceptance process. Mission critical facilities require more thorough and comprehensive testing to ensure continuous operations. MCF commissioning requires demonstrating that redundant, backup, and interrelated systems automatically recognize and respond to expected and unexpected anomalies. Costs of formal commissioning can be significant. They will increase as a percentage of the overall project budget as the criticality and complexity of the site infrastructure increase. This is especially true of the Level 4 and Level 5 commissioning activities. The higher the owners expectation that the facility perform reliably to support a critical mission, the more effort and resources should be dedicated to a thorough and comprehensive commissioning program. The more sophisticated and complex the facility and associated infrastructure, the more complex and demanding the commissioning program becomes. As with any effective quality assurance program, commissioning will ultimately pay far more dividends than the initial investment, provided the investment is managed wisely. REFERENCES ASHRAE. 1996. ASHRAE Guideline 1-1996, The HVAC Commissioning Process. Atlanta: American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. ASHRAE. 2004. Thermal Guidelines for Data Processing Environments. Atlanta: American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. Covey, S.R. 1989. The 7 Habits of Highly Effective People: Powerful Lessons in Personal Change. New York: Fireside. PECI. 1998. The Model Commissioning Plan and Guide Specifications, version 2.05. Portland Energy Conservation Inc. http://www.peci.org/library/mcpgs.htm. Turner, W.P. IV, and K.G. Brill. 2001. Industry standard tier classifications define site infrastructure performance. The Uptime Institute. http://www.upsite.com/TUIpages/ whitepapers/tuitiers.html. DISCUSSION Jack Glass, SVP-DC Planning, Citigroup, New York, N.Y.: How can the owner be protected from additional costs when level S testing uncovers a problem in one vendors equipment that requires re-testing other vendors equipment?
ASHRAE Transactions: Symposia

Table 2.
Level Name

Recommended Owner Activities for Commissioning of Mission Critical Facilities

Description of Recommended Owner Activities Set budget for project, and budget for commissioning of the project. Develop an RFP for a third-party commissioning agent. The proposal should list in detail the agents experience with similar mission critical facilities. Consider allowing qualified design team firms to bid provided an acceptable level of objectivity can be assured. Interview commissioning agents with the owners internal commissioning team so that the strengths and weaknesses of the commissioning agent can be examined from several perspectives. Define the purpose and mission that the facility will perform. Define the critical operations and processes. Describe the key performance indicators that characterize the facility (e.g., power capacity and density, environmental conditions, and tolerances, etc.). Establish flexibility and growth requirements. Define the expected facility performance over the lifetime of the facility (timeline and phasing). Determine required level of facility availability. Establish contingency objectives (e.g., 72 hours of on-site, stored makeup water, 30 minutes of battery-run, wind and storm severity, etc.). Develop the owners program document soliciting both design team and commissioning agent input. Develop the basis-of-design document. Expand on standard templates to ensure that high availability is incorporated into all aspects of design. Brainstorm failure modes with all parties, drawing especially on O&M department experience. Establish system-level redundancies. Involve O&M department in detailed specification of equipment, facility layout, and systems design. Develop the commissioning plan including roles and responsibilities. Review scope and methodology of tests to ensure that they provide the maximum assurance possible to the owner. Have O&M staff (and when justified, A&E staff) visit factory for critical equipment to meet staff and learn as much about the equipment as possible, especially for customized or unfamiliar equipment. For critical items, make sure that a thorough and comprehensive verification protocol is in place. Coordinate delivery schedule, phasing, transportation, and receiving expectations. Ensure ownership and liability for securing, inventorying, and tracking of received products and materials are established. Ensure adequate storage is available. Provide owner input into the design of this testing stage. Make sure that O&M staff are on board and at the site to review this process. Define frequency of inspections and list required participants at inspections. Owner should make sure that all contractual roles and responsibilities for these activities have been spelled out and agreed to in writing. The owner should consult with the primary representatives of the design team, construction contractors, and commissioning agent to ensure clarity and comprehension by all parties. O&M staff should be at full complement and an integral part of the acceptance testing. Ensure tests are comprehensive and consider all modes of operations. Ensure tests consider all failure mechanisms and tests all safeties. Ensure the BAS system is tested and made fully operational. Have O&M staff involved in building automation system (BAS) commissioning, and BAS involved in the remainder of Level 5 commissioning. Make sure that all known failure modes are tested to ensure that system responses are appropriate and result in maximum system availability. Subject the facility to real or simulated design-base anomalies and verify acceptable performance.

Commissioning Team Selection Project Programming Schematic Design

Final Design

Factory Acceptance Tests

Field Component Verification

System Construction Verification Site Acceptance Testing

Integrated Systems Testing

ASHRAE Transactions: Symposia

625

Terry L. Rodgers: The owners best protection (though not a fail-safe) from change order costs associated with necessary retesting following a failed level-5 commissioning test would be to have clear contractual obligations regarding commissioning efforts that include contingency clauses that address the need for repeat testing to attain successful results. These clauses should stipulate that all parties must include in their scope that successful level-5 commissioning is a shared responsibility by all involved in a specific integrated test, and could stipulate a fixed level of effort beyond which additional costs would apply at an agreed upon rate. The owner is also well advised to include commissioning contingencies in the project budget that anticipate that additional testing as well as unanticipated rework may be encountered before the project is completed. Reinhard Seidl, Mechanical Engineer, Taylor Engineering, Alameda, Calif.: (1) What experiences exist within the

speakers or TC 9.9 with formalized methods of addressing failure mode analysis? Risk-hazard analysis? (2) How is the issue of three-dimensional as-builts addressed to calibrate CFD models? Rodgers: Owners typically do not participate directly in formalized failure mode analysis from a component or equipment standpoint (mean time between failure, component design lifespan, etc.). This would fall in the realm of manufacturers and assurance gained by objective third-party certifications (ANSI, ASME, ASHRAE, UL, etc.). Risk-hazard analysis is typically accomplished by hiring consulting expertise with broad experience and/or databases and software applications that can model or simulate the proposed system and predict reliability or availability values. CFD modeling and associated calibration techniques are the realm of the CFD vendors and would probably include computational algorithms and calculations specific to their product.

626

ASHRAE Transactions: Symposia