Professional Documents
Culture Documents
Green Plum 300-012-943
Green Plum 300-012-943
Recovery
Configuration Guide
P/N 300-012-943 REV A01
Copyright 2011 EMC Corporation. All rights reserved. Published August, 2011 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners.
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Contents
Chapter 2
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Configuring the local DCA for the Symmetrix ..................... Configuring the remote DCA .................................................. Operating a SAN Mirror configuration.................................. Summary............................................................................................
38 42 42 47
Appendix A
Appendix B
Appendix C
Appendix D
Appendix E
Appendix F
Appendix G
Appendix H
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Appendix I
Appendix J
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Preface
This guide documents how to configure an EMC Greenplum Data Computing Appliance (DCA) to attach to an EMC Symmetrix VMAX system, where the primary data copy resides on the DCA and the VMAX is used as a mirror. The facilities of the VMAX storage array can then be used to provide remote replicas for disaster recovery and create point-in-time copies to allow quick recovery from equipment or data center outages. This document specifies the steps to move the data manually. The appendices provide detailed, step-by-step instructions and output from commands, taken from EMC's engineering lab tests. Forthcoming tools from Greenplum will help customers automate the movement of data to external storage. As part of an effort to improve and enhance the performance and capabilities of its product line, EMC from time to time releases revisions of its hardware and software. Therefore, some functions described in this document may not be supported by all revisions of the software or hardware currently in use. For the most up-to-date information on product features, refer to your product release notes. If a product does not function properly or does not function as described in this document, contact your EMC representative. Audience This guide is intended for EMC Field personnel who are implementing or considering a SAN Mirror solution for the Greenplum DCA. It will also be useful for customers and partners to gain understanding of the benefits and requirements of SAN Mirror implementations.
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Preface
Related documentation
More information is available about each of the components discussed in this document. A partial list is shown next. Documents are available at http://powerlink.emc.com. Related documents include:
White Paper: EMC Greenplum Data Computing Appliance Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery White Paper: EMC Greenplum Data Computing Appliance: High Performance for Data Warehousing and Business intelligence - An Architectural Overview White Paper: EMC Greenplum Database 4.0 - Critical Mass Innovation Greenplum Database 4.1 Administrator Guide (describes the database operation in great detail) Greenplum Database 4.1 Installation Guide Greenplum DCA and DIA 1.0.3.x Getting Started Guide Symmetrix VMAX Series Product Guide Symmetrix Remote Data Facility (SRDF) Product Guide Symmetrix TimeFinder Product Guide Solutions Enabler 7.3 Documentation Set PowerPath Family Product Guide PowerPath for Linux Installation and Administration Guide 5.5 A03
EMC uses the following conventions for notes, cautions, and warnings.
Note: A note presents information that is important, but not hazard-related.
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Preface
Typographical conventions EMC uses the following type style conventions in this document:
Normal Used in running (nonprocedural) text for: Names of interface elements (such as names of windows, dialog boxes, buttons, fields, and menus) Names of resources, attributes, pools, Boolean expressions, buttons, DQL statements, keywords, clauses, environment variables, filenames, functions, utilities URLs, pathnames, filenames, directory names, computer names, links, groups, service keys, file systems, notifications Used in running (nonprocedural) text for: Names of commands, daemons, options, programs, processes, services, applications, utilities, kernels, notifications, system call, man pages Used in procedures for: Names of interface elements (such as names of windows, dialog boxes, buttons, fields, and menus) What user specifically selects, clicks, presses, or types
Bold
Italic
Used in all text (including procedures) for: Full titles of publications referenced in text Emphasis (for example a new term) Variables Used for: System output, such as an error message or script URLs, complete paths, filenames, prompts, and syntax when shown outside of running text Used for: Specific user input (such as commands) Used in procedures for: Variables on command line User input variables Angle brackets enclose parameter or variable values supplied by the user Square brackets enclose optional values Vertical bar indicates alternate selections - the bar means or Braces indicate content that you must specify (that is, x or y or z) Ellipses indicate nonessential information omitted from the example
Courier
Courier bold
Courier italic
<> [] | {} ...
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Preface
EMC support, product, and licensing information can be obtained as follows. Product information For documentation, release notes, software updates, or for information about EMC products, licensing, and service, go to the EMC Powerlink website (registration required) at:
http://Powerlink.EMC.com
Technical support For technical support, go to EMC Customer Service on Powerlink. To open a service request through Powerlink, you must have a valid support agreement. Please contact your EMC sales representative for details about obtaining a valid support agreement or to answer any questions about your account. Your comments Your suggestions will help us continue to improve the accuracy, organization, and overall quality of the user publications. Please send your opinion of this guide to:
techpub_comments@EMC.com
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
1
Introducing the SAN Mirror Solution
Introduction ........................................................................................ Using a SAN Mirror solution ........................................................... Components of a SAN Mirror configuration ................................. Considerations and best practices ...................................................
10 13 18 25
Introduction
This section provides a brief introduction to the following:
Business need on page 10 EMC Greenplum Data Computing Appliance (DCA) on page 11 Terminology on page 12
Business need
Data warehouses are a critical tool for making business decisions. As data warehouse and business intelligence systems (DW/BI) continue to grow, they have faced some difficult challenges:
DW/BI infrastructures are growing exponentially over time and often require many terabytes or even petabytes of high-performance, protected storage. DW/BI systems must be continually available. High performance update, backup and disaster recovery are necessary to prevent and recover from outages.
Currently, data warehouse environments are demanding a more comprehensive strategy for data protection, security, and high availability than ever before, based on the business impact of data unavailability. Data recovery options must align with application and business requirements to yield the highest availability. To integrate the EMC Greenplum Data Computing Appliance (DCA) into larger data centers, some customers require compatibility with advanced storage and software infrastructures, such as the EMC Symmetrix VMAX in order to achieve the highest levels of fault tolerance and to provide industry-leading disaster recovery. To facilitate this, EMC engineered a solution that integrates the DCA with the VMAX, where the VMAX provides local and remote replicas of the data for disaster recovery and point-in-time snapshots. This allows customers to recover data warehouse/business intelligence (DW/BI) functionality quickly in the face of hardware or software failure, or even total site loss.
10
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Introduction
11
Terminology
The following list key terms used in this document. Term Definition
Business Intelligence The effective use of information assets to (BI) improve the profitability, productivity, or efficiency of a business. IT professionals use this term to refer to the business applications and tools that enable such information usage. The source of information is frequently the data warehouse. CNA Data Warehouse (DW) Converged Network Adapter The process of organizing and managing information assets of an enterprise. IT professionals often refer to the physical stored data content in some databases managed by database management software as the data warehouse. They refer to the applications that manipulate the data stored in such databases as DW applications. The DCA is a purpose-built, highly scalable, parallel DW appliance that architecturally integrates database, compute, storage, and network into an enterprise-class, easy-to-implement system. A type of distributed computing architecture where tens to hundreds of processors team up to work concurrently to solve large computational problems
12
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Overview on page 13 How SAN Mirror increases the capacity of the DCA on page 15 How EMC Greenplum DCA mirrors data on page 15 Mirroring data to the SAN with Greenplum SAN Mirror on page 16
Overview
The EMC Greenplum DCA maintains two copies of customer data and normally handles all data replication and protection tasks internally to the appliance. This generally achieves the highest level of performance for DW/BI tasks. In a SAN Mirror solution, the second copy of the data is moved to SAN-based storage. The DCA retains the primary copy of the data in order to maximize query performance. The SAN Mirror copy is updated with writes, but it is not read unless a primary database segment becomes inaccessible. By keeping the mirrored copy on the SAN, customers can use storage facilities such as EMC TimeFinder and SRDF to create remote copies or point-in-time images for backup and disaster recovery. Figure 1 and Figure 2 on page 14 compare a stand-alone DCA configuration to different SAN Mirror configurations.
13
Figure 1
Figure 2
14
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
15
In the EMC Greenplum DCA, Six primary segment instances and six mirror segment instances run on each segment server. The segments use internal DCA storage. The primary and mirror segments run on separate servers in the DCA. A high-level picture of the mirrored configuration is shown in Figure 3.
Figure 3
DCA mirroring
16
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Figure 4
17
EMC Greenplum DCA on page 18 EMC PowerPath is loaded on all segment servers. PowerPath is additional, separately chargeable software. Trunking is enabled on internal EMC Connectrix MP-8000B switches to provide a large, load-balanced pipe between the DCA and the SAN. Trunking is an additional, separately chargeable software function.
EMC Symmetrix VMAX storage system on page 21 EMC VMAX system, with four storage engines VP EMC Virtual Provisioning TimeFinder/SNAP for point-in-time copies SRDF for remote replication Solutions Enabler Symmetrix Management Console (SMC)
18
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
The EMC Greenplum DCA runs the Greenplum Database relational database management system (RDBMS) software. The EMC Greenplum Database enables large data queries and analysis. The DCA is offered in quarter-rack, half-rack, full-rack, and multiple-rack appliance configurations to achieve maximum flexibility and scalability for organizations faced with terabyte to petabyte scale data analyses. A full rack DCA configuration is shown in Figure 5.
Figure 5
PowerPath
PowerPath is host-based multipath software that provides high availability and load balancing across multiple Host Bus Adapter/Converged Network Adapter (HBA/CNA) ports. In a PowerPath configuration, each LUN is assigned to multiple HBA ports on a single server. PowerPath uses sophisticated algorithms to deliver I/O optimally down each path, to ensure the highest overall system performance. If a path fails, PowerPath continues to load balance down the remaining paths to each device, enabling the server to continue operation. PowerPath automatically re-enables failed paths when the failures are corrected.
19
PowerPath load balancing is an additional, separately-chargeable software product. Additional capabilities for data migration and data encryption can also be purchased for PowerPath. While Linux Multi-Path I/O (MPIO) also delivers primitive load balancing and path failure protection, it is more difficult to manage than PowerPath. MPIO was not explored as an option for the SAN Mirror solution. Trunking software The EMC Connectrix trunking feature optimizes the use of bandwidth by allowing a group of inter-switch links (ISLs) to merge into a single logical link. Trunking is automatically implemented for any eligible ISLs after you install the Connectrix ISL Trunking license. The license must be installed on each switch that participates in trunking. The trunking license should be installed on both Connectrix MP-8000B switches within the DCA, and also on the SAN switches to which the DCA is connected. In a fabric with numerous switches, you can increase the bandwidth between switches by enabling multiple physical ports to appear as a single port. Enabling multiple physical ports form a trunking group where the traffic is distributed dynamically and in order at the frame level, thus achieving greater performance with fewer inter-switch links. Trunking groups are based on the user port number with contiguous eight ports as one group, such as 0-7, 8-15, and 16-23. Figure 6 shows how trunking can result in more throughput by distributing data over four ISLs with no congestion. In a fabric that does not have trunking capability, some paths would be congested and other paths under-utilized.
Figure 6
20
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
21
TimeFinder local replication solutions include TimeFinder/Clone and TimeFinder/Snap. TimeFinder/Clone creates full-device and extent-level point-in-time copies while TimeFinder/Snap creates pointer-based logical copies that consume less storage space on physical drives. TimeFinder/Snap was used for the SAN Mirror solution described in this document. TimeFinder/Snap gives maximum flexibility with the lowest capacity requirement for point-in-time copies. In an SRDF/A configuration, it is essential to enable Write Pacing on the VMAX before taking Snaps on the R2 VMAX. Write Pacing became available with EMC Enginuity v5875 release; it protects SRDF/A sessions by slowing down host writes if the host write bandwidth is greater than the R2 VMAX can support. Since this could impact the performance of ongoing data warehouse loads, it is strongly advised to engage a Symmetrix Performance Guru when planning the SAN Mirror configuration.
Note: Symmetrix Performance Gurus can be located on the SPEED website, accessible to EMC customers, partners, and field personnel, at http://speed.corp.emc.com , Tools > Guru List. If you do not have access to this list, contact your local EMC Sales Representative.
SRDF
The Symmetrix Remote Data Facility (SRDF) is a business continuance solution that maintains a mirror image of data at the device level in Symmetrix arrays located in physically separate sites. There are three modes of SRDF:
Asynchronous mode (SRDF/A) was used in this solution, as it provides a high level of consistency with minimal impact to system performance. The local and remote systems were immediately adjacent to each-other, so the effective distance was 0 km. For longer distances, it is critical to correctly size the links between data centers to support the expected write workload. SRDF/A creates a dependent-write consistent, point-in-time image on the target (R2) device, which is a short period of time behind the source (R1) device. Managed in sessions, SRDF/A transfers data in
22
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
cycles or delta sets to ensure that data at the R2 site is dependent-write consistent. The Symmetrix system acknowledges all writes to the R1 devices in exactly the same way as other non-R1 local devices. Host writes accumulate on the R1 side until the cycle time is reached, and are then transferred to the R2 Symmetrix in one delta set. The writes are then destaged to the R2 devices to create a permanent, consistent image of the R1 data. Because writes are transferred in cycles, writes can be coalesced, and duplicate writes to the same tracks require only a single transfer across the links. This ensures optimal link bandwidth utilization. EMC Enginuity Consistency Assist (ECA) ensures that snaps and remote replicas are write-order consistent across multiple devices and even across multiple Symmetrix VMAX systems. This is a required capability for SAN Mirror, to ensure recoverability of database on the remote Symmetrix system. Solutions Enabler The EMC Solutions Enabler kit contains all the base management software that provides a host with SYMCLI commands and APIs to configure the Symmetrix and to control Symmetrix operations for SRDF and TimeFinder. SYMCLI resides on the host system to monitor and control operations on Symmetrix storage arrays. SYMCLI commands are invoked from the host operating system via command line or scripts. SYMCLI commands invoke low-level channel communications to specialized gatekeeper devices on the Symmetrix. SYMCLI is required to control TimeFinder and SRDF operations from the DCA. Because of this, EMC Solutions Enabler should be loaded on the database master servers in the DCA. Symmetrix Management Console (SMC) The EMC Symmetrix Management Console (SMC) is an intuitive, browser-based user interface that configures and manages Symmetrix system arrays. SMC presents the functionality of the Solutions Enabler SYMCLI (command line interface) in a browser interface. Use Symmetrix Management Console to:
Manage Symmetrix Access Controls, user accounts, and permission roles Discover Symmetrix arrays
23
Perform configuration operations (create devices, map and mask devices, set Symmetrix system attributes, set device attributes, set port flags, create SAVE device pools) Manage devices (change device configuration, set device status, reserve devices, duplicate devices, create/dissolve meta devices) Perform and monitor replication operations (TimeFinder/Snap, TimeFinder/Clone, SRDF, Open Replicator) Configure and manage Fully Automated Storage Tiering (FAST) Monitor alerts Monitor an application's performance
Symmetrix Management Console is preinstalled on the Symmetrix VMAX system Service Processor and can also run on a data center host. Licenses are available individually, or as part of a Symmetrix VMAX Management Integration bundle.
24
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Performance
Connecting the DCA to SAN-based storage delivers many manageability and functionality benefits, and it is important to configure the storage properly to ensure system performance. It is essential to configure enough engines, ports and disks to support the expected read and write workloads. A full DCA can ingest up to 13.4 TB/hr of new data (almost 4 GB/s), and can scan at up to 23.5 GB/s. A system configured for this maximum load may require one or more dedicated VMAX systems. However, very few customers will push a DCA to its bandwidth limits. In many cases, an existing Symmetrix might be used to support the needs of the data warehouse. As long as the Symmetrix configuration has enough spare bandwidth, the data warehouse workload can co-exist with other production data. The optimal Symmetrix configuration will be a function of the customer's actual expected workload. In SRDF configurations, it is critical to size the connection between VMAX systems properly. The number and speed of links will determine how much write data the system can sustain. Too few connections may significantly impact write performance. Please consult a SymmetrixPerformance Guru when planning a SAN Mirror configuration, to ensure that the VMAX will support the required bandwidth.
Note: Symmetrix Performance Gurus can be located on the SPEED website, accessible to EMC customers, partners, and field personnel, at http://speed.corp.emc.com , Tools > Guru List.. If you do not have access to this list, contact your local EMC Sales Representative.
The DCA contains EMC Connectrix MP-8000B 8000 switches to support internal communication between servers. In a SAN Mirror configuration, the Fibre ports on the switches are configured to attach
Considerations and best practices
25
to a Fibre Channel SAN. Customers should purchase optional SAN Trunking software for the switches to provide the greatest possible throughput. This requires an additional software license for the Connectrix switches. PowerPath should be loaded on each internal server to provide the greatest possible bandwidth and resiliency for the SAN Mirror configuration. PowerPath is a chargeable feature. Standard Linux MPIO can also be used, but may be much more difficult to manage as PowerPath is aware of the Symmetrix devices in the VMAX. Configuring and managing MPIO for the DCA is outside of the scope of this document.
Data integrity
A SAN Mirror configuration moves the "mirror" database segment storage to the SAN. In normal operation, the mirror segments are only used for writes to ensure resiliency in case a primary segment becomes unavailable. Query activity uses the primary segments on internal DCA disk, since that provides the highest performance. For a particular segment of data, Greenplum database mirroring relies on the primary instance process to send writes over the IP interconnect to a mirror instance process on a different segment server host. If a segment server host becomes unavailable, primary instances on that host will be taken out of operation and their corresponding mirrors activated on other segment hosts. Additionally, mirror instances on that host will no longer be able to receive replication updates and write them to SAN storage.
26
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Since the SAN copy of the database is taken from the mirror, the SAN copy of the database will no longer be consistent. This is illustrated in Figure 7.
Figure 7
To ensure there is always a restartable copy of the database if a failure occurs, TimeFinder/Snap is used to create "gold copy" images periodically on the remote VMAX. The gold copy will be slightly behind the SRDF image, but could be brought to currency by re-applying updates to the remote database. It is necessary to verify that there are no segment failures and that the system is operating normally when taking a snap, to ensure the gold copy is consistent. If a mirror segment is inoperable for any reason, no snap copy should be taken. This will ensure a recent, consistent gold copy on the remote VMAX. Procedures for making and verifying a gold copy are documented in Appendix G, Creating SNAPs on Remote Symmetrix for Disaster Recovery, and in Appendix H, Mounting and Checking a Symmetrix SNAP of the Greenplum Database.
27
28
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
2
Setup and Operation of SAN with VMAX
This chapter provides the following information on the setup and operation of SAN Mirror with VMAX. This chapter details the test system that was used and gives configuration details for both the EMC/Greenplum Data Computing Appliance (DCA) and the EMC Symmetrix VMAX system.
Introduction to the SAN Mirror test configuration....................... Detailed list of tested components .................................................. Prerequisites........................................................................................ Configuring for SAN Mirror ............................................................ Summary .............................................................................................
30 31 34 35 47
29
Figure 8
30
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Greenplum DCA components Component Master server Segment servers Interconnection Switch Administration Switch Quantity 1 (primary, no standby configured) 8 (GP100, half-rack configuration) 2 (EMC Connectrix MP-8000B) 1 (Allied Telesis)
Table 2
Greenplum DCA software Software EMC Greenplum-db PowerPath Solutions Enabler Version 4.1.1.1 5.5 7.2.1.0 Comments EMC Greenplum database software Strongly advised for multipathing and load balancing Enables the DCA to issue TimeFinder and SRDF commands to the VMAX
31
VMAX
Table 3, Table 4, and Table 5 list the VMAX components and software.
Table 3
Primary VMAX hardware components Hardware 4-Engine VMAX Fibre Channel directors RF (Remote Fibre) directors 15k RPM Fibre Channel Disk 8 Gb/s Fibre 8 Gb/s connection for SRDF replication 600 GB capacity, 15k RPM, configured RAID 5 7+1 and used with Virtual Provisioning Specifications Quantity 1 24 8 192
Table 4
Remote VMAX hardware component Hardware 4-Engine VMAX Fibre Channel directors RF (Remote Fibre) directors 15k RPM Fibre Channel Disk 8 Gb/s Fibre 8 Gb/s connection for SRDF replication 600 GB capacity, 15k RPM, enough to accommodate snaps and remote replicas Configured RAID 5 7+1 and used with Virtual Provisioning Specifications Quantity 1 24 8 470
32
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Table 5
VMAX software Software Enginuity 5875.198 Virtual Provisioning Specifications Base Enginuity Operating Environment EMC Virtual Provisioning allocates only the storage that was actually used, and stripes wide across all disks Not used for this paper Create asynchronous remote replicas (this test used zero-distance SRDF/A) Not used for this paper Create pointer-based gold copy of database volumes Graphical managing and monitoring interface for Symmetrix VMAX systems Quantity 2 2
2 2 2 2 2
33
Prerequisites
It is assumed that the following steps have been taken prior to configuring the SAN Mirror solution, and thus this solution will not detail the installation, setup and configuration of the components listed here. Further information can be located in the documents listed inRelated documentation on page 6.
The EMC/Greenplum DCA should be installed and configured in the data center. Specific information on the initial configuration of the EMC/Greenplum hardware and software is described in the Greenplum Database 4.1 Installation Guide. The installation guide discusses the installation process for the DCA servers, switches and hosts. This guide also discusses the basic configuration process for the Greenplum database. EMC PowerPath should already be installed on the DCA segment servers and database masters. PowerPath is not included with the DCA and should be installed by EMC Professional Services on each server in the DCA. EMC Solutions Enabler should already be installed on the database masters. Solutions Enabler is not included with the DCA and should be installed by EMC Professional Services on the master and the standby master. VMAX systems should be installed and configured in the local data center, and also in the remote data center if SRDF is used to replicate data between data centers. SRDF should be installed, configured and replicating between the VMAX systems. The data center SAN should be installed and operational, with the VMAX systems attached to it.
34
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Configuring a new system on page 35 Preparing the VMAX for SAN Mirror attachment on page 36 Configuring the local DCA for the Symmetrix on page 38 Configuring the remote DCA on page 42 Operating a SAN Mirror configuration on page 42
35
d. Zone Fibre Channel over Ethernet (FCoE) to the Symmetrix. e. Ensure Symmetrix volumes are seen by the DCA servers. f. Format and mount Symmetrix volumes on mirror locations. 3. Move master databases to VMAX storage. 4. Create a test database utilizing the VMAX SAN Mirror configuration. 5. Test failover to remote DCA to run directly off R2 volumes. a. Split SRDF links. b. Mount R2 volumes on Remote DCA. c. Recover remote DCA internal volumes from R2 image.
Creating volumes for DCA master and mirror databases on page 36 Configuring SRDF copies on page 37 Snaps on page 38 Clones on page 38
Creating volumes for DCA master and mirror databases Each segment server supports two, 2.7 TB database LUNs. In normal operation, one of these LUNs is used for a mirror database instance, and the other is used for a primary instance. The master server supports a single, 2.1 TB LUN. To implement SAN Mirror, LUNs of the same size should be created on the VMAX and made visible to the servers. The mirror LUN may or may not be used as an additional primary in a SAN Mirror configuration. The LUNs can be verified by issuing omreport storage vdisk on the segment servers. By issuing the gpssh -f gpssh.txt omreport storage
36
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
vdisk command, it is possible to see the exact LUN configuration on each segment server. Omreport output from the DCA and a script to create Symmetrix volumes are shown in Appendix C, Creating VMAX Volumes for DCA Mirror Database. Configuring SRDF copies To have a consistent, remote replica of the database, all of the DCA mirror volumes are replicated to the remote VMAX. This includes all volumes from the master server and each segment server. To ensure consistency during SRDF operations, all of the SAN Mirror volumes must be part of the same SRDF group. Configurations that span SRDF groups or Symmetrix VMAX systems must use SRDF Consistency Groups, which is beyond the scope of this paper. See EMC SRDF/A and SRDF/A Multi-Session Consistency on Unix and Windows for more information about SRDF consistency groups. If the primary site is lost for any reason, the database can be brought up on a remote DCA attached to the remote VMAX. The procedure to do this is described in Appendix J, Starting EMC Greenplum Database from Mirror Segments, (such as, on the R2 VMAX). The test system used SRDF/A to replicate data from the primary to the remote VMAX System. When using SRDF, it is important to ensure there is enough bandwidth between the systems to handle the expected write workload. A Symmetrix Performance Guru should be consulted to verify the configuration has adequate bandwidth.
Note: Symmetrix Performance Gurus can be located on the SPEED website, accessible to EMC customers, partners, and field personnel, at http://speed.corp.emc.com , Tools > Guru List. If you do not have access to this list, contact your local EMC Sales Representative.
To use TimeFinder/Snap on the remote VMAX, the Symmetrix must be configured for SRDF Device Write Pacing. With write pacing enabled, the primary Symmetrix will slow down host writes if the primary DCA's write workload is greater than the remote VMAX's ability to destaged and switch cycles. This preserves SRDF/A's ability to keep propagate data to the remote VMAX. Sample SYMCLI commands and output from the SRDF configuration are shown in Appendix D, Creating the SRDF Configuration.
37
Snaps The test configuration used TimeFinder/Snap to create point-in-time copies of the data on the remote VMAX. Snaps provide a flexible mechanism to make instant, consistent copies of the database volumes. As hosts write data, the original data is preserved in a "SAVE" pool, and pointers for the Snap volumes are changed to point to this preserved data. Only changed data is tracked by TimeFinder/Snap. This results in the lowest capacity overhead when creating multiple Snaps of the data. Taking frequent Snaps allows the database to be quickly rolled back to a known good state if data corruption is discovered. The SAVE pool must be sized to accommodate the expected capacity and bandwidth of changed data. It is considered best practice to configure SAVE devices across as many physical disks as possible in order to optimize the system's performance. Sample commands and output are shown in Appendix G, Creating SNAPs on Remote Symmetrix for Disaster Recovery. Clones TimeFinder/Clone is a facility that creates full copies of Symmetrix volumes. Clones are very useful when making copies of data for mounting and querying on a separate host, as the clone volumes can be placed on separate physical drives. By separating the clones from the source data, users can work on the clones without impacting production. The tested configuration used TimeFinder/Snap instead of clones because multiple point-in-time copies of the data are required. Clones would require significantly more storage since each clone is a full copy of the source volume.
38
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Enabling FCoE on the internal EMC Connectrix MP-8000B Switches on page 39 Changing mirror locations on the DCA (if needed) on page 39 Moving master database for existing DCA on page 41 Moving segment database mirrors for existing DCA on page 41
Discovering the DCA server WWNs Each master server and segment server contains a Brocade 1020 CNA for Fibre Channel Over Ethernet (FCOE) connectivity to the SAN. The internal Connectrix MP-8000B switch strips the Ethernet headers when forwarding traffic to the SAN. Thus, issuing "bcu port -list | grep fcoe" on each DCA server will display the adapters' WWNs. The output of the command on the test harness is shown in Appendix A, Discovering DCA Internal Server WWNs. Enabling FCoE on the internal EMC Connectrix MP-8000B Switches FCoE must be enabled on the DCA to connect the it to external SAN storage. Each CNA in each server will then show up as a separate initiator to be zoned on the SAN. The DCA is configured with two, EMC Connectrix MP-8000B switches. Each switch contains eight Fibre ports for SAN connection. FCoE must be enabled individually on each of the MP-8000Bs internal Ethernet ports in order for the internal servers to appear on the SAN. The procedure to accomplish this manually is shown in Appendix E, Configuring DCA Internal Switch for SAN Mirror. The procedure can be automated using Expect, which is a standard Linux utility on the DCA. A sample script is shown in Appendix F, Automating DCA Internal SAN Configuration. Changing mirror locations on the DCA (if needed) Users specify where the DCA stores primary and mirrored data segments at installation time when gpinitsystem is run. The default is to create multiple data directories (/data1 and /data2), where each /data directory contains ./primary and ./mirror subdirectories. See The Greenplum Database Installation Guide, Chapter 3, for more information about default configurations. In a SAN Mirror configuration, internal DCA storage should be mounted on the ./primary directories. SAN storage should be mounted on the ./san_mirror directories.
Configuring for SAN Mirror
39
Storage for the master and standby master servers must also be moved to the VMAX to enable failover to the remote VMAX. Moving an operational master to the VMAX will require a short period of downtime as the database requires restart during these operations.
IMPORTANT For the purposes of this paper, the SAN Mirror storage was mounted directly on the ./mirror directories. An upcoming release of the Greenplum database for DCA will streamline and automate the SAN Mirror installation process. Default mount points for SAN Mirror disk will be ./san_mirror. A high-level diagram of the recommended configuration is shown in Figure 9.
Figure 9
40
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
If the initial database was created without mirrors. the gpaddmirrors utility can be run to dynamically add SAN Mirrors to the existing database. More information on gpaddmirrors can be found in the Greenplum Database 4.1 Administrator's Guide. Moving master database for existing DCA Downtime is required to move the master database to VMAX storage. The test configuration did not use a standby master, so downtime had to be taken to manually copy data from the /data/master directory to the VMAX, and to mount the VMAX volume on the /data/master directory. For existing DCAs with mirrored master databases, it is possible to remove and reinitialize the standby master server on VMAX storage. When this process is complete, the database can be failed over to the standby master while the primary master server is moved to VMAX. This greatly reduces the downtime required to implement a SAN Mirror configuration. The Greenplum command gpinitstandby can be used to change between the master server and the standby master. Moving segment database mirrors for existing DCA If SAN Mirror is being added to a pre-existing DCA configuration, where both the mirrors and the primaries reside on the DCA, the gpmovemirrors utility can be used to move the existing mirrors to external SAN storage. This is an online process for the segment servers. Gpmovemirrors ships with Greenplum database release 4.1.1.1. When using gpmovemirrors, create an input file that specifies the host address, port and system file space location of the current mirror, and the host address, port, replication port, and system file space location of the new mirror. The input file should be a plain text file with the format:
[<filespace1_fsname>[:<filespace2_fsname>:...] <old_address>:<port>:<system_filespace_location> [<new_address:port>:<replication_port>:<system_filespace_location>[:<fselocat ion>:...]]
For example:
sdw1-1:50000:/data1/mirror/gpseg11 sdw1-1:50000:51000:/data1/san_mirror/gpseg11
The host address, ports and system filespace locations can be found in the gp_segment_configuratoin and pg_filespace tables. For more information, refer to the Greenplum Database 4.1 Administrator's Guide.
Configuring for SAN Mirror
41
IMPORTANT It is critical that all of the mirrors reside on the VMAX to ensure a recoverable database on the remote DCA.
42
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Two tables within the GP database contain the configuration and location of the GP segment databases. These tables are critical to SAN Mirror operations that bring the database up on a different DCA. They are gp_segment_configuration and pg_filespace_entry. The gp_segment_configuration table maintains information about what database is located on which segment server, as well as what role each segment database has (that is, primary or mirror). In order to configure the GP database to use only the mirror segments, the gp_segment_configuration table space needs to be modified. The pg_filespace_entry table maintains the filesystem and directory location of each of the segment databases. It is important for manageability to have consistent, easy-to-understand directory structures for the primary and mirror instances. The procedure to bring up the database on the remote VMAX and to copy data from the remote VMAX to the remote DCA is detailed in Appendix J, Starting EMC Greenplum Database from Mirror Segments, (such as, on the R2 VMAX). The following are further described in this section:
Using TimeFinder/SNAP to create point-in-time images on the remote VMAX on page 43 Recovering a corrupt database using TimeFinder/SNAP and SRDF on page 44 Failover on page 45 Failback on page 46
Using TimeFinder/SNAP to create point-in-time images on the remote VMAX In normal operation a DCA will serve user requests from the primary copies of the data and will write updates to the SAN Mirror copies to ensure consistency and resiliency of the database. In a SAN Mirror configuration, additional TimeFinder/Snap images are created on the remote DCA to provide quick recovery in the event of data corruption. The procedure outlined in Appendix G, Creating SNAPs on Remote Symmetrix for Disaster Recovery and Appendix H, Mounting and Checking a Symmetrix SNAP of the Greenplum Database can be scripted and run at regular intervals to ensure easy recoverability of the database. The frequency with which copies are made is a function of the customer's business requirements.
43
Typical steps in normal operation are shown below: 1. Data is continually replicated to the remote VMAX using SRDF/A. 2. TimeFinder/Snap is used to take periodic copies of the data on the remote VMAX, as long as all of the DCAs primary and mirrored segments are operational. See Appendix G, Creating SNAPs on Remote Symmetrix for Disaster Recovery. 3. Each Snap image is verified on the remote DCA. The steps listed below are detailed in Appendix H, Mounting and Checking a Symmetrix SNAP of the Greenplum Database. a. Verify the current master server; use that master image to obtain all current configuration data. b. Mount the Snap volumes to the remote DCA. c. Verify that all segment servers were operational and reporting correct status. d. Run gpcheckcat to verify that the database is correct. 4. Older Snaps may be deleted, if desired, once the new Snap is verified. Recovering a corrupt database using TimeFinder/SNAP and SRDF If the database becomes corrupted for any reason, it can be restored to a previous snapped image. Since the snaps are taken on the remote VMAX, the procedure to restore the primary site is as shown below. Details are shown in Appendix I, Recovering a DCA Using a SNAP Session and SRDF. 1. Stop the primary database and unmount volumes on primary DCA. 2. Split the SRDF devices so the snap can be restored to the R2 volumes. 3. Restore the Snaps to the R2 volumes, then terminate the Snap restore session. 4. Restore the SRDF R2 devices to the R1 devices, to restore the R1 image to a known good point. 5. Remount the R1 devices on the primary DCA
44
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
6. Start the database in maintenance mode on the primary DCA and mark the primary segments as "down" so the database will come up on the mirrored segments. 7. Restart the database normally on the primary DCA. 8. Type gprecoverseg -F to recover the primary volumes from the mirrors. 9. Verify that all segments have synchronized by typing gprecoverseg -m. 10. Either restart the database or type gprecoverseg -r to move primary activity to internal primary volumes. Failover The database can be brought up on the remote DCA by running a series of SRDF and Greenplum commands. The step-by-step procedure to do this is detailed in Appendix J, Starting EMC Greenplum Database from Mirror Segments, (such as, on the R2 VMAX), and is summarized below: 1. Use SYMCLI commands to fail over the storage to the remote site. 2. Mount the R2 volumes to the remote DCA and start the database in maintenance mode. 3. Mark the remote DCA's primary segments as "down" and the mirrored segments as "up" so the database will come up on the mirrored (SAN) segments. 4. Restart the database normally. 5. Type gprecoverseg -F to recover the remote DCA's primary (internal) storage from mirrored (SAN) storage. 6. Verify that all segments have synchronized by typing gprecoverseg -m. 7. Either restart the database or type gprecoverseg -r to move primary activity to internal primary volumes. 8. When the primary site is restored, begin synchronizing from the remote VMAX to the primary VMAX.
45
Failback Failback from the remote to the primary site is simply the reverse of the failover process. A high-level description is listed below. 1. Ensure the primary DCA has unmounted the mirrored segments from the primary VMAX. 2. Synchronize the primary VMAX from the remote VMAX. This can be done while the database is running on the remote VMAX. 3. Use SYMCLI or the Symmetrix Management Console to issue SRDF commands to fail the database back to the primary site. 4. Mount the R1 volumes on the primary DCA and start the database in maintenance mode 5. Mark the primary DCA's primary segments as "down" and the mirrored segments as "up" so the database will come up on the mirrored (SAN) segments. 6. Restart the database normally. 7. Type gprecoverseg -F to recover the DCA's primary (internal) volumes from the mirrored (SAN) segments.
8. Verify that all segments have synchronized by typing gprecoverseg -m. 9. Either restart the database or type gprecoverseg -r to move primary activity to internal primary volumes.
46
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Summary
The EMC Greenplum database offers industry-leading capability for "big data" analysis, allowing customers to make precise business decisions more quickly. The EMC Greenplum Data Compute Appliance (DCA) is a self-contained system, designed to produce optimal performance using commodity components. It contains all the hardware and software necessary and is a complete data warehouse appliance. By using SAN Mirror to place the DCA mirrored databases on external storage, users leverage the capabilities of external storage to augment the performance and simplicity of the DCA. The EMC Symmetrix VMAX brings robust multipathing, replication and data management to the Greenplum database, to facilitate disaster recovery and to create gold copies of the database that guard against corruption of the data warehouse. The procedure to place SAN copies on external storage is documented here as a manual, step-by-step process. Following these procedures enables SAN Mirror to run on external VMAX arrays. Future releases of the Greenplum database will include tools to simplify SAN Mirror implementation.
Summary
47
48
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
A
Discovering DCA Internal Server WWNs
49
Figure 10
Sample output
50
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
B
Verifying DCA Volume Sizes
51
52
C
Creating VMAX Volumes for DCA Mirror Database
Creating VMAX volumes for DCA mirror database .................... 54 Assigning volumes to the DCA segment servers.......................... 56
53
54
New symdevs: 027F:028A [Striped meta, head 027F, member size 262668 New symdevs: 028B:0296 [Striped meta, head 028B, member size 262668 New symdevs: 0297:02A2 [Striped meta, head 0297, member_size 262668 New symdevs: 02A3:02AE [Striped meta, head 02A3, member size 262668 New symdevs: 02AF:02BA [Striped meta, head 02AF, member size 262668 New symdevs: 02BB:02C6 [Striped meta, head 02BB, member size 262668 New symdevs: 02C7:02D2 [Striped meta, head 02C7, member size 262668 New symdevs: 02D3:02DE [Striped meta, head 02D3, member size 262668 New symdevs: 02DF:02EA [Striped meta, head 02DF, member size 262668 New symdevs: 02EB:02F6 [Striped meta, head 02EB, member size 262668 New symdevs: 02F7:0302 [Striped meta, head 02F7, member size 262668 New symdevs: 0303:030E [Striped meta, head 0303, member size 262668 New symdevs: 030F:031A [Striped meta, head 030F, member size 262668 New symdevs: 031B:0326 [Striped meta, head 031B, member size 262668 New symdevs: 0327:0332 [Striped meta, head 0327, member size 262668 Terminating the configuration change session..............Done. The configuration change session has successfully completed. ?
cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl] cyl]
create -name MDW1 -type initiator -wwn 1000000533482f24 -type initiator add -name MDW1 -wwn 1000000533482f25 create -name SDW1 -type initiator -wwn 100000053348159a -type initiator add -name SDW1 -wwn 100000053348159b create -name SDW2 -type initiator -wwn 10000005334817e8 -type initiator add -name SDW2 -wwn 10000005334817e9 create -name SDW3 -type initiator -wwn 10000005334819b8 -type initiator add -name SDW3 -wwn 10000005334819b9 create -name SDW4 -type initiator -wwn 100000053348146c -type initiator add -name SDW4 -wwn 100000053348146D create -name SDW5 -type initiator -wwn 1000000533482554 -type initiator add -name SDW5 -wwn 1000000533482555 create -name SDW6 -type initiator -wwn 1000000533481f58 -type initiator add -name SDW6 -wwn 1000000533481f59 create -name SDW7 -type initiator -wwn 100000053348256c -type initiator add -name SDW7 -wwn 100000053348256D create -name SDW8 -type initiator -wwn 1000000533481f48 -type initiator add -name SDW8 -wwn 1000000533481f49
symaccess -sid 4467 create -name DCA -type port -dirport 6g:0,8g:0,10g:0,12g:0,5f:0,7f:0,9f:0,11f:0 symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess symaccess -sid -sid -sid -sid -sid -sid -sid -sid -sid 4467 4467 4467 4467 4467 4467 4467 4467 4467 create create create create create create create create create view view view view view view view view view -name -name -name -name -name -name -name -name -name MDW1 SDW1 SDW2 SDW3 SDW4 SDW5 SDW6 SDW7 SDW8 -sg -pg -pg -pg -pg -pg -pg -pg -pg MDW1 -ig MDW1 -pg DCA DCA -ig SDW1 -sg SDW1 DCA -ig SDW2 -sg SDW2 DCA -ig SDW3 -sg SDW3 DCA -ig SDW4 -sg SDW4 DCA -ig SDW5 -sg SDW5 DCA -ig SDW6 -sg SDW6 DCA -ig SDW7 -sg SDW7 DCA -ig SDW8 -sg SDW8
56
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
D
Creating the SRDF Configuration
57
symrdf -file rdf_pairs.txt -sid 04467 -rdfg 1 -noprompt -v createpair -type r1 -establish -rdf_mode sync -g DCA_rdfg
The file rdf_pairs.txt contained the following lines, listing the device pairs to create:
025B0405 02670411 0273041D 027F0429 028B0435 02970441 02A3044D 02AF0459 02BB0465 02C70471 02D3047D 02DF0489 02EB0495 02F704A1 030304AD 030F04B9 031B04C5 032704D1
58
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
E
Configuring DCA Internal Switch for SAN Mirror
59
This section details the procedure to configure the switches. The next section provides a script that could be used to automate the switch configuration. 1. Log in to switch 1 on the DCA. ssh 172.28.0.170 username = admin password = changeme (default password) 2. If connecting the switch to an existing fabric make sure to change the domain ID of the switch to one that is not already assigned in the fabric. The default ID for the switches is 1. 3. Enter into the configuration shell. cmsh 4. Enter into config mode. configure terminal (short hand = conf t) 5. Set the protocol to spanning-tree Rapid Spanning Tree (rstp) and the bridge priority to a high number so the switch will not become the root switch when plugged into the fabric. protocol spanning-tree rstp bridge-priority 8192
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
60
i-sw-1(config)# 6. Create a CEE-MAP for the FCoE protocol and set the priority-group table weights. priority-group-table 1 weight 40 pfc priority-group-table 1 weight 60 priority-table 2 2 2 1 2 2 2 15.0 exit Sample command screen output:
i-sw-1(config)#cee-map fcoe-vlan i-sw-1(conf-ceemap)#priority-group-table 1 weight 40 pfc i-sw-1(conf-ceemap)#priority-group-table 2 weight 60 i-sw-1(conf-ceemap)#priority-table 2 2 2 1 2 2 2 15.0 i-sw-1(conf-ceemap)#exit i-sw-1(config)#
7. Set up the lldp protocol. Protocol lldp advertise dcbx-fcoe-app-tlv advertise dcbx-fcoe-logical-link-tlv exit Sample command screen output:
i-sw-1(config)#protocol lldp i-sw-1(conf-lldp)#advertise dcbx-fcoe-app-tlv i-sw-1(conf-lldp)#advertise dcbx-fcoe-logical-link-tlv i-sw-1(conf-lldp)#exit i-sw-1(config)#
The vlan number can be any number other than 1 or 199. The default setting on the DCA is to route network traffic thru vlan 199 on switch 1, and 299 on switch 2. For this environment, FCOE traffic is routed thru vlan 1002 on switch 1 and 2002 on switch 2. interface vlan 1002 (on switch one; should be 2002 on switch 2) description FCoE VLAN fcf forward
Configuring DCA internal switch for SAN Mirror
61
exit Sample command output screen capture: i-sw-1(config)#interface vlan 1002 (on switch one; should be 2002 on switch 2) i-sw-1(conf-if-vl-1002)#description FCoE VLAN i-sw-1(conf-if-vl-1002)#fcf forward i-sw-1(conf-if-vl-1002)#exit i-sw-1(config)# 9. Setup vlan rules for FCoE traffic and add the rules to a group for assignment to the interfaces. Group 2 is assigned to the Ethernet vlan rules configured on the switch by default at the factory. vlan classifier rule 1 proto fip encap ethv2 vlan classifier rule 2 proto fcoe encap ethv2 vlan classifier group 1 add rule 1 vlan classifier group 1 add rule 2 Sample command output screen capture:
i-sw-1(config)#vlan i-sw-1(config)#vlan i-sw-1(config)#vlan i-sw-1(config)#vlan classifier classifier classifier classifier rule 1 proto fip encap ethv2 rule 2 proto fcoe encap ethv2 group 1 add rule 1 group 1 add rule 2
i-sw-1(config)# 10. Configure each of the connected network ports. For a full rack ports 0/0 0/17 are used. For a rack ports 0/0 0/7 & 0/16 & 0/17 are used. Ports 0/16 abd 0/17 are for the master and standby servers. Ports 0/0 0/15 are for the segment servers. Repeat the below commands for each interface incrementing the int te 0/# for each port. int te 0/0 switchport switchport mode converged vlan classifier activate group 2 vlan 199 vlan classifier activate group 1 vlan 1002 no shutdown lldp fcoe-priority-bits 0x8 spanning-tree edgeport spanning-tree edgeport bpdu-guard cee fcoe-vlan exit Sample command output screen capture:
i-sw-1(config)#int te 0/0 i-sw-1(conf-if-te-0/0)#switchport
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
62
i-sw-1(conf-if-te-0/0)#switchport mode converged i-sw-1(conf-if-te-0/0)#vlan classifier activate group 2 vlan 299 i-sw-1(conf-if-te-0/0)#vlan classifier activate group 1 vlan 2002 i-sw-1(conf-if-te-0/0)#no shutdown i-sw-1(conf-if-te-0/0)#lldp fcoe-priority-bits 0x8 i-sw-1(conf-if-te-0/0)#spanning-tree edgeport i-sw-1(conf-if-te-0/0)#spanning-tree edgeport bpdu-guard i-sw-1(conf-if-te-0/0)#cee fcoe-vlan i-sw-1(conf-if-te-0/0)#exit
12. Create a copy of the startup-config file so that you have the default configuration in the event that you have to revert back.
copy startup-config orig-startup-config
13. Copy the running-config to the startup-config so that you dont lose the settings on a reboot or power outage. Copy running-config startup-config. Sample command output screen capture:
i-sw-1# copy startup-config orig-startup-config Building configuration... i-sw-1# copy running-config startup-config Overwrite the startup config file (y/n): y Building configuration... i-sw-1# dir Contents of flash:// -rw-r----4589 Wed Apr 27 18:47:25 2011 i-sw-1#
orig-startup-config
14. Exit out of the cee shell back into the switch shell
Exit
15. To validate that the changes made were correct check the fcoe port logins to make sure you have the proper number of WWNs logging into the switch.
fcoe loginshow
16. You can also check each of the nodes and make sure that the fcoe port link status is up and running using gpssh with a host file containing the nodes in the DCA.
gpssh -f seg_srvr_list 'bcu port --list |grep fcoe'
64
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
17. Repeat the switch steps for switch 2 on the DCA. The IP is 172.28.0.180. Make sure to use a different vlan number for the FCoE vlan such as 2002 to keep them separate from switch 1. Sample command output for creating vlan on switch 2:
i-sw-2(config)#interface vlan 2002 i-sw-2(conf-if-vl-1002)#description FCoE VLAN i-sw-2(conf-if-vl-1002)#fcf forward i-sw-2(conf-if-vl-1002)#exit i-sw-2(config)#
Once the FCoE ports are logged into the switch you can alias and zone them to the SAN storage as a normal fiber port is zoned.
Configuring DCA internal switch for SAN Mirror
65
66
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
F
Automating DCA Internal SAN Configuration
67
# The "switch" variable is the last octet of the internal IP address of the # MP-8000B switch. The DCA has two switches: 172.28.0.170 and 172.28.0.180 # Both switches need to be set up with an FCOE VLAN, and ports need to route to them set switch "170" while { $switch < "181" } { puts "\r\rswitch $switch\r" if {$switch == "170"} { puts "THIS IS THE FIRST SWITCH\r" set enet_vlanid "199" set fcoe_vlanid "1002" } elseif {$switch == "180"} { puts "THIS IS THE SECOND SWITCH\r" set enet_vlanid "299" set fcoe_vlanid "2002" } puts "\r\rLogging into switch at 172.28.0.$switch\r" spawn ssh "admin\@172.28.0.$switch" expect { timeout abort failed abort "could not reach" abort "password:" } send "changeme\r" expect { timeout abort failed abort "Permission denied" abort ":admin>"
68
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
} sleep 1 send "cmsh\r" expect { timeout abort failed abort "Error:" abort "#" } sleep 1 puts "\r\rSetting switch protocol to Rapid Spanning Tree, and setting priority\r" send "conf t\r" expect { timeout abort failed abort "Error:" abort "(config)#" } send "protocol spanning-tree rstp\r" expect { timeout abort failed abort "Error:" abort "(conf-rstp)#" } send "bridge-priority 8192\r" expect { timeout abort failed abort "Error:" abort "(conf-rstp)#" } puts "\r\rCreating CEE Map for FCOE protocol\r" send "cee-map fcoe-vlan\r" expect { timeout abort failed abort "Error:" abort "(conf-ceemap)#" } send "priority-group-table 1 weight 40 pfc\r" expect { timeout abort failed abort "Error:" abort "(conf-ceemap)#" } send "priority-group-table 2 weight 60\r" expect { timeout abort failed abort
Script to automate DCA internal SAN configuration
69
"Error:" abort "(conf-ceemap)#" } send "priority-table 2 2 2 1 2 2 2 15.0\r" expect { timeout abort failed abort "Error:" abort "(conf-ceemap)#" } send "exit\r" expect { timeout abort failed abort "Error:" abort "(config)#" } puts "\r\rSetting up lldp protocol\r" send "protocol lldp\r" expect { timeout abort failed abort "Error:" abort "(conf-lldp)#" } send "advertise dcbx-fcoe-app-tlv\r" expect { timeout abort failed abort "Error:" abort "(conf-lldp)#" } send "advertise dcbx-fcoe-logical-link-tlv\r" expect { timeout abort failed abort "Error:" abort "(conf-lldp)#" } send "exit\r" expect { timeout abort failed abort "Error:" abort "(config)#" } puts "\r\rSetting up vlan for FCOE traffic\r" send "interface vlan $fcoe_vlanid\r" expect { timeout abort failed abort "Error:" abort
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
70
"(conf-if-vl-$fcoe_vlanid)#" } send "description FCOE VLAN\r" expect { timeout abort failed abort "Error:" abort "(conf-if-vl-$fcoe_vlanid)#" } send "fcf forward\r" expect { timeout abort failed abort "Error:" abort "(conf-if-vl-$fcoe_vlanid)#" } send "exit\r" expect { timeout abort failed abort "Error:" abort "(config)#" } puts "\r\rSetting up vlan rules for FCOE traffic\r" send "vlan classifier rule 1 proto fip encap ethv2\r" expect { timeout abort failed abort "Error:" abort "(config)#" } send "vlan classifier rule 2 proto fcoe encap ethv2\r" expect { timeout abort failed abort "Error:" abort "(config)#" } send "vlan classifier group 1 add rule 1\r" expect { timeout abort failed abort "Error:" abort "(config)#" } send "vlan classifier group 1 add rule 2\r" expect { timeout abort failed abort "Error:" abort "(config)#"
Script to automate DCA internal SAN configuration
71
} # # # # Loop to set each of the configured network ports for FCOE For full rack, ports 0-17 are used For half rack, ports 0-7 and 16-17 are used This script sets ALL ports to FCOE puts "\r\rSetting up individual ports for FCOE...\r" for {set i 0} {$i < 18} {incr i 1} { puts "Port $i...\r" send "int te 0/$i\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "switchport\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "switchport mode converged\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } # Note ENET vlan id is set to 199 on switch .170, 299 on switch .180 send "vlan classifier activate group 2 vlan $enet_vlanid\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } # Note FCOE vlan id is set to 1002 on switch .170, 2002 on switch .180 send "vlan classifier activate group 1 vlan $fcoe_vlanid\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "no shutdown\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#"
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
72
} send "lldp fcoe-priority-bits 0x8\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "spanning-tree edgeport\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "spanning-tree edgeport bpdu-guard\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "cee fcoe-vlan\r" expect { timeout abort failed abort "Error:" abort "(conf-if-te-0/$i)#" } send "exit\r" expect { timeout abort failed abort "Error:" abort "(config)#" } } # End FOR PORT loop send "exit\r" expect { timeout abort failed abort "Error:" abort "#" } puts "\r\rSaving original config as orig-startup-config-before-fcoe\r" send "copy startup-config orig-startup-config-before-fcoe\r" expect { timeout abort failed abort "Error:" abort
Script to automate DCA internal SAN configuration
73
"Overwrite it" {send "y\r"; exp_continue} "Building configuration" } sleep 2 puts "\r\rCopying running config to startup-config\r" send "Copy running-config startup-config\r" expect { timeout abort failed abort "Error:" abort "Overwrite the startup config file (y/n):" } send "y\r" expect { timeout abort failed abort "Error:" abort "Building configuration" } sleep 2 send "exit\r" expect { timeout abort failed abort "Error:" abort ":admin" } puts "\r\rIMPORTANT NOTE: PLEASE LOG BACK IN TO SWITCHES AND ISSUE THE FOLLOWING TO\r" puts " VERIFY THAT PORTS HAVE LOGGED IN: fcoe --loginshow\r\r" send "exit\r" set switch [expr $switch + 10]; # END WHILE SWITCH loop } ?
74
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
G
Creating SNAPs on Remote Symmetrix for Disaster Recovery
Creating SNAPs on remote Symmetrix for disaster recovery ..... 76 Creating a single SNAP of R2 standard devices............................ 77 Creating multiple SNAPS of R2 standard devices ........................ 91
75
76
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
2. To list available save device pools (snap pools) and verify the state of the poo, make sure the pool that will be used is Enabled.
symsnap -sid <symm serial #> -pools list
3. Verify that the Symmetrix where the snap will be conducted has the proper amount of virtual devices configured and that they have the same layout as the R2 standard volumes.
symdev list -sid <symm serial #> -vdev -meta
78
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
. Data cut due to output length 071F Not Visible VDEV 072B Not Visible VDEV 0737 Not Visible VDEV 0743 Not Visible VDEV 074F Not Visible VDEV 075B Not Visible VDEV
12 12 12 12 12 12
4. Create a device group to maintain the R2 standard devices and the virtual devices that will be used for the snap session.
symdg create <device group name> -type RDF2
5. Add all of the standard R2 devices to the newly created device group.
symdg -g <device group name> addall -sid <symm serial #> -devs <symdev>:<symdev>
6. Verify the devices were added to the newly created device group.
symdg show <device group name>
Group Type Device Group in GNS Valid Symmetrix ID Group Creation Time Vendor ID Application ID Number Number Number Number Number Number Number Number of of of of of of of of
STD Devices in Group : Associated GK's : Locally-associated BCV's : Locally-associated VDEV's : Locally-associated TGT's : Remotely-associated VDEV's(STD RDF): Remotely-associated BCV's (STD RDF): Remotely-associated TGT's(TGT RDF) :
Remotely-associated BCV's (BCV RDF): 0 Remotely-assoc'd RBCV's (RBCV RDF) : 0 Remotely-assoc'd BCV's (Hop-2 BCV) : 0 Remotely-assoc'd VDEV's(Hop-2 VDEV): 0 Remotely-assoc'd TGT's (Hop-2 TGT) : 0 Composite Groups : 0 Group Names : N/A
Standard (STD) Devices (17): { -------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------DEV001 /dev/emcpowerf 0411 RDF2+TDEV (M) RW 2955015 DEV002 N/A 041D RDF2+TDEV (M) RW 2955015 DEV003 N/A 0429 RDF2+TDEV (M) RW 2955015 DEV004 N/A 0435 RDF2+TDEV (M) RW 2955015 DEV005 N/A 0441 RDF2+TDEV (M) RW 2955015 DEV006 N/A 044D RDF2+TDEV (M) RW 2955015 DEV007 N/A 0459 RDF2+TDEV (M) RW 2955015 DEV008 N/A 0465 RDF2+TDEV (M) RW 2955015 DEV009 N/A 0471 RDF2+TDEV (M) RW 2955015 DEV010 N/A 047D RDF2+TDEV (M) RW 2955015 DEV011 N/A 0489 RDF2+TDEV (M) RW 2955015 DEV012 N/A 0495 RDF2+TDEV (M) RW 2955015 DEV013 N/A 04A1 RDF2+TDEV (M) RW 2955015 DEV014 N/A 04AD RDF2+TDEV (M) RW 2955015 DEV015 N/A 04B9 RDF2+TDEV (M) RW 2955015 DEV016 N/A 04C5 RDF2+TDEV (M) RW 2955015 DEV017 N/A 04D1 RDF2+TDEV (M) RW 2955015 } Device Group RDF Information { RDF Type RDF (RA) Group Number Remote Symmetrix ID
: R2 : 1 : 000192604467
(00)
R2 Device Is Larger Than The R1 Device : False Paired with a Diskless Device Paired with a Concurrent Device Paired with a Cascaded Device Thick Thin Relationship RDF Pair Configuration RDF STAR Mode : : : : False False False False
: Normal : False
80
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Mode Adaptive Copy Adaptive Copy Write Pending State Adaptive Copy Skew (Tracks)
: : : :
RDF Device Domino RDF Link Configuration RDF Link Domino Prevent Automatic RDF Link Recovery Prevent RAs Online Upon Power ON Device RDF Status
: Ready
Device RA Status : Ready Device Link Status : Not Ready Time of Last Device Link Status Change : N/A Device Suspend State Device Consistency State Device Consistency Exempt State RDF R2 Not Ready If Invalid Device Write Pacing Exempt State Effective Write Pacing Exempt State Device RDF State Remote Device RDF State RDF Pair State ( R1 <=\=> R2 ) : : : : : : Offline Enabled Disabled Disabled Disabled Disabled
(RW) (RW)
Number of R1 Invalid Tracks Number of R2 Invalid Tracks RDFA Information: { Session Number Cycle Number Number of Devices in the Session Session Status Consistency Exempt Devices Write Pacing Exempt Devices Session Consistency State Minimum Cycle Time Average Cycle Time Duration of Last cycle Session Priority
: : : : : : : : : : :
Tracks not Committed to the R2 Side: 0 Time that R2 is behind R1 : 291:39:24 R2 Image Capture Time : Wed Jun 15 07:43:38 2011 R2 Data is Consistent : True
Creating a single SNAP of R2 standard devices
81
R1 Side Percent Cache In Use R2 Side Percent Cache In Use Transmit Idle Time R1 Side DSE Used Tracks R2 Side DSE Used Tracks R1 Side Shared Tracks } } [root@smdw config]#
: 0 : 0 : : : : 00:00:00 0 0 0
7. Add the virtual devices that will be used for the snaps created against the R2 devices to the device group.
symdg -g <device group name> addall -sid <symm serial #> -devs <symm dev>:<symm dev> -vdev
Group Type Device Group in GNS Valid Symmetrix ID Group Creation Time Vendor ID Application ID Number Number Number Number Number Number Number Number Number Number Number Number Number of of of of of of of of of of of of of
STD Devices in Group : Associated GK's : Locally-associated BCV's : Locally-associated VDEV's : Locally-associated TGT's : Remotely-associated VDEV's(STD RDF): Remotely-associated BCV's (STD RDF): Remotely-associated TGT's(TGT RDF) : Remotely-associated BCV's (BCV RDF): Remotely-assoc'd RBCV's (RBCV RDF) : Remotely-assoc'd BCV's (Hop-2 BCV) : Remotely-assoc'd VDEV's(Hop-2 VDEV): Remotely-assoc'd TGT's (Hop-2 TGT) :
82
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Number of Composite Groups Composite Group Names Standard (STD) Devices (17): {
: 0 : N/A
-------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------DEV001 /dev/emcpowerf 0411 RDF2+TDEV (M) RW 2955015 DEV002 N/A 041D RDF2+TDEV (M) RW 2955015 DEV003 N/A 0429 RDF2+TDEV (M) RW 2955015 DEV004 N/A 0435 RDF2+TDEV (M) RW 2955015 DEV005 N/A 0441 RDF2+TDEV (M) RW 2955015 DEV006 N/A 044D RDF2+TDEV (M) RW 2955015 DEV007 N/A 0459 RDF2+TDEV (M) RW 2955015 DEV008 N/A 0465 RDF2+TDEV (M) RW 2955015 DEV009 N/A 0471 RDF2+TDEV (M) RW 2955015 DEV010 N/A 047D RDF2+TDEV (M) RW 2955015 DEV011 N/A 0489 RDF2+TDEV (M) RW 2955015 DEV012 N/A 0495 RDF2+TDEV (M) RW 2955015 DEV013 N/A 04A1 RDF2+TDEV (M) RW 2955015 DEV014 N/A 04AD RDF2+TDEV (M) RW 2955015 DEV015 N/A 04B9 RDF2+TDEV (M) RW 2955015 DEV016 N/A 04C5 RDF2+TDEV (M) RW 2955015 DEV017 N/A 04D1 RDF2+TDEV (M) RW 2955015 } VDEV Devices Locally-associated (17): { -------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------VDEV001 N/A 04DF VDEV (M) NR 2955015 VDEV002 N/A 04EB VDEV (M) NR 2955015 VDEV003 N/A 04F7 VDEV (M) NR 2955015 VDEV004 N/A 0503 VDEV (M) NR 2955015 VDEV005 N/A 050F VDEV (M) NR 2955015 VDEV006 N/A 051B VDEV (M) NR 2955015 VDEV007 N/A 0527 VDEV (M) NR 2955015 VDEV008 N/A 0533 VDEV (M) NR 2955015 VDEV009 N/A 053F VDEV (M) NR 2955015 VDEV010 N/A 054B VDEV (M) NR 2955015 VDEV011 N/A 0557 VDEV (M) NR 2955015 VDEV012 N/A 0563 VDEV (M) NR 2955015 VDEV013 N/A 056F VDEV (M) NR 2955015 VDEV014 N/A 057B VDEV (M) NR 2955015
Creating a single SNAP of R2 standard devices
83
Device Group RDF Information { RDF Type RDF (RA) Group Number Remote Symmetrix ID
: R2 : 1 : 000192604467
(00)
R2 Device Is Larger Than The R1 Device : False Paired with a Diskless Device Paired with a Concurrent Device Paired with a Cascaded Device Thick Thin Relationship RDF Pair Configuration RDF STAR Mode RDF RDF RDF RDF Mode Adaptive Copy Adaptive Copy Write Pending State Adaptive Copy Skew (Tracks) : : : : False False False False
RDF Device Domino RDF Link Configuration RDF Link Domino Prevent Automatic RDF Link Recovery Prevent RAs Online Upon Power ON Device RDF Status
: Ready
Device RA Status : Ready Device Link Status : Not Ready Time of Last Device Link Status Change : N/A Device Suspend State Device Consistency State Device Consistency Exempt State RDF R2 Not Ready If Invalid Device Write Pacing Exempt State Effective Write Pacing Exempt State Device RDF State Remote Device RDF State RDF Pair State ( R1 <=\=> R2 ) : : : : : : Offline Enabled Disabled Disabled Disabled Disabled
(RW) (RW)
84
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Number of R1 Invalid Tracks Number of R2 Invalid Tracks RDFA Information: { Session Number Cycle Number Number of Devices in the Session Session Status Consistency Exempt Devices Write Pacing Exempt Devices Session Consistency State Minimum Cycle Time Average Cycle Time Duration of Last cycle Session Priority
: :
106 0
: : : : : : : : : : :
Tracks not Committed to the R2 Side: 0 Time that R2 is behind R1 : 291:46:58 R2 Image Capture Time : Wed Jun 15 07:43:38 2011 R2 Data is Consistent : True R1 Side Percent Cache In Use : 0 R2 Side Percent Cache In Use : 0 Transmit Idle Time R1 Side DSE Used Tracks R2 Side DSE Used Tracks R1 Side Shared Tracks } } [root@smdw config]# : : : : 00:00:00 0 0 0
If the save device pool is the 'DEFAULT_POOL' then the -svp <save device pool name> parameters can be dropped as the 'DEFAULT_POOL' name is the default.
10. Query the device group to verify the snap was created.
symsnap -g <device group name> query
Source Device Target Device State Copy ------------------------- -------------------- ---------- ------------ ---Protected Changed Logical Sym Tracks Logical Sym GD Tracks SRC <=> TGT (%) ------------------------- -------------------- ---------- ------------ ---DEV001 DEV002 DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017 Total Track(s) MB(s) Legend:
86
0411 041D 0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1
47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 -------803764080 50235264
VDEV001 VDEV002 VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017
04DF 04EB 04F7 0503 050F 051B 0527 0533 053F 054B 0557 0563 056F 057B 0587 0593 059F
X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. ---------0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Created Created Created Created Created Created Created Created Created Created Created Created Created Created Created Created Created
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
(G): X . (D): X M
The Target device is associated with this group, The Target device is not associated with this group. The Target device has one or more inactive duplicates. The Target device has one or more inactive duplicates AND maximum inactive duplicates exist for this Source device. . = The Target device has no inactive duplicates.
= = = =
[root@smdw ~]#
11. Activate the snap so it is available for mounting on a host or set of hosts. This puts the virtual devices in a "copy on write" state so that any changes to the R2 that the snap was taken of or the virtual devices used for the snap are tracked. To make sure that all devices are activated at the same time,use the consistent option with the activate command.
symsnap -g <device group name> activate -consistent -noprompt
12. Query the device group to verify the snap was activated. The devices should now be in a "CopyOnWrite" state.
symsnap -g <device group name> query
Source Device Target Device State Copy ------------------------- -------------------- ---------- ------------ ---Protected Changed Logical Sym Tracks Logical Sym GD Tracks SRC <=> TGT (%) ------------------------- -------------------- ---------- ------------ ---DEV001 DEV002 DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017 Total Track(s) MB(s) Legend: (G): X . (D): X M The Target device is associated with this group, The Target device is not associated with this group. The Target device has one or more inactive duplicates. The Target device has one or more inactive duplicates AND maximum inactive duplicates exist for this Source device. . = The Target device has no inactive duplicates. = = = = 0411 041D 0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 -------803764080 50235264 VDEV001 VDEV002 VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017 04DF 04EB 04F7 0503 050F 051B 0527 0533 053F 054B 0557 0563 056F 057B 0587 0593 059F X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. ---------0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[root@smdw ~]#
13. Mount the snap to another host or set of hosts make sure the virtual devices used for the snap are visible to the host either using symaccess commands or another means. Once the devices are assigned to the host(s) performs a rescan on the hosts and mounts them to the appropriate locations. 14. Recreate the snap session so it picks up all changes from the R2 and starts tracking changes from this new point.
88
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Make sure if the snap was mounted that it is unmounted before recreating the snap.
symsnap -g <device group name> recreate -noprompt
15. Query the device group to verify the snap was recreated.
symsnap -g <device group name> query
Source Device Target Device State Copy ------------------------- -------------------- ---------- ------------ ---Protected Changed Logical Sym Tracks Logical Sym GD Tracks SRC <=> TGT (%) ------------------------- -------------------- ---------- ------------ ---DEV001 DEV002 DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017 0411 041D 0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 VDEV001 VDEV002 VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017 04DF 04EB 04F7 0503 050F 051B 0527 0533 053F 054B 0557 0563 056F 057B 0587 0593 059F X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated Recreated 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
-------803764080 50235264
---------0 0
The Target device is associated with this group, The Target device is not associated with this group. The Target device has one or more inactive duplicates. The Target device has one or more inactive duplicates AND maximum inactive duplicates exist for this Source device. . = The Target device has no inactive duplicates.
= = = =
[root@smdw ~]#
17. Query the device group to verify the snap was destroyed.
symsnap -g <device group name> query
Device group 'DCA2880_RDF2_SNAP_1' does not have any devices that are Snap source devices [root@smdw ~]#
90
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
With this environment variable set, Solutions Enabler will allow up to 128 snaps for a single source. The root users .bash_profile file is located in root's home directory. Typically, /root/.bash_profile a. Enable SYMCLI_MULTI_VIRTUAL_SNAP by adding the following lines in the .bash_profile file of the user that will conduct the snap operations. Typically root. SYMCLI_MULTI_VIRTUAL_SNAP=ENABLED export SYMCLI_MULTI_VIRTUAL_SNAP b. Source the .bash_profile file after saving the changes to pick up the new environment parameters. source .bash_profile 3. Verify that the Symmetrix where the snap will be conducted has the proper amount of snap save devices configured for the number of snap sessions that will be created and the save devices are assigned to a snap pool.
symsnap list -sid <symm serial #> -savedevs
92
---0
4. To list available save device pools (snap pools) and verify the state of the pool, make sure the pool that will be used is Enabled.
symsnap -sid <symm serial #> -pools list
S A V E D E V I C E P O O L S -------------------------------------------------------------------------------Dev Total Enabled Used Free Full Pool Session Pool Name Emul Tracks Tracks Tracks Tracks (%) State Status -------------------------------------------------------------------------------DEFAULT_POOL FBA 0 0 0 0 0 Disabled Inactive DEFAULT_POOL 3390 0 0 0 0 0 Disabled Inactive DEFAULT_POOL 3380 0 0 0 0 0 Disabled Inactive DEFAULT_POOL AS400 0 0 0 0 0 Disabled Inactive R2_SNAP_POOL FBA 594870540 594870540 0 594870540 0 Enabled Inactive [root@smdw ~]#
5. Verify that the Symmetrix where the snap will be conducted has the proper amount of virtual devices configured for the number of snap sessions that will be created and that they have the same layout as the R2 standard volumes.
symdev list -sid <symm serial #> -vdev -meta
04F7 Not Visible VDEV 0503 Not Visible VDEV 050F Not Visible VDEV 051B Not Visible VDEV 0527 Not Visible VDEV 0533 Not Visible VDEV 053F Not Visible VDEV 054B Not Visible VDEV 0557 Not Visible VDEV 0563 Not Visible VDEV . Data cut due to output length 071F Not Visible VDEV 072B Not Visible VDEV 0737 Not Visible VDEV 0743 Not Visible VDEV 074F Not Visible VDEV 075B Not Visible VDEV [root@mdw gpadmin]#
N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd N/Asst'd
Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped Striped
960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k 960k
12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12
2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015 2955015
7. Add the R2 standard devices to device group 1 that was just created.
symdg -g <device group name> addall -sid <symm serial #> -devs <start dev>:<end dev>
8. Add the virtual devices to the device group 1 that will be used for snap session 1.
symdg -g <device group name> addall -sid 452 -devs <start dev>:<end dev> -vdev
9. Display the device group to verify the standard devices and the virtual devices were correctly added to the group.
symdg show <device group name>
94
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Group Type Device Group in GNS Valid Symmetrix ID Group Creation Time Vendor ID Application ID Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Composite
STD Devices in Group : 17 Associated GK's : 0 Locally-associated BCV's : 0 Locally-associated VDEV's : 17 Locally-associated TGT's : 0 Remotely-associated VDEV's(STD RDF): 0 Remotely-associated BCV's (STD RDF): 0 Remotely-associated TGT's(TGT RDF) : 0 Remotely-associated BCV's (BCV RDF): 0 Remotely-assoc'd RBCV's (RBCV RDF) : 0 Remotely-assoc'd BCV's (Hop-2 BCV) : 0 Remotely-assoc'd VDEV's(Hop-2 VDEV): 0 Remotely-assoc'd TGT's (Hop-2 TGT) : 0 Composite Groups : 0 Group Names : N/A
Standard (STD) Devices (17): { -------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------DEV001 /dev/emcpowerf 0411 RDF2+TDEV (M) RW 2955015 DEV002 N/A 041D RDF2+TDEV (M) RW 2955015 DEV003 N/A 0429 RDF2+TDEV (M) RW 2955015 DEV004 N/A 0435 RDF2+TDEV (M) RW 2955015 DEV005 N/A 0441 RDF2+TDEV (M) RW 2955015 DEV006 N/A 044D RDF2+TDEV (M) RW 2955015 DEV007 N/A 0459 RDF2+TDEV (M) RW 2955015 DEV008 N/A 0465 RDF2+TDEV (M) RW 2955015 DEV009 N/A 0471 RDF2+TDEV (M) RW 2955015 DEV010 N/A 047D RDF2+TDEV (M) RW 2955015 DEV011 N/A 0489 RDF2+TDEV (M) RW 2955015 DEV012 N/A 0495 RDF2+TDEV (M) RW 2955015 DEV013 N/A 04A1 RDF2+TDEV (M) RW 2955015 DEV014 N/A 04AD RDF2+TDEV (M) RW 2955015
Creating multiple SNAPS of R2 standard devices
95
VDEV Devices Locally-associated (17): { -------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------VDEV001 N/A 04DF VDEV (M) NR 2955015 VDEV002 N/A 04EB VDEV (M) NR 2955015 VDEV003 N/A 04F7 VDEV (M) NR 2955015 VDEV004 N/A 0503 VDEV (M) NR 2955015 VDEV005 N/A 050F VDEV (M) NR 2955015 VDEV006 N/A 051B VDEV (M) NR 2955015 VDEV007 N/A 0527 VDEV (M) NR 2955015 VDEV008 N/A 0533 VDEV (M) NR 2955015 VDEV009 N/A 053F VDEV (M) NR 2955015 VDEV010 N/A 054B VDEV (M) NR 2955015 VDEV011 N/A 0557 VDEV (M) NR 2955015 VDEV012 N/A 0563 VDEV (M) NR 2955015 VDEV013 N/A 056F VDEV (M) NR 2955015 VDEV014 N/A 057B VDEV (M) NR 2955015 VDEV015 N/A 0587 VDEV (M) NR 2955015 VDEV016 N/A 0593 VDEV (M) NR 2955015 VDEV017 N/A 059F VDEV (M) NR 2955015 } Device Group RDF Information { RDF Type RDF (RA) Group Number Remote Symmetrix ID
: R2 : 1 : 000192604467
(00)
R2 Device Is Larger Than The R1 Device : False Paired with a Diskless Device Paired with a Concurrent Device Paired with a Cascaded Device Thick Thin Relationship RDF Pair Configuration RDF STAR Mode RDF Mode RDF Adaptive Copy RDF Adaptive Copy Write Pending State : : : : False False False False
96
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
RDF Adaptive Copy Skew (Tracks) RDF Device Domino RDF Link Configuration RDF Link Domino Prevent Automatic RDF Link Recovery Prevent RAs Online Upon Power ON Device RDF Status
: Ready
Device RA Status : Ready Device Link Status : Not Ready Time of Last Device Link Status Change : N/A Device Suspend State Device Consistency State Device Consistency Exempt State RDF R2 Not Ready If Invalid Device Write Pacing Exempt State Effective Write Pacing Exempt State Device RDF State Remote Device RDF State RDF Pair State ( R1 <=\=> R2 ) : : : : : : Offline Enabled Disabled Disabled Disabled Disabled
(RW) (RW)
Number of R1 Invalid Tracks Number of R2 Invalid Tracks RDFA Information: { Session Number Cycle Number Number of Devices in the Session Session Status Consistency Exempt Devices Write Pacing Exempt Devices Session Consistency State Minimum Cycle Time Average Cycle Time Duration of Last cycle Session Priority
: : : : : : : : : : :
Tracks not Committed to the R2 Side: 0 Time that R2 is behind R1 : 291:46:58 R2 Image Capture Time : Wed Jun 15 07:43:38 2011 R2 Data is Consistent : True R1 Side Percent Cache In Use : 0 R2 Side Percent Cache In Use : 0
Transmit Idle Time R1 Side DSE Used Tracks R2 Side DSE Used Tracks R1 Side Shared Tracks } } [root@smdw config]#
: : : :
00:00:00 0 0 0
10. Create a second device group for the second snap session.
symdg create <device group name> -type RDF2
11. )Add the same R2 standard devices that were added to device group 1 to device group 2 that was just created.
IMPORTANT If the SYMAPI environment parameters were not enabled from step 1 of this section this step will fail since by default a device can only be in one device group at a time by default.
symdg -g <device group name> addall -sid <symm serial #> -devs <start dev>:<end dev>
12. Add the virtual devices to device group 2 that will be used for the second snap session.
Note: These should not be the same virtual devices added to device group 1 in step 8 of this section.
symdg -g <device group name> addall -sid <symm serial #> -devs <start dev>:<end dev> -vdev
13. Display the device group to verify the R2 standard devices and virtual devices were added correctly to the group
symdg show <device group name>
98
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Group Type Device Group in GNS Valid Symmetrix ID Group Creation Time Vendor ID Application ID Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Number of Composite
STD Devices in Group : 17 Associated GK's : 0 Locally-associated BCV's : 0 Locally-associated VDEV's : 17 Locally-associated TGT's : 0 Remotely-associated VDEV's(STD RDF): 0 Remotely-associated BCV's (STD RDF): 0 Remotely-associated TGT's(TGT RDF) : 0 Remotely-associated BCV's (BCV RDF): 0 Remotely-assoc'd RBCV's (RBCV RDF) : 0 Remotely-assoc'd BCV's (Hop-2 BCV) : 0 Remotely-assoc'd VDEV's(Hop-2 VDEV): 0 Remotely-assoc'd TGT's (Hop-2 TGT) : 0 Composite Groups : 0 Group Names : N/A
Standard (STD) Devices (17): { -------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------DEV001 /dev/emcpowerf 0411 RDF2+TDEV (M) RW 2955015 DEV002 N/A 041D RDF2+TDEV (M) RW 2955015 DEV003 N/A 0429 RDF2+TDEV (M) RW 2955015 DEV004 N/A 0435 RDF2+TDEV (M) RW 2955015 DEV005 N/A 0441 RDF2+TDEV (M) RW 2955015 DEV006 N/A 044D RDF2+TDEV (M) RW 2955015 DEV007 N/A 0459 RDF2+TDEV (M) RW 2955015 DEV008 N/A 0465 RDF2+TDEV (M) RW 2955015 DEV009 N/A 0471 RDF2+TDEV (M) RW 2955015 DEV010 N/A 047D RDF2+TDEV (M) RW 2955015 DEV011 N/A 0489 RDF2+TDEV (M) RW 2955015 DEV012 N/A 0495 RDF2+TDEV (M) RW 2955015 DEV013 N/A 04A1 RDF2+TDEV (M) RW 2955015 DEV014 N/A 04AD RDF2+TDEV (M) RW 2955015
Creating multiple SNAPS of R2 standard devices
99
VDEV Devices Locally-associated (17): { -------------------------------------------------------------------------------Sym Device Cap LdevName PdevName Dev Config Att. Sts (MB) -------------------------------------------------------------------------------VDEV001 N/A 05AB VDEV (M) NR 2955015 VDEV002 N/A 05B7 VDEV (M) NR 2955015 VDEV003 N/A 05C3 VDEV (M) NR 2955015 VDEV004 N/A 05CF VDEV (M) NR 2955015 VDEV005 N/A 05DB VDEV (M) NR 2955015 VDEV006 N/A 05E7 VDEV (M) NR 2955015 VDEV007 N/A 05F3 VDEV (M) NR 2955015 VDEV008 N/A 05FF VDEV (M) NR 2955015 VDEV009 N/A 060B VDEV (M) NR 2955015 VDEV010 N/A 0617 VDEV (M) NR 2955015 VDEV011 N/A 0623 VDEV (M) NR 2955015 VDEV012 N/A 062F VDEV (M) NR 2955015 VDEV013 N/A 063B VDEV (M) NR 2955015 VDEV014 N/A 0647 VDEV (M) NR 2955015 VDEV015 N/A 0653 VDEV (M) NR 2955015 VDEV016 N/A 065F VDEV (M) NR 2955015 VDEV017 N/A 066B VDEV (M) NR 2955015 } Device Group RDF Information { RDF Type RDF (RA) Group Number Remote Symmetrix ID
: R2 : 1 : 000192604467
(00)
R2 Device Is Larger Than The R1 Device : False Paired with a Diskless Device Paired with a Concurrent Device Paired with a Cascaded Device Thick Thin Relationship RDF Pair Configuration RDF STAR Mode RDF Mode RDF Adaptive Copy RDF Adaptive Copy Write Pending State : : : : False False False False
100
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
RDF Adaptive Copy Skew (Tracks) RDF Device Domino RDF Link Configuration RDF Link Domino Prevent Automatic RDF Link Recovery Prevent RAs Online Upon Power ON Device RDF Status
: Ready
Device RA Status : Ready Device Link Status : Not Ready Time of Last Device Link Status Change : N/A Device Suspend State Device Consistency State Device Consistency Exempt State RDF R2 Not Ready If Invalid Device Write Pacing Exempt State Effective Write Pacing Exempt State Device RDF State Remote Device RDF State RDF Pair State ( R1 <=\=> R2 ) : : : : : : Offline Enabled Disabled Disabled Disabled Disabled
(RW) (RW)
Number of R1 Invalid Tracks Number of R2 Invalid Tracks RDFA Information: { Session Number Cycle Number Number of Devices in the Session Session Status Consistency Exempt Devices Write Pacing Exempt Devices Session Consistency State Minimum Cycle Time Average Cycle Time Duration of Last cycle Session Priority
: : : : : : : : : : :
Tracks not Committed to the R2 Side: 0 Time that R2 is behind R1 : 291:47:05 R2 Image Capture Time : Wed Jun 15 07:43:38 2011 R2 Data is Consistent : True R1 Side Percent Cache In Use : 0 R2 Side Percent Cache In Use : 0
Transmit Idle Time R1 Side DSE Used Tracks R2 Side DSE Used Tracks R1 Side Shared Tracks } } [root@smdw config]#
: : : :
00:00:00 0 0 0
14. Repeat steps 10 - 13 for any additional snap sessions that are needed. 15. Follow steps 9 - 17 from Creating a single SNAP of R2 standard devices on page 77 to perform snap operations on any of the newly created device groups. Each device group will be treated independently from the others, so it is possible to generate rolling snaps of the Greenplum database to maintain multiple point-in-time copies.
102
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
H
Mounting and Checking a Symmetrix SNAP of the Greenplum Database
103
2. Query the device group to verify the snap was activated. Devices should report that they are in "CopyOnWrite" state.
symsnap -g <device group name> query
104
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
DEV001 DEV002 DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017 Total Track(s) MB(s)
0411 041D 0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1
47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 -------803764080 50235264
VDEV001 VDEV002 VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017
05AB 05B7 05C3 05CF 05DB 05E7 05F3 05FF 060B 0617 0623 062F 063B 0647 0653 065F 066B
X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. ---------0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Legend: (G): X = The Target device is associated with this group, . = The Target device is not associated with this group. (D): X = The Target device has one or more inactive duplicates. M = The Target device has one or more inactive duplicates AND maximum inactive duplicates exist for this Source device. . = The Target device has no inactive duplicates.
3. Assign the SNAP virtual devices to the appropriate storage groups for the DCA nodes where the devices need to be mounted. a. Gather information from each of the DCA nodes to identify which system is using which R1 device so that they can be mapped to the R2 and subsequently to the virtual device used for the SNAP.
gpssh -f <host file> 'powermt display dev=all |egrep "Pseudo|Logical"'
b. Run a query against the SRDF device group to get the device R1/R2 mapping to compare with the SNAP device group to identify how the devices are paired
symrdf cg <composite group> query
c. Run a query against the SNAP device group to map the R2 to virtual device used for the snap
symsnap g <device group name> query
d. Add the devices to the appropriate storage group based on the mapping identified in steps a c.
symaccess sid <symm serial #> -type storage name <sg name> add devs <symdev>,<symdev>
106
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
Source (R1) View Target (R2) View MODES -------------------------------- ------------------------- ----ST LI ST Standard A N A Logical Sym T R1 Inv R2 Inv K T R1 Inv R2 Inv Device Dev E Tracks Tracks S Dev E Tracks Tracks MDAE -------------------------------- -- ----------------------- ----DEV4467* 0267 RW 0 0 RW 0411 WD 0 0 S... X DEV4467* 0273 RW 0 0 RW 041D WD 0 0 S... X DEV4467* 027F RW 0 0 RW 0429 WD 0 0 S... X DEV4467* 028B RW 0 0 RW 0435 WD 0 0 S... X DEV4467* 0297 RW 0 0 RW 0441 WD 0 0 S... X DEV4467* 02A3 RW 0 0 RW 044D WD 0 0 S... X DEV4467* 02AF RW 0 0 RW 0459 WD 0 0 S... X DEV4467* 02BB RW 0 0 RW 0465 WD 0 0 S... X DEV4467* 02C7 RW 0 0 RW 0471 WD 0 0 S... X DEV4467* 02D3 RW 0 0 RW 047D WD 0 0 S... X DEV4467* 02DF RW 0 0 RW 0489 WD 0 0 S... X DEV4467* 02EB RW 0 0 RW 0495 WD 0 0 S... X DEV4467* 02F7 RW 0 0 RW 04A1 WD 0 0 S... X DEV4467* 0303 RW 0 0 RW 04AD WD 0 0 S... X DEV4467* 030F RW 0 0 RW 04B9 WD 0 0 S... X DEV4467* 031B RW 0 0 RW 04C5 WD 0 0 S... X DEV4467* 0327 RW 0 0 RW 04D1 WD 0 0 S... X Total Track(s) MBs Legend for MODES: M(ode of Operation) : D(omino) : A(daptive Copy) : (Consistency) E(xempt): Legend for STATES: Cons(istency State) Susp(end State) A X D X = = = = Async, S = Enabled, . Disk Mode, Enabled, . ------- ------0 0 0.0 0.0 ------- ------0 0 0.0 0.0
STATES -----C S o u n s RDF Pair s p STATE -----Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized Synchronized
Sync, E = Semi-sync, C = Adaptive Copy = Disabled W = WP Mode, . = ACp off = Disabled, M = Mixed, - = N/A
Device Group (DG) Name: DCA2880_RDF2_SNAP_2 DG's Type : RDF2 DG's Symmetrix ID : 000192604452
Mounting and checking Symmetrix SNAP of Greenplum database
107
Source Device Target Device State Copy ------------------------- -------------------- ---------- ------------ ---Protected Changed Logical Sym Tracks Logical Sym GD Tracks SRC <=> TGT (%) ------------------------- -------------------- ---------- ------------ ---DEV001 DEV002 DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017 Total Track(s) MB(s) 0411 041D 0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 47280240 -------803764080 50235264 VDEV001 VDEV002 VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017 05AB 05B7 05C3 05CF 05DB 05E7 05F3 05FF 060B 0617 0623 062F 063B 0647 0653 065F 066B X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. X. ---------0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite CopyOnWrite 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Legend: (G): X = The Target device is associated with this group, . = The Target device is not associated with this group. (D): X = The Target device has one or more inactive duplicates. M = The Target device has one or more inactive duplicates AND maximum inactive duplicates exist for this Source device. . = The Target device has no inactive duplicates. [root@smdw ~]# [root@smdw ~]# symaccess -sid 452 list Symmetrix ID : 000192604452 Group Name Type -------------------------------- --------mdw1_ig Initiator sdw1_ig Initiator sdw2_ig Initiator sdw3_ig Initiator sdw4_ig Initiator sdw5_ig Initiator sdw6_ig Initiator
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
108
sdw7_ig sdw8_ig DCA_PG mdw1_sg sdw1_sg sdw2_sg sdw3_sg sdw4_sg sdw5_sg sdw6_sg sdw7_sg sdw8_sg
Initiator Initiator Port Storage Storage Storage Storage Storage Storage Storage Storage Storage
[root@smdw ~]# symaccess -sid 452 -type storage -name mdw1_sg add devs 05AB [root@smdw ~]# symaccess -sid 452 -type storage -name sdw1_sg add devs 05B7,05C3 [root@smdw ~]# symaccess -sid 452 -type storage -name sdw2_sg add devs 05CF,05DB [root@smdw ~]# symaccess -sid 452 -type storage -name sdw3_sg add devs 05E7,05F3 [root@smdw ~]# symaccess -sid 452 -type storage -name sdw4_sg add devs 05FF,060B [root@smdw ~]# symaccess -sid 452 -type storage -name sdw5_sg add devs 0617,0623 [root@smdw ~]# symaccess -sid 452 -type storage -name sdw6_sg add devs 062F,063B [root@smdw ~]# symaccess -sid 452 -type storage -name sdw7_sg add devs 0647,0653 [root@smdw ~]# symaccess -sid 452 -type storage -name sdw8_sg add devs 065F,066B [root@smdw ~]#
4. Force the new DCA nodes where the SNAP will be mounted to probe their buses to find the newly added devices. The command provided when used with gpssh will successfully probe all required buses on the DCA. Some errors will occur but can be ignored because the host buses 3 and 4 do not exist on the segment servers. They only exist on the master and standby master. For i in 0 1 2 3 4, do
echo - - - > /sys/class/scsi_host/host$i/scan; done
[sdw2] bash: /sys/class/scsi_host/host4/scan: [sdw3] bash: /sys/class/scsi_host/host3/scan: [sdw3] bash: /sys/class/scsi_host/host4/scan: [ mdw] [sdw1] bash: /sys/class/scsi_host/host3/scan: [sdw1] bash: /sys/class/scsi_host/host4/scan: [root@smdw ~]#
No such file or directory No such file or directory No such file or directory No such file or directory No such file or directory
5. Issue a powermt config to each of the segment servers to generate power devices for the newly found devices.
powermt config
6. Verify that the newly added devices are now configured with power devices.
Note: If more than one set of snap devices were assigned to the host, multiple power devices will be seen. powermt display dev=all |egrep "Pseudo|Logical"
110
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
8. Mount the virtual device containing the master database on the master server.
mount /dev/<emcpower device> <mountpoint> -o noatime,inode64,allocsize=16m
[sdw4] [root@smdw ~]# gpssh -h sdw5 'mount noatime,inode64,allocsize=16m;mount noatime,inode64,allocsize=16m' [sdw5] [root@smdw ~]# gpssh -h sdw6 'mount noatime,inode64,allocsize=16m;mount noatime,inode64,allocsize=16m' [sdw6] [root@smdw ~]# gpssh -h sdw7 'mount noatime,inode64,allocsize=16m;mount noatime,inode64,allocsize=16m' [sdw7] [root@smdw ~]# gpssh -h sdw8 'mount noatime,inode64,allocsize=16m;mount noatime,inode64,allocsize=16m' [sdw8] [root@smdw ~]#
112
[sdw3] /dev/emcpowerj [ mdw] /dev/emcpowere [sdw1] /dev/sdb [sdw1] /dev/sdd [sdw1] /dev/emcpowerj [sdw1] /dev/emcpowerk [root@smdw ~]#
1% 1% 8% 8% 1% 1%
14. Run a query against the gp_segment_configuration and pg_filespace_entry table spaces and verify that all of mirror segments were up at the time of the snap. If any of the mirror segments were down during the snap then the snap does not contain a consistent copy of the database since the R2 device associated with the mirror segment that was down was not updated from the primary segment and therefore does not contain the latest data.
select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
19 |
17 | p
| p
| s
| u
| sdw3
| /data2/primary/gpseg17
-- Data cut due to output length 94 | 95 | 96 | 97 | (97 rows) 44 45 46 47 | | | | m m m m | | | | m m m m | | | | s s s s | | | | u u u u | | | | sdw3 sdw4 sdw5 sdw6 | | | | /mirror2/mirror/gpseg44 /mirror1/mirror/gpseg45 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
srdftesting=#
15. If all of the mirror segments were up and synchronized at the time of the snap, bring the database online on the recovery DCA using only the mirror segments. Update the gp_segment_configuration table space and mark the primary segment roles as 'mirror' and the status as 'down'.
update gp_segment_configuration sc set role='m',status='d' from pg_filespace_entry fse where sc.dbid=fse.fsedbid and fse.fselocation like'/<unique dir path value>/%' and sc.content >= 0;
14 15 16 17 18 19
| | | | | |
12 13 14 15 16 17
| | | | | |
m m m m m m
| | | | | |
p p p p p p
| | | | | |
s s s s s s
| | | | | |
d d d d d d
| | | | | |
| | | | | |
-- Data cut due to output length 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | (97 rows) 40 41 42 43 44 45 46 47 | | | | | | | | m m m m m m m m | | | | | | | | m m m m m m m m | | | | | | | | s s s s s s s s | | | | | | | | u u u u u u u u | | | | | | | | sdw4 sdw5 sdw1 sdw2 sdw3 sdw4 sdw5 sdw6 | | | | | | | | /mirror1/mirror/gpseg40 /mirror1/mirror/gpseg41 /mirror2/mirror/gpseg42 /mirror2/mirror/gpseg43 /mirror2/mirror/gpseg44 /mirror1/mirror/gpseg45 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
srdftesting=#
16. Update the gp_segment_configuration table space and set the mirror segments to a role of 'primary' and a mode of 'change tracking'.
update gp_segment_configuration sc set role='p',mode='c' from pg_filespace_entry fse where sc.dbid=fse.fsedbid and fse.fselocation like'/<unique dir path value>/%' and sc.content >= 0;
116
9 10 11 12 13 14 15 16 17 18 19 20
| | | | | | | | | | | |
7 8 9 10 11 12 13 14 15 16 17 18
| | | | | | | | | | | |
m m m m m m m m m m m m
| | | | | | | | | | | |
p p p p p p p p p p p p
| s | s | s | s | s | s | s | s | s | s | s | s
| d | d | d | d | d | d | d | d | d | d | d | d
| sdw2 | sdw2 | sdw2 | sdw2 | sdw2 | sdw3 | sdw3 | sdw3 | sdw3 | sdw3 | sdw3 | sdw4
| /data1/primary/gpseg7 | /data1/primary/gpseg8 | /data2/primary/gpseg9 | /data2/primary/gpseg10 | /data2/primary/gpseg11 | /data1/primary/gpseg12 | /data1/primary/gpseg13 | /data1/primary/gpseg14 | /data2/primary/gpseg15 | /data2/primary/gpseg16 | /data2/primary/gpseg17 | /data1/primary/gpseg18
-- Data cut due to output length 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | (97 rows) 40 41 42 43 44 45 46 47 | | | | | | | | p p p p p p p p | | | | | | | | m m m m m m m m | | | | | | | | c c c c c c c c | | | | | | | | u u u u u u u u | | | | | | | | sdw4 sdw5 sdw1 sdw2 sdw3 sdw4 sdw5 sdw6 | | | | | | | | /mirror1/mirror/gpseg40 /mirror1/mirror/gpseg41 /mirror2/mirror/gpseg42 /mirror2/mirror/gpseg43 /mirror2/mirror/gpseg44 /mirror1/mirror/gpseg45 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
srdftesting=#
17. Exit out of the psql session and back to the command prompt.
q
18. Stop the database master server so it is removed from maintenance mode.
gpstop -m
20110701:07:20:42:gpstop:smdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart' 20110701:07:20:42:gpstop:smdw:gpadmin-[INFO]:-Master host=mdw 20110701:07:20:42:gpstop:smdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart 20110701:07:20:42:gpstop:smdw:gpadmin-[INFO]:-Master segment instance directory=/data/master/gpseg-1 [gpadmin@smdw ~]$
19. Restart the database normally. Watch to make sure that the primary segments were skipped and only the mirror segments are started.
gpstart
118
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
20110701:07:20:57:gpstart:smdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw8 directory /data2/primary/gpseg47 <<<<< 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:--------------------------20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Master instance parameters 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:--------------------------20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Database = template1 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Master Port = 5432 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Master directory = /data/master/gpseg-1 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Timeout = 600 seconds 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Master standby = Off 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:-Segment instances that will be started 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:Host Datadir Port Role 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw2 /mirror2/mirror/gpseg0 50000 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw3 /mirror2/mirror/gpseg1 50001 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw4 /mirror2/mirror/gpseg2 50002 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw5 /mirror1/mirror/gpseg3 50003 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw6 /mirror1/mirror/gpseg4 50004 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw7 /mirror1/mirror/gpseg5 50005 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:- sdw3 /mirror2/mirror/gpseg6 50000 Primary -- Data cut due to output length 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:50004 Primary 20110701:07:20:57:gpstart:smdw:gpadmin-[INFO]:50005 Primary sdw5 sdw6 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
Continue with Greenplum instance startup Yy|Nn (default=N): > y 20110701:07:21:02:gpstart:smdw:gpadmin-[INFO]:-No standby master configured. skipping... 20110701:07:21:02:gpstart:smdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... ... 20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:-Process results... 20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:- Successful segment starts = 48 20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:- Failed segment starts = 0
Mounting and checking Symmetrix SNAP of Greenplum database
119
20110701:07:21:05:gpstart:smdw:gpadmin-[WARNING]:-Skipped segment starts (segments are marked down in configuration) = 48 <<<<<<<< 20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:-Successfully started 48 of 48 segment instances, skipped 48 other segments 20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110701:07:21:05:gpstart:smdw:gpadmin-[WARNING]:-****************************** 20110701:07:21:05:gpstart:smdw:gpadmin-[WARNING]:-There are 48 segment(s) marked down in the database 20110701:07:21:05:gpstart:smdw:gpadmin-[WARNING]:-To recover from this current state, review usage of the gprecoverseg 20110701:07:21:05:gpstart:smdw:gpadmin-[WARNING]:-management utility which will recover failed segment instance databases. 20110701:07:21:05:gpstart:smdw:gpadmin-[WARNING]:-****************************** 20110701:07:21:05:gpstart:smdw:gpadmin-[INFO]:-Starting Master instance mdw directory /data/master/gpseg-1 20110701:07:21:06:gpstart:smdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active 20110701:07:21:19:gpstart:smdw:gpadmin-[WARNING]:-Database started but warnings generated <<<<< 20110701:07:21:19:gpstart:smdw:gpadmin-[INFO]:-Check status of database with gpstate utility [gpadmin@smdw ~]$
20. To finish verifying that the SNAP was a valid snap and can be used to conduct a recovery of the primary system, execute gpcheckcat to check all of the catalogs in the database.
/usr/local/greenplum-db/bin/lib/gpcheckcat -p 5432 <database name>
120
20110701:08:41:54:gpcheckcat:default-[INFO]:-[OK] pg_class.reltoastrelid 20110701:08:41:54:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:54:gpcheckcat:default-[INFO]:-Checking for leaked temporary schemas 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] temporary schemas 20110701:08:41:55:gpcheckcat:default-[INFO]:-Checking missing schema definitions 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] missing schema definitions 20110701:08:41:55:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:55:gpcheckcat:default-[INFO]:-Checking constraints on randomly distributed tables 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] randomly distributed tables 20110701:08:41:55:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:55:gpcheckcat:default-[INFO]:-Checking that unique constraints are only on distribution columns 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] unique constraints 20110701:08:41:55:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:55:gpcheckcat:default-[INFO]:-Checking table ownership 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] table ownership 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] type ownership 20110701:08:41:55:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:55:gpcheckcat:default-[INFO]:-Checking indexes 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] relhasindex 20110701:08:41:55:gpcheckcat:default-[INFO]:-[OK] deprecated indexes 20110701:08:41:55:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:55:gpcheckcat:default-[INFO]:-Checking Object Dependencies 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] basic object dependencies 20110701:08:41:57:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:57:gpcheckcat:default-[INFO]:-Checking persistent tables 20110701:08:41:57:gpcheckcat:default-[WARNING]:-System requires recovery via gprecoverseg 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_filespace_node <=> pg_filespace 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_filespace_node <=> gp_global_sequence 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_database_node <=> pg_database 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_database_node <=> pg_tablespace 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_database_node <=> gp_global_sequence 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_tablespace_node <=> pg_tablespace 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_tablespace_node <=> pg_filespace 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_tablespace_node <=> gp_global_sequence 20110701:08:41:57:gpcheckcat:default-[INFO]:-[OK] gp_persistent_relation_node <=> pg_tablespace 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] gp_persistent_relation_node <=> pg_database 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] gp_persistent_relation_node <=> gp_relation_node
Mounting and checking Symmetrix SNAP of Greenplum database
121
20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] gp_persistent_relation_node <=> pg_class 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] gp_persistent_relation_node <=> gp_global_sequence 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] gp_persistent_relation_node <=> filesystem 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] pg_database <=> filesystem 20110701:08:41:58:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:58:gpcheckcat:default-[INFO]:-Performing foreign key and cross database consistency tests 20110701:08:41:58:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:58:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:41:58:gpcheckcat:default-[INFO]:-Checking foreign keys ... 20110701:08:41:58:gpcheckcat:default-[INFO]:- 28 tables, 102 queries 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(aggfnoid) 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(aggtransfn) 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(agginvtransfn) 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(aggprelimfn) 20110701:08:41:58:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(agginvprelimfn) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(aggfinalfn) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(aggsortop) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_aggregate(aggtranstype) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_am(aminsert) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_am(ambeginscan) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_am(amgettuple) 20110701:08:41:59:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_am(amgetmulti) -- Data cut due to output length 20110701:08:42:00:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_window(winprefunc) 20110701:08:42:00:gpcheckcat:default-[INFO]:-[OK] Foreign key check for pg_window(winfinfunc) 20110701:08:42:00:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:42:00:gpcheckcat:default-[INFO]:-Checking cross database consistency ... 20110701:08:42:00:gpcheckcat:default-[INFO]:-[OK] Cross consistency check for gp_version_at_initdb
122
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
20110701:08:42:01:gpcheckcat:default-[INFO]:-[OK] pg_aggregate 20110701:08:42:01:gpcheckcat:default-[INFO]:-[OK] pg_am 20110701:08:42:02:gpcheckcat:default-[INFO]:-[OK] pg_amop 20110701:08:42:02:gpcheckcat:default-[INFO]:-[OK] pg_amproc 20110701:08:42:02:gpcheckcat:default-[INFO]:-[OK] pg_appendonly 20110701:08:42:02:gpcheckcat:default-[INFO]:-[OK] pg_appendonly_alter_column 20110701:08:42:02:gpcheckcat:default-[INFO]:-[OK] pg_attrdef 20110701:08:42:07:gpcheckcat:default-[INFO]:-[OK] pg_attribute -- Data cut due to output length
Cross consistency check for Cross consistency check for Cross consistency check for Cross consistency check for Cross consistency check for Cross consistency check for Cross consistency check for Cross consistency check for
20110701:08:42:18:gpcheckcat:default-[INFO]:-[OK] Cross consistency acl check for pg_proc 20110701:08:42:18:gpcheckcat:default-[INFO]:-[OK] Cross consistency acl check for pg_tablespace 20110701:08:42:18:gpcheckcat:default-[INFO]:-----------------------------------20110701:08:42:18:gpcheckcat:default-[INFO]:-Check complete [gpadmin@smdw ~]$
22. Unmount all of the devices so that they can be used for a recovery if the checks were all successful or so that the SNAP can be recreated until it is identified as a good snap.
umount <mountpoint>
124
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
I
Recovering a DCA Using a SNAP Session and SRDF
125
2. Split the SRDF devices so that the SNAP can be restored to the R2 volumes.
symrdf -cg <composite group name> split -force
126
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
The RDF 'Split' operation successfully executed for composite group 'DCA_R1_CG'. [root@mdw ~]#
3. Query the SRDF devices and verify that they are in a split state.
symrdf query -cg <composite group name>
Symmetrix ID Remote Symmetrix ID RDF (RA) Group Number Source (R1) View -------------------------------ST Standard A Logical Sym T R1 Inv R2 Inv Device Dev E Tracks Tracks -------------------------------DEV4467* 0267 RW 0 0 DEV4467* 0273 RW 0 0 DEV4467* 027F RW 0 0 DEV4467* 028B RW 0 0 DEV4467* 0297 RW 0 0 DEV4467* 02A3 RW 0 0 DEV4467* 02AF RW 0 0 DEV4467* 02BB RW 0 0 DEV4467* 02C7 RW 0 0 DEV4467* 02D3 RW 0 0 DEV4467* 02DF RW 0 0 DEV4467* 02EB RW 0 0 DEV4467* 02F7 RW 0 0 DEV4467* 0303 RW 0 0 DEV4467* 030F RW 0 0 DEV4467* 031B RW 0 0 DEV4467* 0327 RW 0 0 Total Track(s) MBs ------- ------0 0 0.0 0.0
Target (R2) View ------------------------LI ST N A K T R1 Inv R2 Inv S Dev E Tracks Tracks -- ----------------------NR 0411 RW 2462 0 NR 041D RW 5128 0 NR 0429 RW 5085 0 NR 0435 RW 5159 0 NR 0441 RW 5049 0 NR 044D RW 5036 0 NR 0459 RW 5033 0 NR 0465 RW 5146 0 NR 0471 RW 5019 0 NR 047D RW 5017 0 NR 0489 RW 4998 0 NR 0495 RW 5170 0 NR 04A1 RW 5155 0 NR 04AD RW 5088 0 NR 04B9 RW 4963 0 NR 04C5 RW 5122 0 NR 04D1 RW 5046 0 ------- ------0 0 0.0 0.0
MODES STATES ----- ------ ----C S o u n s RDF Pair MDAE s p STATE ----- ------ ----S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split
Legend for MODES: M(ode of Operation) : D(omino) : A(daptive Copy) : (Consistency) E(xempt): Legend for STATES: Cons(istency State) Susp(end State) [root@mdw ~]# : X = Enabled, . = Disabled, M = Mixed, - = N/A : X = Online, . = Offline, P = Offline Pending, - = N/A A X D X = = = = Async, S = Enabled, . Disk Mode, Enabled, . Sync, E = Semi-sync, C = Adaptive Copy = Disabled W = WP Mode, . = ACp off = Disabled, M = Mixed, - = N/A
4. Execute a restore of the SNAP device group to restore the SNAP to the R2 devices.
symsnap -g <device group name> restore
5. Query the SNAP device group to verify the SNAP devices are in the restored state.
symsnap -g <device group name> query -restore
128
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
DEV003 DEV004 DEV005 DEV006 DEV007 DEV008 DEV009 DEV010 DEV011 DEV012 DEV013 DEV014 DEV015 DEV016 DEV017
0429 0435 0441 044D 0459 0465 0471 047D 0489 0495 04A1 04AD 04B9 04C5 04D1
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -------0 0.0
VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017
05C3 05CF 05DB 05E7 05F3 05FF 060B 0617 0623 062F 063B 0647 0653 065F 066B
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ------0 0.0
VDEV003 VDEV004 VDEV005 VDEV006 VDEV007 VDEV008 VDEV009 VDEV010 VDEV011 VDEV012 VDEV013 VDEV014 VDEV015 VDEV016 VDEV017
05C3 05CF 05DB 05E7 05F3 05FF 060B 0617 0623 062F 063B 0647 0653 065F 066B
Restored Restored Restored Restored Restored Restored Restored Restored Restored Restored Restored Restored Restored Restored Restored
100 100 100 100 100 100 100 100 100 100 100 100 100 100 100
The Target device is associated with this group, The Target device is not associated with this group. The Target device has one or more inactive duplicates. The Target device has one or more inactive duplicates AND maximum inactive duplicates exist for this Source device. . = The Target device has no inactive duplicates.
= = = =
[root@smdw ~]#
6. Terminate the SNAP restore session but, leave the SNAP session active in the event something happens with the restore and the SNAP session is needed again.
symsnap -g <device group name> terminate -restored
7. Query the SNAP device group and verify that it is back into the active state and the devices are marked with a status of "CopyOnWrite".
symsnap -g <device group name> query
[root@smdw ~]#
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
130
8. Query the SRDF composite group to verify that there are tracks to update from the R2 to the R1 devices.
symrdf query -cg <composite group name>
: : : : :
DCA_R1_CG RDF1 1 1 SYNC : 000192604467 : 000192604452 : 1 (00) (Microcode Version: 5875) (Microcode Version: 5875)
Target (R2) View ------------------------LI ST N A K T R1 Inv R2 Inv S Dev E Tracks Tracks -- ----------------------NR 0411 RW 2462 0 NR 041D RW 5128 0 NR 0429 RW 5085 0 NR 0435 RW 5159 0 NR 0441 RW 5049 0 NR 044D RW 5036 0 NR 0459 RW 5033 0 NR 0465 RW 5146 0 NR 0471 RW 5019 0 NR 047D RW 5017 0 NR 0489 RW 4998 0 NR 0495 RW 5170 0 NR 04A1 RW 5155 0 NR 04AD RW 5088 0 NR 04B9 RW 4963 0 NR 04C5 RW 5122 0 NR 04D1 RW 5046 0 ------- ------83676 0 5229.8 0.0
MODES STATES ----- -----------C S o u n s RDF Pair MDAE s p STATE ----- -----------S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split S... X . Split
A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off (Consistency) E(xempt): X = Enabled, . = Disabled, M = Mixed, - = N/A Legend for STATES: Cons(istency State) Susp(end State) [root@mdw ~]# : X = Enabled, . = Disabled, M = Mixed, - = N/A : X = Online, . = Offline, P = Offline Pending, - = N/A
9. Issue a restore against the SRDF composite group to restore and sync the R1 devices to the R2 devices using the data on the R2s.
symrdf -cg <composite group name> restore
132
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
10. Query the SRDF composite group until the devices are in a "Synchronized" state indicating that all of the data was restored from the R2 to the R1.
symrdf query -cg <composite group name>
Source (R1) View Target (R2) View MODES STATES -------------------------------- ------------------------- ----- -----------ST LI ST C S Standard A N A o u Logical Sym T R1 Inv R2 Inv K T R1 Inv R2 Inv n s RDF Pair Device Dev E Tracks Tracks S Dev E Tracks Tracks MDAE s p STATE -------------------------------- -- ----------------------- ----- ------ -----DEV4467* 0267 RW 0 0 RW 0411 WD 0 0 S... X Synchronized DEV4467* 0273 RW 0 0 RW 041D WD 0 0 S... X Synchronized DEV4467* 027F RW 0 0 RW 0429 WD 0 0 S... X Synchronized DEV4467* 028B RW 0 0 RW 0435 WD 0 0 S... X Synchronized DEV4467* 0297 RW 0 0 RW 0441 WD 0 0 S... X Synchronized DEV4467* 02A3 RW 0 0 RW 044D WD 0 0 S... X Synchronized DEV4467* 02AF RW 0 0 RW 0459 WD 0 0 S... X Synchronized DEV4467* 02BB RW 0 0 RW 0465 WD 0 0 S... X Synchronized DEV4467* 02C7 RW 0 0 RW 0471 WD 0 0 S... X Synchronized DEV4467* 02D3 RW 0 0 RW 047D WD 0 0 S... X Synchronized DEV4467* 02DF RW 0 0 RW 0489 WD 0 0 S... X Synchronized DEV4467* 02EB RW 0 0 RW 0495 WD 0 0 S... X Synchronized DEV4467* 02F7 RW 0 0 RW 04A1 WD 0 0 S... X Synchronized DEV4467* 0303 RW 0 0 RW 04AD WD 0 0 S... X Synchronized DEV4467* 030F RW 0 0 RW 04B9 WD 0 0 S... X Synchronized DEV4467* 031B RW 0 0 RW 04C5 WD 0 0 S... X Synchronized DEV4467* 0327 RW 0 0 RW 04D1 WD 0 0 S... X Synchronized Total Track(s) MBs Legend for MODES: M(ode of Operation) : A = Async, S = Sync, E = Semi-sync, C = Adaptive Copy
Recovering a DCA using a SNAP session and SRDF
133
D(omino) : X = Enabled, . = Disabled A(daptive Copy) : D = Disk Mode, W = WP Mode, . = ACp off (Consistency) E(xempt): X = Enabled, . = Disabled, M = Mixed, - = N/A Legend for STATES: Cons(istency State) Susp(end State) [root@mdw ~]# : X = Enabled, . = Disabled, M = Mixed, - = N/A : X = Online, . = Offline, P = Offline Pending, - = N/A
11. Mount the R1 Symmetrix devices back to the DCA that were just restored and unmounted in step 1 of this procedure.
mount <mount point>
134
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
20110705:09:01:05:gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information 20110705:09:01:05:gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master... 20110705:09:01:05:gpstart:mdw:gpadmin-[INFO]:-Master Started... [gpadmin@mdw ~]$
15. Verify that the primary segments set to a role of 'm' mirror and a status of 'd' down and that the mirror segments are set to a role of 'p' primary, a mode of 'c' change tracking and a status of 'u' up. The segments should already be set if the SNAP devices were checked properly and were not set back before stopping the database on the DR system and unmounting the volumes.
select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid;
12 | m | 13 | m | 14 | m | 15 | m | 16 | m | 17 | m | 18 | m | cut due to 40 41 42 43 44 45 46 47 | | | | | | | | p p p p p p p p | | | | | | | |
p | s p | s p | s p | s p | s p | s p | s output length m m m m m m m m | | | | | | | | c c c c c c c c
| | | | | | |
d d d d d d d
| | | | | | |
| | | | | | |
| | | | | | | |
u u u u u u u u
| | | | | | | |
| | | | | | | |
srdftesting=#
16. If the primary and mirror segments were set correctly skip to step 19. Otherwise, use the SQL statements in steps 17 and 18 to update the configuration so that the database starts up using the mirror segments that were just restored from the SNAP. 17. To set the primary segments to a role of 'm' and a status of 'd' issue the following SQL statement.
update gp_segment_configuration sc set role='m',status='d' from pg_filespace_entry fse where sc.dbid=fse.fsedbid and fse.fselocation like'/<unique dir path value>/%' and sc.content >= 0;
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
p | p | p | p | p | p | output length
s s s s s s
| | | | | |
d d d d d d
| | | | | |
| | | | | |
| | | | | | | | | |
d d u u u u u u u u
| | | | | | | | | |
sdw8 sdw8 sdw2 sdw3 sdw4 sdw5 sdw6 sdw7 sdw3 sdw4
| | | | | | | | | |
/data2/primary/gpseg46 /data2/primary/gpseg47 /mirror2/mirror/gpseg0 /mirror2/mirror/gpseg1 /mirror2/mirror/gpseg2 /mirror1/mirror/gpseg3 /mirror1/mirror/gpseg4 /mirror1/mirror/gpseg5 /mirror2/mirror/gpseg6 /mirror2/mirror/gpseg7
| u | u
| sdw5 | sdw6
| /mirror1/mirror/gpseg46 | /mirror1/mirror/gpseg47
18. To set the mirror segments to a role of 'p' and a mode of 'c' issue the following SQL statement.
update gp_segment_configuration sc set role='p',mode='c' from pg_filespace_entry fse where sc.dbid=fse.fsedbid and fse.fselocation like'/<unique dir path value>/%' and sc.content >= 0;
p | p | p | p | p | p | p | output length
s s s s s s s
| | | | | | |
d d d d d d d
| | | | | | |
| | | | | | |
| | | | | | | | | |
d u u u u u u u u u
| | | | | | | | | |
sdw8 sdw2 sdw3 sdw4 sdw5 sdw6 sdw7 sdw3 sdw4 sdw5
| | | | | | | | | |
/data2/primary/gpseg47 /mirror2/mirror/gpseg0 /mirror2/mirror/gpseg1 /mirror2/mirror/gpseg2 /mirror1/mirror/gpseg3 /mirror1/mirror/gpseg4 /mirror1/mirror/gpseg5 /mirror2/mirror/gpseg6 /mirror2/mirror/gpseg7 /mirror2/mirror/gpseg8
| u | u
| sdw5 | sdw6
| /mirror1/mirror/gpseg46 | /mirror1/mirror/gpseg47
19. Exit out of the psql session and back to the command prompt.
\q
138
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart' 20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Master host=mdw 20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart 20110705:09:01:36:gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/data/master/gpseg-1 [gpadmin@mdw ~]$
21. Restart the database normally. Watch to make sure that the primary segments were skipped and only the mirror segments are started.
gpstart
20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:--------------------------20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Master instance parameters 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:--------------------------20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Database = template1 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Master Port = 5432 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Master directory = /data/master/gpseg-1 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Timeout = 600 seconds 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Master standby = Off 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:----------------------------------20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:-Segment instances that will be started 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:----------------------------------20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- Host Datadir Port Role 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw2 /mirror2/mirror/gpseg0 50000 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw3 /mirror2/mirror/gpseg1 50001 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw4 /mirror2/mirror/gpseg2 50002 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw5 /mirror1/mirror/gpseg3 50003 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw6 /mirror1/mirror/gpseg4 50004 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw7 /mirror1/mirror/gpseg5 50005 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw3 /mirror2/mirror/gpseg6 50000 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw4 /mirror2/mirror/gpseg7 50001 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:- sdw5 /mirror2/mirror/gpseg8 50002 Primary -- Data cut due to output length 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:50004 Primary 20110705:09:01:44:gpstart:mdw:gpadmin-[INFO]:50005 Primary sdw5 sdw6 /mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
Continue with Greenplum instance startup Yy|Nn (default=N): > y 20110705:09:01:47:gpstart:mdw:gpadmin-[INFO]:-No standby master configured. skipping... 20110705:09:01:47:gpstart:mdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... .. 20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:-Process results... 20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:----------------------------------20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:- Successful segment starts = 48
140
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:- Failed segment starts = 0 20110705:09:01:49:gpstart:mdw:gpadmin-[WARNING]:-Skipped segment starts (segments are marked down in configuration) = 48 <<<<<<<< 20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:----------------------------------20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:-Successfully started 48 of 48 segment instances, skipped 48 other segments 20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:----------------------------------20110705:09:01:49:gpstart:mdw:gpadmin-[WARNING]:-******************************* ********************************************* 20110705:09:01:49:gpstart:mdw:gpadmin-[WARNING]:-There are 48 segment(s) marked down in the database 20110705:09:01:49:gpstart:mdw:gpadmin-[WARNING]:-To recover from this current state, review usage of the gprecoverseg 20110705:09:01:49:gpstart:mdw:gpadmin-[WARNING]:-management utility which will recover failed segment instance databases. 20110705:09:01:49:gpstart:mdw:gpadmin-[WARNING]:-******************************* ********************************************* 20110705:09:01:49:gpstart:mdw:gpadmin-[INFO]:-Starting Master instance mdw directory /data/master/gpseg-1 20110705:09:01:50:gpstart:mdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active 20110705:09:02:01:gpstart:mdw:gpadmin-[WARNING]:-Database started but warnings generated <<<<< 20110705:09:02:01:gpstart:mdw:gpadmin-[INFO]:-Check status of database with gpstate utility [gpadmin@mdw ~]$
22. Once the database is up and running on the mirror segments that were just restored from the SNAP session, run a full gprecoverseg to rebuild the primary segments with the data from the mirror segments and synchronize the system. A full recovery is necessary since the mirror data was changed by a different DCA and the remote DCA has no knowledge of what changes were made.
142
20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance directory = /data1/primary/gpseg1 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance port = 40001 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance replication port = 41001 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance host = sdw3 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance address = sdw3-2 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance directory = /mirror2/mirror/gpseg1 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance port = 50001 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance replication port = 51001 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Target = in-place . Data cut due to output length 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-----------------------------20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-Recovery 48 of 48 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-----------------------------20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Synchronization mode = Full 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance host = sdw8 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance address = sdw8-2 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance directory = /data2/primary/gpseg47 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance port = 40005 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance replication port = 41005 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance host = sdw6 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance address = sdw6-1 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance directory = /mirror1/mirror/gpseg47 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance port = 50005 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:Recovery Source instance replication port = 51005 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Target = in-place 20110705:09:03:16:gprecoverseg:mdw:gpadmin-[INFO]:-----------------------------Continue with segment recovery procedure Yy|Nn (default=N): > y
Recovering a DCA using a SNAP session and SRDF
143
20110705:09:03:22:gprecoverseg:mdw:gpadmin-[INFO]:-48 segment(s) to recover 20110705:09:03:22:gprecoverseg:mdw:gpadmin-[INFO]:-Ensuring 48 failed segment(s) are stopped 20110705:09:03:23:gprecoverseg:mdw:gpadmin-[INFO]:-Cleaning files from 48 segment(s) ......... 20110705:09:03:32:gprecoverseg:mdw:gpadmin-[INFO]:-Building template directory 20110705:09:03:33:gprecoverseg:mdw:gpadmin-[INFO]:-Validating remote directories . 20110705:09:03:34:gprecoverseg:mdw:gpadmin-[INFO]:-Copying template directory file . 20110705:09:03:35:gprecoverseg:mdw:gpadmin-[INFO]:-Configuring new segments . 20110705:09:03:36:gprecoverseg:mdw:gpadmin-[INFO]:-Cleaning files . 20110705:09:03:37:gprecoverseg:mdw:gpadmin-[INFO]:-Updating configuration with new mirrors 20110705:09:03:39:gprecoverseg:mdw:gpadmin-[INFO]:-Updating mirrors . 20110705:09:03:40:gprecoverseg:mdw:gpadmin-[INFO]:-Starting mirrors 20110705:09:03:40:gprecoverseg:mdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... .. 20110705:09:03:42:gprecoverseg:mdw:gpadmin-[INFO]:-Process results... 20110705:09:03:42:gprecoverseg:mdw:gpadmin-[INFO]:-Updating configuration to mark mirrors up 20110705:09:03:42:gprecoverseg:mdw:gpadmin-[INFO]:-Updating primaries 20110705:09:03:42:gprecoverseg:mdw:gpadmin-[INFO]:-Commencing parallel primary conversion of 48 segments, please wait... .. 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-Process results... 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-Done updating primaries 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-***************************** ************************************* 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-Updating segments for resynchronization is completed. 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-For segments updated successfully, resynchronization will continue in the background. 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-Use gpstate -s to check the resynchronization progress. 20110705:09:03:44:gprecoverseg:mdw:gpadmin-[INFO]:-***************************** ************************************* [gpadmin@mdw ~]$
23. Periodically, run the gpstate command as reported in the gprecoverseg output to obtain the status of the recovery. When all segments report synchronized, the recovery is complete.
144
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
24. Note: the output from gpstate -s is very long - fourteen lines for each segment. The important thing to monitor is the "Mirror status", which will report "Resynchronizing" during the recovery process, and "Synchronized" when it is complete.
gpstate -s
20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Current role = Primary 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Preferred role = Mirror 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Mirror status = Resynchronizing 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Change Tracking Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Change tracking data size = 199 MB 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Resynchronization Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Resynchronization mode = Full 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Data synchronized = 2.50 GB 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Estimated total data to synchronize = 59.3 GB 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Estimated resync progress with mirror = 4.21% 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Estimated resync end time = 2011-07-05 10:06:38 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Status 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:PID = 8937 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Configuration reports status as = Up 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Database status = Up 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Segment Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Hostname = sdw1 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Address = sdw1-1 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Datadir = /data1/primary/gpseg0 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Port = 40000 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Mirroring Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Current role = Mirror 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Preferred role = Primary 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Mirror status = Resynchronizing 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Status 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:PID = 5602 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Configuration reports status as = Up 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Segment status = Up 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:----------------------------------Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
146
. Data cut due to output length 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Segment Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Hostname = sdw8 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Address = sdw8-2 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Datadir = /data2/primary/gpseg47 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Port = 40005 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Mirroring Info 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Current role = Mirror 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Preferred role = Primary 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Mirror status = Resynchronizing 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Status 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:PID = 28224 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Configuration reports status as = Up 20110705:09:06:27:gpstate:mdw:gpadmin-[INFO]:Segment status = Up [gpadmin@mdw ~]$
20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Greenplum current version = PostgreSQL 8.2.15 (Greenplum Database 4.1.1.1 build 1) on x86_64-unknown-linux-gnu, compiled by GCC gcc (GCC) 4.4.2 compiled on May 12 2011 18:07:08 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Postgres version = 8.2.15 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Master standby = No master standby configured 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:-Segment Instance Status Report 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Segment Info 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Hostname = sdw2 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Address = sdw2-2 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Datadir = /mirror2/mirror/gpseg0 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Port = 50000 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Mirroring Info 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Current role = Primary 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Preferred role = Mirror 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Mirror status = Synchronized 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Status 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:PID = 8937 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Configuration reports status as = Up 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Database status = Up 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Segment Info 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Hostname = sdw1 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Address = sdw1-1 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Datadir = /data1/primary/gpseg0 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Port = 40000 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Mirroring Info 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Current role = Mirror 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Preferred role = Primary 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Mirror status = Synchronized 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Status
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
148
20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:PID = 5602 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Configuration reports status as = Up 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Segment status = Up 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:----------------------------------. Data cut due to output length 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:----------------------------------20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Segment Info 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Hostname = sdw8 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Address = sdw8-2 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Datadir = /data2/primary/gpseg47 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Port = 40005 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Mirroring Info 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Current role = Mirror 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Preferred role = Primary 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Mirror status = Synchronized 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Status 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:PID = 28224 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Configuration reports status as = Up 20110705:10:41:37:gpstate:mdw:gpadmin-[INFO]:Segment status = Up [gpadmin@mdw ~]$
25. Once the database is fully synchronized and all segments are up-to-date, stop and restart the database to switch the primary segments back to the primary role. This will happen automatically when the database is restarted. There is no need to go back into maintenance mode and update the gp_segment_configuration table space manually.
gpstop gpstart
150
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
J
Starting EMC Greenplum Database from Mirror Segments
151
1. Issue SRDF commands to split the SRDF link from R1 (symrdf failover), and mount SRDF R2 volumes to the remote DCA. 2. Start the Greenplum database in maintenance mode, which starts only the Master Server and nothing else.
gpstart -m
152
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
4. If not previously done, alter the users search path so that the user can find and access the gp_toolkit schema where the gp_segment_configuration table space is maintained.
ALTER role gpadmin set search_path=gp_toolkit,information_schema,pg_aoseg,pg_bitmapindex,pg_catalog,pg_ toast,public;
5. To see the current configuration of the GP database, run the following SQL command. This SQL select command will show which database is on which segment server as well as its role, status and location on the segment server.
select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid;
Note: A role of 'p' indicates the database has the 'primary' role and a role of 'm' indicates the database has the 'mirror' role. A mode of 's' indicates the database is 'Synchronized', a mode of 'c' indicates the database is in 'change tracking' mode because its mirror or primary is down and a mode of 'r' indicates the database is in 'recovery' mode. A status of 'u' indicates a segment database is 'up' and a status of 'd' indicates the database is 'down'.
6. Set the primary databases to the status of down and a role of mirror so that when the GP database is brought online it will not attempt to start the primary database segments, and if a gprecoverseg is issued it will use the mirror segments as the source for the recovery.
update gp_segment_configuration sc set role='m',status='d' from pg_filespace_entry fse where sc.dbid=fse.fsedbid and fse.fselocation like'/<unique dir path value>/%' and sc.content >= 0;
Sample output from running select statement after this update statement:
srdftesting=# select sc.dbid,sc.content,sc.role,sc.preferred_role,sc.mode,sc.status,sc.hostname,fse.f selocation from gp_segment_configuration sc,pg_filespace_entry fse where sc.dbid=fse.fsedbid order by sc.dbid; dbid | content | role | preferred_role | mode | status | hostname | fselocation ------+---------+------+----------------+------+--------+----------+-----------1 | -1 | p | p | s | u | mdw | /data/master/gpseg-1 2 | 0 | m | p | s | d | sdw1 | /data1/primary/gpseg0 3 | 1 | m | p | s | d | sdw1 | /data1/primary/gpseg1 4 | 2 | m | p | s | d | sdw1 | /data1/primary/gpseg2 5 | 3 | m | p | s | d | sdw1 | /data2/primary/gpseg3 6 | 4 | m | p | s | d | sdw1 | /data2/primary/gpseg4 7 | 5 | m | p | s | d | sdw1 | /data2/primary/gpseg5 8 | 6 | m | p | s | d | sdw2 | /data1/primary/gpseg6 9 | 7 | m | p | s | d | sdw2 | /data1/primary/gpseg7 10 | 8 | m | p | s | d | sdw2 | /data1/primary/gpseg8 Data cut due to length of output 88 | 38 | m | m | s | u | sdw2 | /mirror2/mirror/gpseg38
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
154
89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | (97 rows)
39 40 41 42 43 44 45 46 47
| | | | | | | | |
m m m m m m m m m
| | | | | | | | |
m m m m m m m m m
| | | | | | | | |
s s s s s s s s s
| | | | | | | | |
u u u u u u u u u
| | | | | | | | |
| | | | | | | | |
srdftesting=#
7. Set the mirror database segments to the role of primary and mode of change tracking so that when the GP database is brought online it will start the mirror segments as the primary segments and keep track of any changes that are made to the segments so a gprecoverseg can be issued.
update gp_segment_configuration sc set role='p',mode='c' from pg_filespace_entry fse where sc.dbid=fse.fsedbid and fse.fselocation like'/<unique dir path value>/%' and sc.content >= 0;
| | | |
p p p p
| | | |
m m m m
| | | |
c c c c
| | | |
u u u u
| | | |
| | | |
10. )Start the database as normal. Notice that the primary segments are skipped since they were marked down in the gp_segment_configuration table space and only the mirror segments were started.
gpstart
156
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:Port Role 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50000 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50001 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50002 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50003 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50004 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50005 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50000 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50001 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50002 Primary Data cut due to length of output 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50004 Primary 20110616:08:40:53:gpstart:smdw:gpadmin-[INFO]:50005 Primary
Host sdw2 sdw3 sdw4 sdw5 sdw6 sdw7 sdw3 sdw4 sdw5
Datadir /mirror2/mirror/gpseg0 /mirror2/mirror/gpseg1 /mirror2/mirror/gpseg2 /mirror1/mirror/gpseg3 /mirror1/mirror/gpseg4 /mirror1/mirror/gpseg5 /mirror2/mirror/gpseg6 /mirror2/mirror/gpseg7 /mirror2/mirror/gpseg8
sdw5 sdw6
/mirror1/mirror/gpseg46 /mirror1/mirror/gpseg47
Continue with Greenplum instance startup Yy|Nn (default=N): > y 20110616:08:40:55:gpstart:smdw:gpadmin-[INFO]:-No standby master configured. skipping... 20110616:08:40:55:gpstart:smdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait... ... 20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:-Process results... 20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:Successful segment starts = 48 20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:Failed segment starts = 0 20110616:08:40:58:gpstart:smdw:gpadmin-[WARNING]:-Skipped segment starts (segments are marked down in configuration) = 48 <<<<<<<< 20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:-Successfully started 48 of 48 segment instances, skipped 48 other segments 20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:---------------------------------20110616:08:40:58:gpstart:smdw:gpadmin-[WARNING]:-****************************** 20110616:08:40:58:gpstart:smdw:gpadmin-[WARNING]:-There are 48 segment(s) marked down in the database 20110616:08:40:58:gpstart:smdw:gpadmin-[WARNING]:-To recover from this current state, review usage of the gprecoverseg
158
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery
20110616:08:40:58:gpstart:smdw:gpadmin-[WARNING]:-management utility which will recover failed segment instance databases. 20110616:08:40:58:gpstart:smdw:gpadmin-[WARNING]:-****************************** 20110616:08:40:58:gpstart:smdw:gpadmin-[INFO]:-Starting Master instance mdw directory /data/master/gpseg-1 20110616:08:40:59:gpstart:smdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active 20110616:08:41:11:gpstart:smdw:gpadmin-[WARNING]:-Database started but warnings generated <<<<< 20110616:08:41:11:gpstart:smdw:gpadmin-[INFO]:-Check status of database with gpstate utility [gpadmin@smdw ~]$
11. To recover the primary segments that were marked down, issue gprecoverseg -F to perform a full recovery from the online mirror segments (SAN) to the offline primary segments (internal to the remote DCA). 12. Once the database is fully synchronized and all segments are up-to-date, stop and restart the database to switch the primary segments back to the primary role. This will happen automatically when the database is restarted. There is no need to go back into maintenance mode and update the gp_segment_configuration table space manually.
gpstop gpstart
160
Configuring EMC Greenplum DCA Using SAN Mirror and EMC Symmetrix VMAX for Disaster Recovery