Cluster File Systems

SECTION 10

Cluster File Systems

With a cluster file system it is possible to share a file system between systems. In the cluster jargon these systems are called nodes. Cluster file systems are also known as shared disk file systems. The benefit of these file systems is that data is still available to the remaining nodes if one node fails. Cluster file systems are routinely used in high availability environments. Different cluster file systems are available for Linux, the best known are:

OCFS (Oracle Cluster File System) GPFS (General Parallel File System) GFS (Global File System) LUSTRE

Objectives
1. 2.

Global File System (GFS) Oracle Cluster File System 2 (OCFS2)

Version 1

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

10-1

Migrating from RedHat to SUSE Linux Enterprise Server 10

Objective 1

Global File System (GFS)

Under RHEL, primarily GFS is used as cluster file system. GFS is a cluster file system that was originally developed at the University of Minnesota, later distributed by Sistina Software. Red Hat acquired Sistina Software at the end of 2003 and released their cluster file system as Open Source Software under the GPL in 2004. The Red Hat Cluster Suite for RHEL4 utilizes GFS 6.1 for shared storage for its high-throughput and high-availability features, but it can be used and is supported for the Oracle Real Application Cluster (RAC) as well. All releases up to the current iteration 6.1 are on-disk compatible and run under the GFS label. The successor GFS2, currently in development, will not be on-disk compatible with GFS(1), but will eliminate some of its current caveats. GFS2 has recently been included into the development version of the Linux kernel 2.6.19; GFS1 is developed out-of-tree. One can use GFS as a simple local file system without any cluster locking mechanism, but usually it is run in cluster mode, where it has to be supplemented by a SAN environment. GFS is not available on SUSE Linux Enterprise Server 10.

10-2

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

Version 1

Cluster File Systems

Objective 2

Oracle Cluster File System 2 (OCFS2)

This objective covers

OCFS2 Background OCFS2 Configuration OCFS to OCFS2 Migration GFS - OCFS2 Comparison Table

OCFS2 Background
OCFS2 is a cluster file system that was released by Oracle as Open Source software under the GPL in 2005. Contrary to its predecessor OCFS, released in 2002, it is now a general-purpose cluster file system that is suitable as a storage base for a more widespread type of scenarios than deploying Oracle Real Application Cluster (RAC) only. In SUSE Linux Enterprise Server 10 it can, for instance, be used to give several hosts access to XenVM loopback disk images. Thanks to OCFS2 virtual machines can be relocated from host to host. Novell has integrated OCFS2 version 1.0.8 into SLES 9 SP2, version 1.2.1 into SP3 and SLES 10. As OCFS2 has been included into the mainline Linux kernel starting with version 2.6.16, probably more Linux distributions will integrate OCFS2 in the future. All OCFS2 versionsstarting with version 1.0are on-disk compatible, but their corresponding tools are not. So take care to upgrade them together, and access the cluster filesystem only with management tools that are at the same level as the kernel driver. On SLES10 it is possible to set up an OCFS2 from the command line or with the shipped OCFS console which provides a GUI.

Version 1

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

10-3

Migrating from RedHat to SUSE Linux Enterprise Server 10

OCFS2 has the following features:

POSIX compliant Support for multiple CPU architectures as it is endian-safe. Thread-safe, asynchronous I/O, metadata and data caching provides high performance. Lockless node-local allocation area enables fast creation of new files. Extents-based; allocates metadata groups automatically, no fixed inode limit. Direct I/O to bypass local OS cache. Fully integrated with the Heartbeat 2 cluster solution in SUSE Linux Enterprise Server 10. OCFS2 is certified for Oracle 9iR2, 10g, 10gR2 RAC on RHEL4, SLES 9 SP3 and SLES 10.

OCFS2 Configuration
The following sample configuration assumes that the OCFS2 resides on the device sda. To be able to share this device between nodes, this device could be made available using iSCSI as covered in Section iSCSI on page 9-1.

To avoid having the storage device as a possible single point of failure, which would thwart the purpose of a HA cluster, the storage device should include HA features as well. DRBD (Distributed Replicated Block Device) and Heartbeat could be used for this purpose; configuration of DRBD and Heartbeat is, however, beyond the scope of this course.

10-4

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

Version 1

Cluster File Systems

Most of the following steps to configure OCFS2 need to be taken on every node:
1.

Create SSH keys for graphical OCFS2 frontend ocfs2console. To facilitate the configuration on the different nodes, one usually uses ssh keys. Keys are generated by the command ssh-keygen. The option -N creates a key without a passphrase. ssh-copy-id is a script that adds the public key to the .ssh/authorized_keys file on the destination computer.

da10:~ # ssh-keygen -t rsa -N ""; Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): <ENTER> Created directory '/root/.ssh'. Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: cb:1e:a9:7b:08:a7:82:d5:24:fc:90:49:cc:84:d6:a7 root@da10 da10:~ # ssh-copy-id -i ~/.ssh/id_rsa.pub root@da11; da10:~ # ssh-copy-id -i ~/.ssh/id_rsa.pub root@da12; ...

2.

Install the OCFS2 packages. Use the YaST Software Management module or the command yast -i ocfs2-tools ocfs2console.

Version 1

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

10-5

Migrating from RedHat to SUSE Linux Enterprise Server 10

3.

Create /etc/ocfs2/cluster.conf The following is an example with two nodes, just adapt "node_count" if you add more.

cluster: name = mycluster node_count = 2 node: name = da10 cluster = mycluster number = 0 ip_address = 10.0.0.10 ip_port = 7777 node: name = da11 cluster = mycluster number = 1 ip_address = 10.0.0.11 ip_port = 7777

Instead of the hostname, name could also be the fully qualified domain name; name resolution has to work.
4.

Create an OCFS2 on the shared block device. This is only done on one node. Take care that you use the same label as in the cluster configuration file (this is mycluster in the example above). The command to create an OCFS2 file system is mkfs.ocfs2. Entering the command without options gives a brief syntax overview:

da3:~ # mkfs.ocfs2 Usage: mkfs.ocfs2 [-b block-size] [-C cluster-size] [-N number-of-node-slots] [-T filesystem-type] [-L volume-label] [-J journal-options] [-HFqvV] device [blocks-count]

The manual page explains the available options.

10-6

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

Version 1

Cluster File Systems

The following command creates an OCFS2 file system on /dev/sda:

da10:~ # mkfs.ocfs2 -b 4k -C 128k -N 4 -L mycluster /dev/sda mkfs.ocfs2 1.2.1 Filesystem label=mycluster Block size=4096 (bits=12) Cluster size=131072 (bits=17) Volume size=8595308544 (65577 clusters) (2098464 blocks) 3 cluster groups (tail covers 1065 clusters, rest cover 32256 clusters) Journal size=67108864 Initial number of node slots: 4 Creating bitmaps: done Initializing superblock: done Writing system files: done Writing superblock: done Formatting Journals: done Writing lost+found: done mkfs.ocfs2 successful

5.

Add an entry in /etc/fstab. The following is done on all nodes and assumes that the device (/dev/sda in this example) is the same on all nodes

da10:~ # echo '/dev/sda

/mnt

ocfs2

defaults

0 0' >> /etc/fstab

Version 1

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

10-7

Migrating from RedHat to SUSE Linux Enterprise Server 10

6.

Configure the OCFS2 cluster services. Make sure that Cluster to start on boot is the same as the entry in /etc/ocfs2/cluster.conf

da10:~ # /etc/init.d/o2cb configure Configuring the O2CB driver. This will configure the on-boot properties of the O2CB driver. ... Load O2CB driver on boot (y/n) [y]: <ENTER> Cluster to start on boot (Enter "none" to clear) [mycluster]: <ENTER> Use user-space driven heartbeat? (y/n) [n]: <ENTER> Writing O2CB configuration: OK

7.

Start the storage cluster services automatically. The following commands create the links in /etc/init.d/rcx.d/ needed to start the services at boot time.

da10:~ # chkconfig -a o2cb da10:~ # chkconfig -a ocfs2

10-8

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

Version 1

Cluster File Systems

8.

Start the OCFS2 cluster services manually and mount the OCFS2. The following commands start the services needed for ocfs2:

da10:~ # /etc/init.d/o2cb start; Loading module "configfs": OK Mounting configfs filesystem at /sys/kernel/config: OK Loading module "ocfs2_nodemanager": OK Loading module "ocfs2_dlm": OK Loading module "ocfs2_dlmfs": OK Mounting ocfs2_dlmfs filesystem at /dlm: OK Starting cluster mycluster: OK da10:~ # /etc/init.d/ocfs2 start Starting Oracle Cluster File System (OCFS2)

done

The second command mounts the file system. Use the command mount to see the result.
9.

Check that the cluster file system works.

da10:~ # cat /sys/kernel/config/cluster/mycluster/heartbeat/dead_threshold 7

10. Stop the OCFS2 cluster services manually and unmount the

OCFS2. The following commands stop the OCFS2 cluster services:

da10:~ # /etc/init.d/ocfs2 stop; Stopping Oracle Cluster File System (OCFS2) da10:~ # /etc/init.d/o2cb stop; Cleaning heartbeat on mycluster: OK Stopping cluster mycluster: OK Unloading module "ocfs2": OK Unmounting ocfs2_dlmfs filesystem: OK Unloading module "ocfs2_dlmfs": OK Unmounting configfs filesystem: OK Unloading module "configfs": OK

done

Version 1

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

10-9

Migrating from RedHat to SUSE Linux Enterprise Server 10

You should also familiarize yourself with the graphical OCFS2 frontend ocfs2console which allows to adminster different aspects of OCFS2. Figure 10-1

Figure 10-2

10-10

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

Version 1

Cluster File Systems

OCFS to OCFS2 Migration

The on-disk layouts of OCFS and OCFS2 are not compatible. As drivers for OCFS only exist for 2.4.x linux kernels and drivers for OCFS2 only for 2.6.x linux kernels, you cannot use them both on the same machine concurrently. There are two possibilities to migrate to the new OCFS2:

If the old cluster file system is still online you can replicate it to the new one via the network. You have to ensure that there are no changes to the old OCFS storage cluster after the last replication; during the migration you need double SAN capacity. Oracle provides the FSCat tools that can read OCFS disk images in userland. Using these tools on an OCFS2 cluster node, you can extract your OCFS backup into the OCFS2 mountpoint.

Version 1

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

10-11

Migrating from RedHat to SUSE Linux Enterprise Server 10

GFS - OCFS2 Comparison Table

Table 10-1
Release date Linux kernel inclusion CPU architecture GFSv1 6.1 2006-08 (v1.0 1996) out-of-tree x86, x64, ia64 OCFS2 1.2.3 2006-07 (v1.0 2005) 2.6.16 x86, x64, ia64, ppc, ppc64 255 255 unlimited 512 B - 4 KB 4 KB - 1 MB 4 TB / 16 TB / 4 PB 4 TB / 16 TB / 4 PB LVM2, EVMS OCFSv2 DLM O2CB (NM+HB+TCP) FC, iSCSI -/-/yes yes

Max. supported notes Max. file name length Max. path length Block size Cluster size Max. file size Max. fs size Volume manager Lock manager Cluster manager Shared block devices

256 255 unlimited 512 B - 4 KB 2 TB / 16 TB / 8 EB 2 TB / 16 TB / 8 EB LVM2 + CLVM GULM, kernel DLM CMAN GNBD, FC, iSCSI

Online/Offline resizing grow/User/Group quotas Multipath Direct I/O yes/yes yes yes

10-12

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

Version 1

Cluster File Systems

Exercise 10-1

Set up an OCFS2 The purpose of this exercise is to familiarize you with OCFS2. This exercise builds on the previous one, Set up an iSCSI Target and an iSCSI initiator on page 9-23. You will find this exercise in the workbook.
(End of Exercise)

Version 1

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

10-13

Migrating from RedHat to SUSE Linux Enterprise Server 10

Summary
Objective
1. Global File System (GFS)

Summary Under RHEL, primarily GFS (Global File System) is used as cluster file system. OCFS2 is the cluster file system used under SUSE Linux Enterprise Server 10. The configuration is contained in /etc/ocfs2/cluster.conf; the program to create the file system is mkfs.ocfs2. The clustering functionality is provided by the services o2cb and ocfs2.

2. Oracle Cluster File System 2

(OCFS2)

10-14

Copyright 2007 Novell, Inc. Copying or distributing all or part of this manual is protected by a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 license.

Version 1