Upgrade GPFS 3.4 to Spectrum Scale 4.1.1

It's funny because it's true….

That´s how I want to start with today's story… And today will I be more technical then normally.

Let's start from beginning before I complain too much and showing how easy it is.

I have now heard twice where end-users of GPFS heard it isn't possible to upgrade from GPFS 3.4 filesystem to Spectrum Scale 4.x without to shut down the entire cluster.

This is not true and I will now show it for you how easy it is and how it's works. 

I want with today's topic explain how easy it is to upgrade a GPFS 3.4 filesystem to Spectrum Scale 4.1.1 without to shut down the cluster.

This how my test system looks like and see the time stamp when I start with my upgrade.

[root@gpfs1 gpfs]# date Tue Jul 28 14:52:21 CEST 2015 [root@gpfs1 src]# mmlscluster GPFS cluster information ========================   GPFS cluster name:         gpfstest.cristie.se   GPFS cluster id:           13882454725933626110   GPFS UID domain:           cristie.se   Remote shell command:      /usr/bin/ssh   Remote file copy command:  /usr/bin/scp GPFS cluster configuration servers: -----------------------------------   Primary server:    gpfs1.cristie.se   Secondary server:  gpfs2.cristie.se  Node  Daemon node name            IP address       Admin node name             Designation -----------------------------------------------------------------------------------------------------------------------    1             gpfs1.cristie.se                10.0.0.1          gpfs1.cristie.se                  quorum-manager    2             gpfs2.cristie.se                10.0.0.2          gpfs2.cristie.se                  quorum-manager    3             gpfs3.cristie.se                10.0.0.2          gpfs3.cristie.se                  quorum-manager [root@gpfs1 gpfs]# mmlsconfig Configuration data for cluster gpfstest.cristie.se: --------------------------------------------------- myNodeConfigNumber 1 clusterName gpfstest.cristie.se clusterId 13882454725933626110 autoload no uidDomain cristie.se minReleaseLevel 3.4.0.7 dmapiFileHandleSize 32 adminMode central File systems in cluster gpfstest.cristie.se: -------------------------------------------- /dev/gpfs [root@gpfs1 src]# rpm -qa |grep gpfs gpfs.base-3.4.0-16 gpfs.docs-3.4.0-16 gpfs.msg.en_US-3.4.0-16 gpfs.gpl-3.4.0-16 [root@gpfs1 src]# uname -a Linux gpfs1.cristie.se 2.6.18-406.el5 #1 SMP Tue Jun 2 17:25:57 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

As you can see do I run GPFS 3.4 on 3 Nodes and a CentOS 5.2 Distribution.

Let's start with the major issues here. What kind of problems are we going to face here.

First of all you can't install Spectrum Scale (GPFS) 4.x on a CentOS 5.2 because of GLIBC is to old.

I need to upgrade CentOS 5 to a never Linux Distribution that works with Spectrum Scale 4.1.1

"I can't upgrade" CentOS 5.2 to CentOS 7 officially. I maybe can, but I guarantee it will take much more time than just reinstall the server with the new OS.

If you are running an old Linux Dist such CentOS 5, your environment are probably really old, then you anyway want to do a hardware refresh on new hardware.

But let's say I want to reuse the hardware I already have. (I use VMware in this PoC)

What I have done is installing a temporary server that I install CentOS 7 and installing Spectrum Scale 4.1.1 binaries.

Add a temporary GPFS Node to the cluster to keep it full redundant and add it as Quorum-Manager to the cluster.

[root@gpfs1 gpfs]# mmgetstate -a -L  Node number  Node name       Quorum  Nodes up  Total nodes  GPFS state  Remarks --------------------------------------------------------------------------------------------------------------------        1                        gpfs1                 3                 4               4                   active         quorum node        2                        gpfs2                 3                 4               4                   active         quorum node        3                        gpfs3                 3                 4               4                   active         quorum node        4                        gpfstemp          3                4                4                   active         quorum node

Now when I have more than 3 Quorum Nodes can I then delete GPFS3 from the cluster and reinstall it with CentOS 7.

Of course I need to install all the dependencies on the machine and delete the old ssh key that the other machines had of the old GPFS3 server. But this is fairly easy if you know Linux basic knowledge.

When I finally have installed GPFS 4.1.1 on the GPFS3 server I can then add it to the cluster again and make GPFS3 as the new Secondary server. I can also add gpfstemp as a Primary Server to the cluster.

This is the steps I did for GPFS3.

• Remove the node from the GPFS Cluster (mmdelnode)
• Reinstall CentOS 7 minimum install
• Installed all dependencies Spectrum Scale 4.1.1 have.
• Install and compile Spectrum Scale
• Join the cluster again (mmaddnode)
• Change the to a Quorum-Manager node (mmchnode)
• Change Primary and Secondary Configuration Server (mmchcluster)

If I do that can I then shutdown both GPFS2 and GPFS1 to start reinstall them. But if you want to risk the time period where you have no redundancy then it's fine. But otherwise run one-by-one.

[root@gpfs1 src]# mmlscluster GPFS cluster information ========================   GPFS cluster name:         gpfstest.cristie.se   GPFS cluster id:           13882454725933626110   GPFS UID domain:           cristie.se   Remote shell command:      /usr/bin/ssh   Remote file copy command:  /usr/bin/scp GPFS cluster configuration servers: -----------------------------------   Primary server:    gpfstemp.cristie.se   Secondary server:  gpfs3.cristie.se  Node  Daemon node name            IP address       Admin node name             Designation -----------------------------------------------------------------------------------------------------------------------    1             gpfs1.cristie.se                10.0.0.1          gpfs1.cristie.se                  quorum-manager    2             gpfs2.cristie.se                10.0.0.2          gpfs2.cristie.se                  quorum-manager    4             gpfstemp.cristie.se         10.0.0.4          gpfstemp.cristie.se          quorum-manager    5             gpfs3.cristie.se                10.0.0.3          gpfs3.cristie.se                  quorum-manager

 

When I'm finally have upgrade all nodes to CentOS 7 it should look something like this.

[root@gpfs1 src]# mmlscluster GPFS cluster information ========================   GPFS cluster name:         gpfstest.cristie.se   GPFS cluster id:           13882454725933626110   GPFS UID domain:           cristie.se   Remote shell command:      /usr/bin/ssh   Remote file copy command:  /usr/bin/scp GPFS cluster configuration servers: -----------------------------------   Primary server:    gpfstemp.cristie.se   Secondary server:  gpfs3.cristie.se  Node  Daemon node name            IP address       Admin node name             Designation -----------------------------------------------------------------------------------------------------------------------    4             gpfstemp.cristie.se         10.0.0.4          gpfstemp.cristie.se          quorum-manager    5             gpfs3.cristie.se                10.0.0.3          gpfs3.cristie.se                  quorum-manager    6             gpfs2.cristie.se                10.0.0.2          gpfs2.cristie.se                  quorum-manager    7             gpfs1.cristie.se                10.0.0.1          gpfs1.cristie.se                  quorum-manager

Let's now change back GPFS1 and GPFS2 to my Primary and Secondary Configuration servers.

Let's rock and roll and upgrade the environment now to latest version of GPFS/Spectrum Scale and change to a CCR Enabled cluster instead of Server based.

[root@gpfstemp gpfs]# mmchconfig release=LATEST Verifying that all nodes in the cluster are up-to-date ... mmchconfig: Command successfully completed mmchconfig: Propagating the cluster configuration data to all   affected nodes.  This is an asynchronous process. [root@gpfstemp gpfs]# mmchcluster -p LATEST mmchcluster: GPFS cluster configuration servers: mmchcluster:   Primary server:    gpfs1.cristie.se mmchcluster:   Secondary server:  gpfs2.cristie.se mmchcluster: Propagating the new server information to the rest of the nodes. mmchcluster: Command successfully completed [root@gpfstemp gpfs]# mmlsconfig Configuration data for cluster gpfstest.cristie.se: --------------------------------------------------- clusterName gpfstest.cristie.se clusterId 13882454725933626110 autoload no uidDomain cristie.se dmapiFileHandleSize 32 afmHashVersion 1 pagepool 64M maxFilesToCache 1000 maxMBpS 150 maxStatCache 4000 minReleaseLevel 4.1.1.0 adminMode central File systems in cluster gpfstest.cristie.se: -------------------------------------------- /dev/gpfs [root@gpfstemp gpfs]# mmchfs gpfs -V full You have requested that the file system be upgraded to version 14.23 (4.1.1.0). This will enable new functionality but will prevent you from using the file system with earlier releases of GPFS. Do you want to continue? yes Successfully upgraded file system format version to 14.23 (4.1.1.0). [root@gpfstemp gpfs]# mmlsfs gpfs flag                value                    description ------------------- ------------------------ -----------------------------------  -f                 2048                     Minimum fragment size in bytes  -i                 512                      Inode size in bytes …. ….  -V                 14.23 (4.1.1.0)          Current file system version                     12.10 (3.4.0.7)          Original file system version

Wow it works...

I'm impressed are you? Not really!!! ;) Not me either. This is way GPFS/Spectrum Scale is an Enterprise filesystem that is working.

[root@gpfstemp gpfs]# date Tue Jul 28 16:25:26 CEST 2015

Of course I need now to clean up and remove the gpfstemp server but that it's not too much work. (mmdelnode then shutdown -h now)

But the family is now calling me to get back home and eat dinner with my kid.

Ciao and I hope this helps. If you need more help. Please feel free to contact me.

Your Storage friend

Christian Svensson