POC Linux7 Multinode GFS2 Cluster on CentOS 7.3

Proof of concept Linux high availability GFS2 cluster

Proof of concept

Building a multi-node High Availability GFS2 cluster for CentOS Linux using iSCSI.

By: Kjell Erik Furnes Version 1.02 February 2018

Written by Kjell Erik Furnes [email protected]

Page 1 of 69

Trusted by over 1 million members

Try Scribd FREE for 30 days to access over 125 million titles without ads or interruptions! Start Free Trial Cancel Anytime.


















Contents About the writer: ..................................................................... ................................................................................................................................. ............................................................ 4 Prefix.............................................................................................................. .................................................................................................................................................... ...................................... 4 Disclaimer .......................................................................................................................................... ............................................................................................................................................ .. 5 Credits .............................................................. .................................................................................................................................... ................................................................................... ............. 5 Current environment.............................................................. ........................................................................................................................... ............................................................. 6 My home test environment............................................................................. environment........................................................................................................ ........................... 6 Initial installation ................................................................................ ................................................................................................................................. ................................................. 7 Common server configuration .............................................................. .................................................................................................... ...................................... 7 Initial warnings ................................................................................................................................ ................................................................................................................................ 9 Restrict the server, (part 1), ssh access ................................................................ ......................................................................................... ......................... 10 Repo server...................................................................... server................................................................................................................................ .......................................................... 10 Update the yum config ......................................................................................... .................................................................................................................. ......................... 10 Update the server............................................................... .......................................................................................................................... ........................................................... 11 Restrict the server, (part 2), Restrict host access ............................................................ .......................................................................... .............. 13 TCP_WRAPPER TCP_WRAPPER........................................................................................................................... ........................................................................................................................... 13 Linux jump host ...................................................................................................................... ......................................................................................................................... ... 13 Restrict the server (part 3), Restrict sudo ..................................................................................... ..................................................................................... 13 Preparing the server ........................................................................... .......................................................................................................................... ............................................... 14 Estabilsh old notation of network interfaces ............................................................................ ............................................................................ 16 Uninstall NetWorkManager NetWorkManager ...................................................................................................... 16 Alter the network interface card configuratig files ....................................................................... ....................................................................... 16 Modify grub ................................................................................................................................... ................................................................................................................................... 17 Disable IPv6 ............................................................................................................................... ............................................................................................................................... 17 Time synchronization .................................................................................................................... .................................................................................................................... 18 Setting up chrony ...................................................................................................................... ...................................................................................................................... 18 sudo chronyc sourcestats -v ...................................................................................................... ...................................................................................................... 20 Reboot ........................................................................................................................................... ........................................................................................................................................... 20 Configure firewalld ..................................................................................................................... ........................................................................................................................ ... 20 systemd ................................................................... ......................................................................................................................................... ...................................................................... 22 ISCSI ............................................................................................................................................... ............................................................................................................................................... 23 Install iscsi package ....................................................................................................................... ....................................................................................................................... 23


















Vmware housekeeping.............................................................. .............................................................................................................. ................................................ 31 Planning the t he cluster ................................................................. ........................................................................................................................... .......................................................... 31 Setting up the cluster ........................................................................................................................ ........................................................................................................................ 31 Number of nodes in a cluster ........................................................................... .................................................................................................... ......................... 32 Cluster related package installation ................................................................. .......................................................................................... ......................... 33 Set up the cluster ........................................................................................................................... .............................................................................................................................. ... 33 Encrypting the cluster sync traffic ......................................................... ............................................................................................. .................................... 33 Generating the authkey file: .................................................................. ...................................................................................................... .................................... 33 Verify the cluste configuration ......................................................................... .................................................................................................. ......................... 37 Setting up fencing............................................................... .......................................................................................................................... ........................................................... 39 Fencing agents ............................................................................ ........................................................................................................................... ............................................... 39 Watchdog ...................................................................................................... .......................................................................................................................................... .................................... 44 Install the watchdog service ............................................................................. ...................................................................................................... ......................... 44 Test the fencing mechanism...................................................................................................... mechanism...................................................................................................... 46 GFS2.................................................................................................................................... ............................................................................................................................................... ........... 52 Add more disk to you GFS cluster ..................................................................................................... ..................................................................................................... 56 Extend the VG .................................................................. ............................................................................................................................ .......................................................... 56 Create a new logical volume ..................................................................................................... 57 Create a new GFS2 filesystem .......................................................................... ................................................................................................... ......................... 57 Create a new GFS2 cluster resource............................... resource.......................................................................................... ........................................................... 58 Moifying the custer ........................................................................................................................... ........................................................................................................................... 58 Add Nodes to the cluster............................................................... ............................................................................................................... ................................................ 58 Adding the new node ................................................................................................................ ................................................................................................................ 59 Setting a cluster node in standby mode........................................................... .................................................................................... ......................... 64 Taking a cluster node out of standby mode ................................................................... .............................................................................. ........... 65 Banning a resource from a cluster node ................................................................................... ................................................................................... 66 Removing the ban .................................................................................................................. ..................................................................................................................... ... 66 Moving a cluster resource. ........................................................................................................ ........................................................................................................ 67 Remove nodes from the cluster ........................................................................... .................................................................................................... ......................... 68


















About the writer: In my younger days I held both Norwegian and US Pilot and Instructor licenses, heading for a career in the skies. Now, the closest I get is some cloud services. I am an Oracle database administrator, Linux administrator and Commvault Backup administrator currently employed at The University of Oslo in Norway. I have worked on most of the normal platforms (HP-UX, AIX, TRU64, Linux and Windows) as a system administrator and as an Oracle database administrator administrator since 1991 (Oracle database version 6), and SAP Basis administrator since 1998. I have worked in Norwegian governmental organizations and in the Norwegian private sector. When I am not clonking on a keyboard chances are that I might be out on some bike-ride.You can follow me on STRAVA. https://www.strava.com/athletes/2233700

Prefix When I have written documents before that I plan to publish on the Internet I like to write about something that has not been written much about before. This time I am writing about Linux cluster, and that is a topic where there exist a very large quantity of documents already on the Internet. However I am writing this document because clustering is something that is constantly evolving. Also many of the documents you can find online is related to two-node clusters, and there is some differences between a two-node cluster and a cluster that consist of more than two nodes. In my opinion, a two-node cluster is not production ready. I am going to show you how to build a four node cluster which I am going to expand to a six node cluster, and finally I am going to remove one node from the cluster. The cluster I am going to set up will be a cluster for GFS2. In this document, I have altered some output to make it more readable. Mostly this means that I have removed whitespace whitespace on lines to make the line fit on a single line where that has been possible.


















Disclaimer This document was written as a proof of concept to show the possibilities of a certain configuration. The writer or anyone associated with the writer cannot be made liable for any damage to your systems nor can they be made liable for any loss of data if you choose to follow the concept shown in this document. Trademarked names, logos, and images may appear in this document. Rather than to use a trademark symbol with every occurrence of a trademarked name, logo or image, I use the names, logos or images only in in an editorial fashion and to the benefit of the trademark owner, owner, with no intention or infringement of the trademark. The use in this document of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this document are believed to be true and accurate at the date of writing, neither the writer nor anyone associated with the writer can accept any legal responsibility for any errors or omissions that ma y be made. The writer or anyone associated with the writer makes no warranty, expressed or implied, with respect to the material contained within this document. This document is not meant to replace the official installation documentation or configuration guides provided by The CentOS Project. If you choose to follow this document you do so at your own risk. The comments and statements made in this document are solely those of the writer and does not reflect the views or opinion of the writers employer or anyone associated with the writer. The document use the domain name example.com as defined at

https://en.wikipedia.org/wiki/Examp https://en.wikipedia.o rg/wiki/Example.com le.com

Credits Linux is a registered trademark of Linux Torvalds RedHat is a registered trademark of Red Hat Inc https://www.redhat.com/en/about/trademark-guidelines-and-policies CentOS is a registered trademark of The CentOS Project https://www.centos.org/leagal/trademarks. VMware is a registered trademark of Vmware in the United States and other Countries https://www.vmware.com/help/trademarks.html


















Current environment Myhometestenvironment

NTP DNS amygdala

cerebellum

sinister

Net 10.47.253.0/24 10.47.253.0/24 170.20.16.0/24 170.20.16.0 /24 171.20.16.0/24 171.20.16.0 /24 172.20.16.0/24 172.20.16.0/24 173.20.16.0/24 173.20.16.0 /24

dextro

Function Public storage170 storage171 cluster private storage173

Speed 1gbit 1gbit 1gbit 1gbit 1gbit

MTU 1500 1500 1500 1500 9600

Active Not in use Active Active Not in use

In my configuration I have chosen to separate the different types of network traffic. If you are running your cluster on Fiber-Channel it is obvious that you would reduce the number of network interfaces. But in addition I have separated the traffic so that I have cluster intercommunication intercommunication running on a separate network. This is not necessary. You are able to run the cluster private traffic over the same network as your client traffic. In older days the cluster private interconnect was called a heartbeat, and there was even a process called heartbeat. The cluster private interconnect is now handled by the corosync process. The cluster interconnect is quite chatty, and it is very time critical for the cluster responses. If your system is I/O intensive to such an extent that you risk creating latency over a single network I advise that you separate the cluster private interconnect on a separate network.

Host Sinister Dexter Amygdala Cerebellum Cerebel lum Cortex Thalamus

IP 10.47.253.6 10.47.253.7 10.47.253.36 10.47.253.9 10.47.253.3 10.47.253.99 10.47.253.99

Function VMware ESXi 6.5 Free VMWare ESXi 6.5 Free jump host DNS, NTP Storage, NAS, ISCSI Local yum repository server

NIC’S

5 5 1 1 4 1




















Initial installation Into my basic environment I am going to place a Linux cluster Cluster name: cerebrum cerebrum

poc-gfs-cl01

poc-gfs-cl02 poc-gfs-cl03 poc-gfs-cl04 poc-gfs-cl05 NTP DNS

amygdala

cerebellum

sinister

dextro

All my servers that will be included in this document are vmware guest servers located on my ESXi host servers. Odd number guests are located on the sinister host, and even numbered guests are located l ocated on the dexter host. Both the public name and the private name are registered in my dns. Public host name poc-gfs-cl01 poc-gfs-cl02 poc-gfs-cl03 poc-gfs-cl04 poc-gfs-cl05 poc-gfs-cl06

Public IP 10.47.253.101 10.47.253.1 01 10.47.253.102 10.47.253.1 02 10.47.253.103 10.47.253.1 03 10.47.253.104 10.47.253.1 04 10.47.253.105 10.47.253.1 05 10.47.253.106 10.47.253.1 06

Private name privcl01 privcl02 privcl03 privcl04 privcl05 privcl06

Private IP 172.20.16.101 172.20.16.102 172.20.16.103 172.20.16.104 172.20.16.105 172.20.16.106


















During the installation and configuration I will keep to the default settings as much as possible. I have made some exceptions where it has been necessary to implement the configuration I want to demonstrate, and some alterations in order to restrict the access to the server.


















Initial warnings

I will repeat these warnings at the appropriate location within the document. I just list some of them here to give you a heads up before you start reading.

History lesson: I once tried to remove all core dump files from a running production system. I had a

word document with the command find / -name core –exec rm –rf {} \; characterset difference caused the – before the name parameter to be void. The resulting command turned out to be find / -exec rm –rf {} \; which is a good way to say find anything and delete it. I almost put a company company out of business that day. The system was saved because the find started in the /bin directory making the find command command unable to fork and delete the rest of the system. Lesson learned: Don’t cu t

Warning:

and paste from a word document.


















I am not a big fan of disabling security measures. So because I am going to reduce security by running selinux in Permissive mode I am going to increase security on my servers by other means.

Restrict the server, (part 1), ssh access

Reposerver

If you are planning to have an environment where you will be adding new servers into your cluster configuration it is wise to be able to control the configuration configuration and versions of packages to be installed on your systems in order to have consistency over time. I have done this by installing a local yum repository server. If you don’t have a local yum repository server you can skip the configuration changes I am outlining


















24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

#baseurl=http://mirror.centos.org/centos/$releasever/updates/$basearch/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 #additional packages that may be useful [extras] name=CentOS-$releasever - Extras mirrorlist=http://mirrorlist.centos.org/?release=$releasev mirrorlist=http://mirrorl ist.centos.org/?release=$releasever&arch=$basearch&repo=ex er&arch=$basearch&repo=extras&infra=$infra tras&infra=$infra #baseurl=http://mirror.centos.org/centos/$releasever/extras/$basearch/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 #additional packages that extend extend functionality of existing packages [centosplus] name=CentOS-$releasever - Plus mirrorlist=http://mirrorlist.centos.org/?release=$releasev mirrorlist=http://mirrorl ist.centos.org/?release=$releasever&arch=$basearch&repo=centosplu er&arch=$basearch&repo=centosplus&infra=$infra s&infra=$infra #baseurl=http://mirror.centos.org/centos/$releasever/centosplus/$basearch/ gpgcheck=1 enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

Change each occurance occurance of the mirrorlist url to the url of your local server baseurl=http:///centos/$releasever/updates/$basearch/

Update the server




















Since I am in control of my package repository I now know that the installed pacakges on my server is the same for all of my installations, and it is time to start the modifications to prepare the server to become a cluster server.




















Restrict the server, (part 2), Restrict host access

TCP_WRAPPER

As part of the update from the vanilla installation (from the DVD) our system now also has TCP_WRAPPERS TCP_WRAPPERS installed which will allow me to restrict which host are allowed to connect to my server. Linuxjumphost

In my configuration I have a jump host to reduce the attack surface of my servers. My jump host has the IP address 10.47.253.36, and by applying a TCP_WRAPPER TCP_WRAPPER config to my client servers I reduce the attack surface accordingly.

With these settings the only ssh connection in to my server is from my jump host, and I have disabled a direct root acces over ssh. Do a reboot now to start the the server on the new kernel (there was a new kernel in the update )

Restrict the server (part 3), Restrict sudo




















Preparing the server In RHEL7 and clones there was a change in how the interfaces on the server was named. Traditionally the network interface was named eth[0123]. With the release of version 7 there are five different ways (schemes) to naming the interface as long as biosdevname is enabled (default). The system process will do a cascading loop from 1 to 5. Sheme 1: BIOS or firmware indexing for onboard devices (eno) Scheme 2: BIOS or firmware indexing indexin g for PCI devices (ens) Scheme 3: Naming by physical location of the device (enp) Scheme 4: Naming by MAC address (enx) Scheme 5: If all else fail, use traditional naming (eth).




















To disable the new naming schemes of the interface you need to modify the parameters used during the boot sequence. NOTE: Prior to release 7.3 there was a compatibility issue between the system process and VMware

causing the interface naming to get an acpi index numbers during the boot sequence. (Actually you would get a non-initialized parameter result of the acpi index). If you were lucky your interface would get the same number after a boot, but you could also get a different interface name after a boot. This is fixed in release 7.3

Source: https://thorn https://thornelabs.net/20 elabs.net/2016/07/2 16/07/23/kickstart-centos3/kickstart-centos-7-with-eth0-instea 7-with-eth0-instead-of-predicta d-of-predictableblenetwork-interface-names.html




















Estabilsholdnotationofnetworkinterfaces UninstallNetWorkManager

In order to replace the new notation of the network interface we need to remove the NetworkManager NetworkManager package from the server






















Copy and alter the interface scripts accordingly. accordingly. I also create a backup file of the t he original file. (I also remove the " basicly because it looks out of place) Change ifcfg-ens192 to ifcfg-eth2

Change ifcfg-ens224 to ifcfg-eth1

Change ifcfg-ens256 to ifcfg-eth0






















Now you are done with modifying the grub textfile. Now you need to create a binary file which will be read during the t he boot sequence.

Time synchron synchronization ization
































































If you are still not getting the correct zones. That is a known issue. Don’t worry you have done the correct steps. We just need to tell the server that we really mean what we have just done.















































ISCSI Install iscsi package

Install the needed iscsi pacakges.















































Then we need to create a pseudo iscsi interface for the storage network.





















































































































































instead gained sufficient control of the boot sequence that we














































































































































































































































































































































































































Check to see the current quorum state of the cluster

Because the new cluster node node is not included in any fencing mechanism mechanism . The services will not start


















Remember we set the pcmk_host_list pcmk_host_list when we generated the fencing mechanism.

There are possibilities to modify the fencing device. The downside of doing that is that all services that are currently protected by the fencing device will be stopped. In my case I have multiple fencing devices which are protecting my resources. What I can do, and this will not stop the resources, is to delete the fencing devices and recreate them one by one with the new node included.

Check to see the current state of the cluster.


















That was fun. Let’s do it again. To add node number 6.

What I am going to do is to go through the routine one more time, and I will just show you that there is a node number 6 in my cluster when I am done. There we go. A 6 node cluster.

Settingaclusternodeinstandbymode

…



































… Banningaresourcefromaclusternode

In stead of setting a node in standby mode we are able to ban certain resources from running on a specific node. If I would like to make sure that the GFS2 filesystems filesystems will never run on my number 6 node I could ban the GFS2 resources. But since we have already set p that the dlm resource must start first on the node before nay GFS2 resources are started I am going to ban the dlm resource.

The warnign text is telling you that you have banned he resource forever, and this may have some consequences if all other nodes goes down.

Removingtheban




















Now you can remove the ban like it is a normal constraint

Movingaclusterresource.

Since my cluster is a opt-in cluster where all nodes are able to run all resources there are not very many resources to move. But to show how it is done I am going to move the scsi-shooter scsi-shooter resource.


















You should take care with moving the resoures because they will become INFINITY. So let’s have some fun. I have moved the scsi-shooter resource to my privcl04 node. What will happen if I fence the number 4 node ?

Rest easy, your cluster remains operational, the fencing resource is still relocated to a different cluster node, but what happens is that when the number 4 node is re-introduced re -introduced to the cluster the fencing device is relocated back to the number 4 node. Therefor I am going to delete delete the move constraint for the fencing device. device.

Remove nodes from the cluster

My cluster now consist of 6 nodes.


















To check that the cluster node is now really gone.

That’s it

Thank you for reading. I hope it was useful.

Kjell Erik Furnes

POC Linux7 Multinode GFS2 Cluster on CentOS 7.3

Recommend Documents