Spectrum Scale: How to set systemd to auto start GPFS daemons after reboot

GPFS comes with a systemd unit file (/usr/lpp/mmfs/lib/systemd/gpfs.service) that comes with the RPMs by default.

systemctl status gpfs.service


[root@IBMPOWER_HOST1 ~]#  mmgetstate -a

 Node number  Node name        GPFS state
-------------------------------------------
       1      IBMPOWER_HOST1        active
       2      IBMPOWER_HOST2        active
       3      IBMPOWER_HOST3        active
[root@IBMPOWER_HOST1 ~]# systemctl status gpfs.service
● gpfs.service - General Parallel File System
   Loaded: loaded (/usr/lib/systemd/system/gpfs.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-04-19 00:39:51 EDT; 5 days ago
  Process: 65927 ExecStart=/usr/lpp/mmfs/bin/mmremote startSubsys systemd $STARTSUBSYS_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 65959 (runmmfs)
    Tasks: 689
   Memory: 1.0G
   CGroup: /system.slice/gpfs.service
           ├─65959 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs
           └─66297 /usr/lpp/mmfs/bin/mmfsd

Apr 19 00:39:51 IBMPOWER_HOST1 systemd[1]: Started General Parallel File System.
Apr 19 00:39:55 IBMPOWER_HOST1 mmfs[66297]: [N] CCR: failed to connect to node 9.114.75.217:1191 (sock 53 err 79)
Apr 19 00:39:56 IBMPOWER_HOST1 mmfs[66297]: [N] Connecting to 9.114.75.217 IBMPOWER_HOST3 <c0p0>
Apr 19 00:39:56 IBMPOWER_HOST1 mmfs[66297]: [N] This node (9.114.75.216 (IBMPOWER_HOST1)) is now Cluster Manager for SMPICI_gpfs.IBMPOWER_HOST1.
Apr 19 00:39:56 IBMPOWER_HOST1 mmfs[66297]: [N] mmfsd ready
Apr 19 00:39:57 IBMPOWER_HOST1 mmfs[66297]: [N] Node 9.114.75.217 (IBMPOWER_HOST3) appointed as manager for gpfs_fs.
Apr 20 16:08:19 IBMPOWER_HOST1 mmfs[66297]: [N] Node 9.114.75.215 (IBMPOWER_HOST2) lease renewal is overdue. Pinging to check if it is alive
Apr 20 18:23:10 IBMPOWER_HOST1 mmfs[66297]: [N] Node 9.114.75.217 (IBMPOWER_HOST3) lease renewal is overdue. Pinging to check if it is alive
Apr 20 20:25:15 IBMPOWER_HOST1 mmfs[66297]: [N] Node 9.114.75.217 (IBMPOWER_HOST3) lease renewal is overdue. Pinging to check if it is alive
Apr 20 22:46:10 IBMPOWER_HOST1 mmfs[66297]: [N] Node 9.114.75.217 (IBMPOWER_HOST3) lease renewal is overdue. Pinging to check if it is alive
[root@IBMPOWER_HOST1 ~]#


[root@IBMPOWER_HOST1 ~]# systemctl is-active gpfs.service
active
[root@IBMPOWER_HOST1 ~]# systemctl is-enabled gpfs.service
disabled
[root@IBMPOWER_HOST1 ~]#
[root@IBMPOWER_HOST1 ~]# systemctl is-failed gpfs.service
active
[root@IBMPOWER_HOST1 ~]#



This will return active if it is running properly or failed if an error occurred.
If the unit was intentionally stopped, it may return unknown or inactive


---------------------------------

[root@IBMPOWER_HOST1 ~]# rm -rf /etc/systemd/system/multi-user.target.wants/gpfs.service
[root@IBMPOWER_HOST1 ~]# systemctl enable  gpfs.service
Created symlink from /etc/systemd/system/multi-user.target.wants/gpfs.service to /usr/lib/systemd/system/gpfs.service.
[root@IBMPOWER_HOST1 ~]# ls -alsrt /etc/systemd/system/multi-user.target.wants/gpfs.service
0 lrwxrwxrwx 1 root root 36 Apr 24 03:41 /etc/systemd/system/multi-user.target.wants/gpfs.service -> /usr/lib/systemd/system/gpfs.service
[root@IBMPOWER_HOST1 ~]#

This will create a symbolic link from the system’s copy of the service file (usually in /lib/systemd/system or /etc/systemd/system)
into the location on disk where systemd looks for autostart files (usually /etc/systemd/system/some_target.target.wants.


--------------------------------------------
[root@IBMPOWER_HOST1 ~]# systemctl enable  gpfs.service
Created symlink from /etc/systemd/system/multi-user.target.wants/gpfs.service to /usr/lib/systemd/system/gpfs.service.
[root@IBMPOWER_HOST1 ~]#

[root@IBMPOWER_HOST1 ~]# systemctl enable  gpfs.service
Created symlink from /etc/systemd/system/multi-user.target.wants/gpfs.service to /usr/lib/systemd/system/gpfs.service.
[root@IBMPOWER_HOST1 ~]# ls -alsrt /etc/systemd/system/multi-user.target.wants/gpfs.service
0 lrwxrwxrwx 1 root root 36 Apr 24 03:41 /etc/systemd/system/multi-user.target.wants/gpfs.service -> /usr/lib/systemd/system/gpfs.service
[root@IBMPOWER_HOST1 ~]#


Lets check the status

[root@IBMPOWER_HOST1 ~]# systemctl status gpfs.service
● gpfs.service - General Parallel File System
   Loaded: loaded (/usr/lib/systemd/system/gpfs.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2020-04-24 03:35:55 EDT; 10min ago
 Main PID: 59025 (runmmfs)
   CGroup: /system.slice/gpfs.service
           ├─59025 /usr/lpp/mmfs/bin/mmksh /usr/lpp/mmfs/bin/runmmfs
           └─59548 /usr/lpp/mmfs/bin/mmfsd

Apr 24 03:35:55 IBMPOWER_HOST1 systemd[1]: Starting General Parallel File System...
Apr 24 03:35:55 IBMPOWER_HOST1 systemd[1]: Can't open PID file /var/mmfs/gen/runmmfsPid (yet?) after start: No such file or directory
Apr 24 03:35:55 IBMPOWER_HOST1 systemd[1]: Started General Parallel File System.
Apr 24 03:36:00 IBMPOWER_HOST1 mmfs[59548]: [N] Connecting to 9.114.75.215 IBMPOWER_HOST2 <c0p2>
Apr 24 03:36:00 IBMPOWER_HOST1 mmfs[59548]: [N] mmfsd ready
Apr 24 03:36:01 IBMPOWER_HOST1 mmfs[59548]: [N] Connecting to 9.114.75.217 IBMPOWER_HOST3 <c0n0>
[root@IBMPOWER_HOST1 ~]#




[root@IBMPOWER_HOST1 ~]#
[root@IBMPOWER_HOST1 ~]#  mmgetstate -a

 Node number  Node name        GPFS state
-------------------------------------------
       1      IBMPOWER_HOST1        active
       2      IBMPOWER_HOST2        active
       3      IBMPOWER_HOST3        active
[root@IBMPOWER_HOST1 ~]#
[root@IBMPOWER_HOST1 ~]# systemctl is-enabled gpfs.service
enabled
[root@IBMPOWER_HOST1 ~]#


How to auto mount the filesystem when GPFS daemon starts - if you have not done at installation 


export PATH=$PATH:/usr/lpp/mmfs/bin

[root@IBMPOWER_HOST1 ~]#  mmgetstate -a

 Node number  Node name        GPFS state
-------------------------------------------
       1      IBMPOWER_HOST1        active
       2      IBMPOWER_HOST2        active
       3      IBMPOWER_HOST3        active
[root@IBMPOWER_HOST1 ~]# mmchfs gpfs_fs -A yes
mmchfs: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[root@IBMPOWER_HOST1 ~]#

-A {yes | no | automount}
    Indicates when the file system is to be mounted:

    yes
        When the GPFS daemon starts.


Verify:

[root@IBMPOWER_HOST1 ~]# mmlsfs gpfs_fs  -A
flag                value                    description
------------------- ------------------------ -----------------------------------
 -A                 yes                      Automatic mount option
[root@IBMPOWER_HOST1 ~]#

How to Auto-load and auto-mount Spectrum scale in simple steps ?

[root@IBMPOWER_HOST1 ~]# mmchconfig autoload=yes
mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[root@IBMPOWER_HOST1 ~]#

 Reference:
https://www.ibm.com/support/knowledgecenter/STXKQY_5.0.4/com.ibm.spectrum.scale.v5r04.doc/bl1adm_mmchfs.htm
https://www.ibm.com/support/knowledgecenter/en/SSCKLT_2.0.0/UG/sec_ug_starting_scale_in_system_file.html

Bash Script: How to check the status of background processes

Example 1 :  Failure case with exit 24

[smpici@c712f6n06 myscripts]$ cat simple_bkGroundproc.sh
#!/bin/sh

cmd() { sleep 5; exit 24; }

cmd &   # Run the long running process
pid=$!  # Record the pid

# Spawn a process that coninually reports that the command is still running
while echo "$(date): $pid is still running"; do sleep 1; done &
echoer=$!

# Set a trap to kill the reporter when the process finishes
trap 'kill $echoer' 0

# Wait for the process to finish
if wait $pid; then
    echo "cmd succeeded"
else
    echo "cmd FAILED!! (returned $?)"
fi
[sachin@host1 myscripts]$ ./simple_bkGroundproc.sh
Wed Mar  4 06:36:34 EST 2020: 96853 is still running
Wed Mar  4 06:36:35 EST 2020: 96853 is still running
Wed Mar  4 06:36:36 EST 2020: 96853 is still running
Wed Mar  4 06:36:37 EST 2020: 96853 is still running
Wed Mar  4 06:36:38 EST 2020: 96853 is still running
cmd FAILED!! (returned 24)
[sachin@host1 myscripts]$

--------------------------------------------

Example 1 :  Passed case with exit 0


#!/bin/sh

cmd() { sleep 5; exit 0; }

cmd &   # Run the long running process
pid=$!  # Record the pid

# Spawn a process that coninually reports that the command is still running
while echo "$(date): $pid is still running"; do sleep 1; done &
echoer=$!

# Set a trap to kill the reporter when the process finishes
trap 'kill $echoer' 0

# Wait for the process to finish
if wait $pid; then
    echo "cmd succeeded"
else
    echo "cmd FAILED!! (returned $?)"
fi

------------
OUTPUT
[sachin@host1 myscripts]$ ./simple_bkGroundproc.sh
Wed Mar  4 06:37:04 EST 2020: 97118 is still running
Wed Mar  4 06:37:05 EST 2020: 97118 is still running
Wed Mar  4 06:37:06 EST 2020: 97118 is still running
Wed Mar  4 06:37:07 EST 2020: 97118 is still running
Wed Mar  4 06:37:08 EST 2020: 97118 is still running
cmd succeeded


 ++++++++++++++++++++++++++++++++++++++++

Example 3 :

#! /bin/bash
fvt_test_arg="Default HCOLL IBV_ENABLE pami_noib"
IFS=', ' read -r -a fvt_test_arg_array <<< "$fvt_test_arg"

items="1 2 3 4"
pids=""

for item in $items; do
    sleep $item &
    pids+="$! "
done
echo "list of pids $pids"
i=0


for pid in $pids; do
    wait $pid
    if [ $? -eq 0 ]; then
        echo "${fvt_test_arg_array[i]} SUCCESS - Job $pid exited with a status of $?"
    else
        echo "${fvt_test_arg_array[i]} FAILED - Job $pid exited with a status of $?"
    fi
    i=$((i+1))
done
-----------------------
OUTPUT:
[sachin@host1 myscripts]$ ./pid_array2.sh
list of pids 81822 81823 81824 81825
Default SUCCESS - Job 81822 exited with a status of 0
HCOLL SUCCESS - Job 81823 exited with a status of 0
IBV_ENABLE SUCCESS - Job 81824 exited with a status of 0
pami_noib SUCCESS - Job 81825 exited with a status of 0
[sachin@host1 myscripts]$

-----------------------

Reference:
https://stackoverflow.com/questions/1570262/get-exit-code-of-a-background-process

Bash script : How to split string to array and count/print array elements based on index

IFS    The Internal Field Separator that is used for word splitting after expansion and to split lines into words with the read builtin command. The default value is <space><tab><newline>.


Example 1 :

#!/bin/bash
fvt_test_arg="DEFAULT HCOLL IBV PAMI_NOIB"
echo "This is fvt_test_arg : $fvt_test_arg"
IFS=', ' read -r -a fvt_test_arg_array <<< "$fvt_test_arg"
echo "This is fvt_test_arg_array : ${fvt_test_arg_array[@]}"
echo "Number of elements in the array : ${#fvt_test_arg_array[@]}"
echo "1st element is :${fvt_test_arg_array[0]}"
echo "2nd element is :${fvt_test_arg_array[1]}"
echo "3rd element is :${fvt_test_arg_array[2]}"
echo "4th element is :${fvt_test_arg_array[3]}"


----------------------------
OUTPUT:

[sachin@host1 myscripts]$ ./list4.sh
This is fvt_test_arg : DEFAULT HCOLL IBV PAMI_NOIB        ===> String
This is fvt_test_arg_array : DEFAULT HCOLL IBV PAMI_NOIB  ====> array
Number of elements in the array : 4
1st element is :DEFAULT
2nd element is :HCOLL
3rd element is :IBV
4th element is :PAMI_NOIB
[sachin@host1 myscripts]$

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
EXAMPLE 2:

#!/bin/bash
fvt_test_arg="DEFAULT HCOLL IBV PAMI_NOIB"
echo "This is fvt_test_arg : $fvt_test_arg"
IFS=', ' read -r -a fvt_test_arg_array <<< "$fvt_test_arg"
echo "This is fvt_test_arg_array : ${fvt_test_arg_array[@]}"
echo "Number of elements in the array : ${#fvt_test_arg_array[@]}"
echo "1st element is :${fvt_test_arg_array[0]}"
echo "2nd element is :${fvt_test_arg_array[1]}"
echo "3rd element is :${fvt_test_arg_array[2]}"
echo "4th element is :${fvt_test_arg_array[3]}"

pids="1111 2222 3333 4444"
i=0
for pid in ${pids[@]}; do
    echo " $pid is the PID of the process for :${fvt_test_arg_array[i]} is successful"
    i=$((i+1))
done

 ---------------
OUTPUT:
[sachin@host1 myscripts]$ ./list5.sh
This is fvt_test_arg : DEFAULT HCOLL IBV PAMI_NOIB
This is fvt_test_arg_array : DEFAULT HCOLL IBV PAMI_NOIB
Number of elements in the array : 4
1st element is :DEFAULT
2nd element is :HCOLL
3rd element is :IBV
4th element is :PAMI_NOIB
 1111 is the PID of the process for :DEFAULT is successful
 2222 is the PID of the process for :HCOLL is successful
 3333 is the PID of the process for :IBV is successful
 4444 is the PID of the process for :PAMI_NOIB is successful
[smpici@c712f6n06 myscripts]$


------------------------------------
 EXAMPLE  3:

#!/bin/bash

declare -a num
declare -a words

num=(ddd1 dd2 dd3 dd4 dd5 dd6 dd7)
words=(one two three four five six seven)

n=${#num[@]}
echo $n
for ((i=0;i<$n;i++)); do
  echo ":${num[$i]} :${words[$i]}:"
done
[sachin@host1 myscripts]$

--------------------------

OUTPUT:
[sachin@host1 myscripts]$ ./list3.sh
7
:ddd1 :one:
:dd2 :two:
:dd3 :three:
:dd4 :four:
:dd5 :five:
:dd6 :six:
:dd7 :seven:
-----------------------------------------------

Reference:
https://stackoverflow.com/questions/10586153/split-string-into-an-array-in-bash/13196466