HACMP TL Upgrade procedure

Pre-requistes for both the nodes.

1: Make sure you have fresh copy of mksysb for A and B node
2: Make sure you have sufficient space on /tmp and /usr

3: Take the following outputs for both nodes

a: lspv,lsvg,netstat -in,netstat -ir,lsfs

b: clstat,cltopinfo,clRGinfo

c: cluster snapshots

d: HDLM Settings.

4: Create a temporary logical volume on both nodes and copy the tl8,sp6,hacmp and hacmp service pack.

Location of the Softwares for TL8,SP6,HACMP AND HACMP Service Pack.

/data/OS_install/TL8/5300tl8sp2.tar.gz ( For TL8) --- this is my location on NIM server :)

/data/OS_install/Update53TL8SP6 ( Service pack)

/data/OS_install/HACMP5.4 ( Location for HACMP)

/data/OS_install/HACMP5.4/sps ( HACMP Service Pack )


Implementation Plan :

1: Shutdown the cluster services on Node B(which one is passive) after informing respective application team
a: check the output of clstat

b: sync the cluster smitty hacmp------>Problem Determination ----------> HACMP VERIFICATION OR cldare -tr)

c: Bring the resource group offline on Node B
d: shutdown the cluster gracefully.

e: lssrc -g cluster clRGinfo resource group should show offline ( This output should as empty )


2: copy the TL8 software from NIM backup network to the local filesystem unzip and untar it.

smitty update_all

Do preview as “yes” with Commit software updates as “Yes” and Accept License Agreements as “Yes”
If no errors, proceed with installation by changing preview as “no”.
After installation check “oslevel –s”.If the oslevel still shows less than TL8,then probably few filesets are not in applied state. Reapply the filesets that are at lower level (lower level file sets can be found using instfix command).

Note: Entire TL,SP6,HACMP you can also install it in applied state and after checking with application team regarding, if everything is fine and make it commit also.

bootlist –m normal –o (verify both the disks in the rootvg are shown)


Reboot the server and check the output oslevel -s


3: SP6 installation

Mount the file system /data/OS_install from nim as /mnt using backup network. Omit this if already in mounted state.
a. Check whether fileset rsct.opt.storagerm is installed or not. If not, install base fileset (2.4.5.0) found under /mnt. Verify it using lslpp.
b. Use “smitty update_all” and select the source directory as /mnt/Update53TL8SP6
c. Do preview as “yes” with Commit software updates as “Yes” and Accept License Agreements as “Yes”
d. If no errors, proceed with installation by changing preview as “no”.
e. After installation check “oslevel –s”.It should show 5300-08-06-0918 and check the version of fileset rsct.opt.storagerm version (2.4.9.0).
f. bootlist –m normal –o (verify both the disks in the rootvg are shown)
g. Reboot the server.

4: HACMP Upgrade Procedure :

a: Check if the cluster is running or not,If running stop it

b: Remove the cluster filesets

smitty remove

c: mv /usr/es/sbin/cluster /usr/es/sbin/cluster.old


d: Install the HACMP Filesets which are copied to the temporary filesystem on Node B

smitty install_latest

Do preview as “yes” with Commit software updates as “Yes” and Accept License Agreements as “Yes”. If no errors, proceed with installation by changing preview as “no”.


e: Install the HACMP Service pack

f: Reboot the server


5: HDLM UPGRADE:

5.2 For HDLM 5.42
a. Backup HDLM settings – ** dlnkmgr view –sys > /var/dlmset1
** dlmodmset –o > /var/dlmset2
Execute “dlmmigsts” under /data/OS_install/HDLM6.1/HDLM_AIX_6101/HS066-59_HDLM_AIX6100sw/hdlmtool directory as below
** #./dlmmigsts –b -odm /var/dlmset3 –set /var/dlmset4. This backs up ODM settings.
b. dlmrmdev –A ? This unmounts the file systems and varyoff the vg’s that are used by HDLM.If any error, check and manually and unmount them and then reexecute the command.
c. Remove the HDLM software using smitty.
d. Create the directory /var/DLM
e. echo "X0YAFGHIJKLMX9A89ABCDHIJK0123PQTST2VX0YZ62A6D768" > /var/DLM/dlm.lic_key
f. Copy the license key file hdlm_license under to /var/tmp
g. Go to directory /mnt/HDLM6.1/HDLM_AIX_6101/HS066-59_HDLM_AIX6100sw and type # installp -aXgd . all. This will install HDLM 6.1. Sometimes it gives error that “Hitachi.aix.support.rte” is missing. In that case go to directory /mnt/HDLM6.1/HDLM_AIX_6101/HS066-59_HDLM_AIX6100sw/ AIX_ODM/HTC_ODM and install the fileset.Once done proceed with HDLM 6.1 installation. There is HDLM service pack that needs to be installed. File is found under
/data/OS_install/HDLM6.1/HDLM_AIX_6101/HS278-04_HDLM_AIX6101servicepack/DLM06-10-01P_E01_AIX. Go to this directory.
h. Run installp -aXgd . all.

i. Run cfgmgr
j. dlnkmgr view –sys should show service pack version as 6.1.0-01P.
k. Restore the HDLM settings saved as above.
#./dlmmigsts -r -odm /var/dlmset3 -set /var/dlmset4
j. dlnkmgr view –sys to verify the settings (failback, path health check etc.) and they should be same as /var/dlmset1.
k. dlnkmgr view –drv should show all devices as hdisk* and not as dlm devices.
l. dlnkmgr view –path shows all the path status.


Repeat the same steps for Node 1 ,check if any backup running on Node 1


Applying the cluster snapshot
After reboot of the both nodes apply the cluster snapshot on primrary node as follows


On the primary node, copy the "odm" portion of the snapshot from your
home directory to /usr/es/sbin/cluster/snapshots.


Apply the Snapshot now which is converted.

Sync the cluster from primrary node

cldare -tr

make sure it completes successfully


Start the cluster.

First on primary:
smitty clstart
make sure to startup cluster information daemon
tail -f /tmp/hacmp.out -> to watch it come up, watch for errors

After first node is up, start cluster on secondary.