On December 12th, I presented on the DB2 Night Show on HADR and TSAMP Advanced Topics. See the replay here: http://www.dbisoftware.com/blog/db2nightshow.php?id=531

Using TSA/db2haicu to automate failover – Part 1: The Preparation

Edited 8/3/2012 to add a detail about db2cptsa and to put db2nodes.cfg in its own section.

I’ve actually been hating db2haicu lately. I had 3 different environments where I had to set it up over the course of about 2 weeks, and none of them went simply and smoothly. As of the writing of this post, I’ve managed to get one working by myself and one working with the help of one of our Linux gurus. The other is still hanging out there, waiting to be revisited. I’ve learned some things on the other two that I’m going to try, and if they don’t work, then it’ll be PMR-land for me.

Concepts Of db2haicu And Automated Failover

I originally heard that failover was automated in 9.5. Silly me for thinking it was fully integrated and could be set up via a few parameters. My bubble on that was burst a couple of years ago. Expect this to be like many DB2 features early in their lifecycle – like user exit was early on, this is not fully integrated or super easy.
The over all picture for automating failover is:

  1. Set up HADR using hostnames, not IP addresses
  2. Prepare for db2haicu
  3. Run db2haicu
  4. Test failovers

Behind the scenes, db2haicu is issuing Tivoli System Automation (TSA) commands. It doesn’t take much googling to find basically what it is doing, because there are details on the more mannual way there was of doing this back with 8.2. And you’ll still need root or sudo permissions for a few direct commands – both on setup and for ongoing support.

Prep Work, Part One: Having All Your Information Ready

The first thing to do is to make several decisions about your environment and how your setup will work. This generally includes:

    1. Will you have a Virtual IP for the database connection?
    2. What will you use for a network Quorum device? A network quorum is just a pingable IP address that is used to decide which node in the cluster will serve as the “active” node during a site failure, and which nodes will be offline. It should be a server that is very highly available such as a domain controller.
    3. Is there a private network between the two database servers that should be used by HADR and as a part of automated failover? If so, you’ll need details on that.

With those decisions in mind, here are the inputs you’ll need to have ready:

REQUIRED

  1. Public IP address of both database servers
  2. Fully qualified host names of both database servers
  3. IP address of the Quorum device
  4. Names of network cards associated with the public network on the servers IF there is more than one network card on each server(for example, ‘eth0’)

OPTIONAL

  1. Virtual IP address if using one, along with the subnet mask
  2. Private IP addresses, if private network being used in addition to the public network
  3. Name of network cards for the private network, if one is being used (for example, ‘eth0’)

Prep Work, Part Two: Actual Actions to Take Before You Run db2haicu

Software Installed

Yes, by default, the TSA components are installed with db2. However, it is a fairly common failure that on installing db2 the TSA components fail to be installed. This is an installation message that can be ignored most of the time for non-production servers because you’re not likely to use TSA there. One interesting scenario that I’ve run into is that when installing on Red Hat 6, the version of TSA that is part of the 9.7 base code cannot be installed because it does not work with Red Hat 6. But with I think Fixpack 1 (though it could be 3 or 4), the version of TSA bundled with DB2 does work on Red Hat 6. But if you install from the base code and then apply the FixPack, you didn’t install the code so the FixPack won’t update it. In this case you have to manually install this component. It’s not that hard.

If you’re lucky, everything went OK on DB2 installation, and you won’t have to do this step.

To install the code(yes, the example below is from Red Hat 5.6 – it’s what I had available):
Log in as root
cd path_where_fixpack_code_is/universal/db2/linuxamd64/tsamp

# ./prereqSAM
prereqSAM: All prerequisites for the ITSAMP installation are met on operating system:
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
# ./installSAM
prereqSAM: All prerequisites for the ITSAMP installation are met on operating system:
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
SAM is currently not installed.
installSAM: The following package is not installed yet and needs to be installed:  ./Linux/i386/sam-3.2.1.1-10347.i386.rpm

installSAM: A general License Agreement and License Information specifically for System Automation will be shown. Scroll down usi                             ng the Enter key (line by line) or Space bar (page by page). At the end you will be asked to accept the terms to be allowed to in                             stall the product. Select Enter to continue.

installSAM: To accept all terms of the preceding License Agreement and License Information type 'y', anything else to decline.

y
installSAM: You accepted the terms in License Agreement and License information. System Automation will now be installed.

installSAM: Installing System Automation on platform: x86_64
installSAM: Packages will be installed from directory:  ./Linux/i386

installSAM: Installing
 ./Linux/i386/src-1.3.1.1-10320.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.utils-3.1.0.3-10320.i386.rpm
 ./Linux/i386/rsct.core-3.1.0.3-10320.i386.rpm
 ./Linux/i386/rsct.basic-3.1.0.3-10320.i386.rpm
 ./Linux/i386/rsct.opt.storagerm-3.1.0.3-10320.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.64bit-3.1.0.3-10313.x86_64.rpm
package rsct.core.cimrm is not installed

installSAM: Installing
 ./Linux/i386/src.msg.de_DE-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.de_DE.ISO-8859-1-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.de_DE@euro-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.de_DE.UTF-8-1.3.0.3-0.i386.rpm

installSAM: Installing
 ./Linux/i386/src.msg.es_ES-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.es_ES.ISO-8859-1-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.es_ES@euro-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.es_ES.UTF-8-1.3.0.3-0.i386.rpm

installSAM: Installing
 ./Linux/i386/src.msg.fr_FR-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.fr_FR.ISO-8859-1-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.fr_FR@euro-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.fr_FR.UTF-8-1.3.0.3-0.i386.rpm

installSAM: Installing
 ./Linux/i386/src.msg.it_IT-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.it_IT.ISO-8859-1-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.it_IT@euro-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.it_IT.UTF-8-1.3.0.3-0.i386.rpm

installSAM: Installing
 ./Linux/i386/src.msg.ja_JP.eucJP-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.ja_JP.UTF-8-1.3.0.3-0.i386.rpm

installSAM: Installing
 ./Linux/i386/src.msg.ko_KR.eucKR-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.ko_KR.UTF-8-1.3.0.3-0.i386.rpm

installSAM: Installing
 ./Linux/i386/src.msg.pt_BR-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.pt_BR.UTF-8-1.3.0.3-0.i386.rpm

installSAM: Installing
 ./Linux/i386/src.msg.zh_CN.GB2312-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.zh_CN.GB18030-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.zh_CN.GBK-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.zh_CN.UTF-8-1.3.0.3-0.i386.rpm

installSAM: Installing
 ./Linux/i386/src.msg.zh_TW-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.zh_TW.Big5-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.zh_TW.eucTW-1.3.0.3-0.i386.rpm
 ./Linux/i386/src.msg.zh_TW.UTF-8-1.3.0.3-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.utils.msg.de_DE-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.de_DE.ISO-8859-1-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.de_DE@euro-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.de_DE.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.utils.msg.es_ES-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.es_ES.ISO-8859-1-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.es_ES@euro-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.es_ES.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.utils.msg.fr_FR-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.fr_FR.ISO-8859-1-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.fr_FR@euro-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.fr_FR.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.utils.msg.it_IT-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.it_IT.ISO-8859-1-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.it_IT@euro-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.it_IT.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.utils.msg.ja_JP.eucJP-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.ja_JP.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.utils.msg.ko_KR.eucKR-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.ko_KR.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.utils.msg.pt_BR-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.pt_BR.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.utils.msg.zh_CN.GB2312-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.zh_CN.GB18030-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.zh_CN.GBK-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.zh_CN.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.utils.msg.zh_TW-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.zh_TW.Big5-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.zh_TW.eucTW-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.utils.msg.zh_TW.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.msg.de_DE-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.de_DE.ISO-8859-1-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.de_DE@euro-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.de_DE.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.msg.es_ES-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.es_ES.ISO-8859-1-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.es_ES@euro-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.es_ES.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.msg.fr_FR-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.fr_FR.ISO-8859-1-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.fr_FR@euro-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.fr_FR.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.msg.it_IT-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.it_IT.ISO-8859-1-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.it_IT@euro-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.it_IT.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.msg.ja_JP.eucJP-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.ja_JP.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.msg.ko_KR.eucKR-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.ko_KR.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.msg.pt_BR-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.pt_BR.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.msg.zh_CN.GB2312-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.zh_CN.GB18030-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.zh_CN.GBK-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.zh_CN.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.core.msg.zh_TW-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.zh_TW.Big5-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.zh_TW.eucTW-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.core.msg.zh_TW.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.basic.msg.de_DE-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.de_DE.ISO-8859-1-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.de_DE@euro-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.de_DE.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.basic.msg.es_ES-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.es_ES.ISO-8859-1-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.es_ES@euro-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.es_ES.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.basic.msg.fr_FR-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.fr_FR.ISO-8859-1-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.fr_FR@euro-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.fr_FR.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.basic.msg.it_IT-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.it_IT.ISO-8859-1-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.it_IT@euro-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.it_IT.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.basic.msg.ja_JP.eucJP-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.ja_JP.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.basic.msg.ko_KR.eucKR-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.ko_KR.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.basic.msg.pt_BR-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.pt_BR.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.basic.msg.zh_CN.GB2312-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.zh_CN.GB18030-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.zh_CN.GBK-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.zh_CN.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.basic.msg.zh_TW-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.zh_TW.Big5-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.zh_TW.eucTW-3.1.0.1-0.i386.rpm
 ./Linux/i386/rsct.basic.msg.zh_TW.UTF-8-3.1.0.1-0.i386.rpm

installSAM: Installing
 ./Linux/i386/sam-3.2.1.1-10347.i386.rpm

installSAM: Installing
 ./Linux/i386/sam.msg.de_DE-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.de_DE.ISO-8859-1-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.de_DE@euro-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.de_DE.UTF-8-3.2.1.0-0.i386.rpm

installSAM: Installing
 ./Linux/i386/sam.msg.es_ES-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.es_ES.ISO-8859-1-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.es_ES@euro-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.es_ES.UTF-8-3.2.1.0-0.i386.rpm

installSAM: Installing
 ./Linux/i386/sam.msg.fr_FR-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.fr_FR.ISO-8859-1-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.fr_FR@euro-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.fr_FR.UTF-8-3.2.1.0-0.i386.rpm

installSAM: Installing
 ./Linux/i386/sam.msg.it_IT-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.it_IT.ISO-8859-1-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.it_IT@euro-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.it_IT.UTF-8-3.2.1.0-0.i386.rpm

installSAM: Installing
 ./Linux/i386/sam.msg.ja_JP.eucJP-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.ja_JP.UTF-8-3.2.1.0-0.i386.rpm

installSAM: Installing
 ./Linux/i386/sam.msg.ko_KR.eucKR-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.ko_KR.UTF-8-3.2.1.0-0.i386.rpm

installSAM: Installing
 ./Linux/i386/sam.msg.pt_BR-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.pt_BR.UTF-8-3.2.1.0-0.i386.rpm

installSAM: Installing
 ./Linux/i386/sam.msg.zh_CN.GB2312-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.zh_CN.GB18030-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.zh_CN.GBK-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.zh_CN.UTF-8-3.2.1.0-0.i386.rpm

installSAM: Installing
 ./Linux/i386/sam.msg.zh_TW-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.zh_TW.Big5-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.zh_TW.eucTW-3.2.1.0-0.i386.rpm
 ./Linux/i386/sam.msg.zh_TW.UTF-8-3.2.1.0-0.i386.rpm

installSAM: Installing
 ./Linux/i386/rsct.opt.stackdump-3.1.0.3-10320.i386.rpm

installSAM: Installing
 ./Linux/i386/sam.sappolicy-3.2.1.1-10347.i386.rpm

installSAM: Installing
 ./Linux/i386/sam.adapter-3.2.1.1-10347.i386.rpm
installSAM: Installed System Automation package and prerequisites:
 rsct.64bit-3.1.0.3-10313.x86_64
 rsct.basic-3.1.0.3-10320.i386
 rsct.basic.msg.de_DE-3.1.0.1-0.i386
 rsct.basic.msg.de_DE@euro-3.1.0.1-0.i386
 rsct.basic.msg.de_DE.ISO-8859-1-3.1.0.1-0.i386
 rsct.basic.msg.de_DE.UTF-8-3.1.0.1-0.i386
 rsct.basic.msg.es_ES-3.1.0.1-0.i386
 rsct.basic.msg.es_ES@euro-3.1.0.1-0.i386
 rsct.basic.msg.es_ES.ISO-8859-1-3.1.0.1-0.i386
 rsct.basic.msg.es_ES.UTF-8-3.1.0.1-0.i386
 rsct.basic.msg.fr_FR-3.1.0.1-0.i386
 rsct.basic.msg.fr_FR@euro-3.1.0.1-0.i386
 rsct.basic.msg.fr_FR.ISO-8859-1-3.1.0.1-0.i386
 rsct.basic.msg.fr_FR.UTF-8-3.1.0.1-0.i386
 rsct.basic.msg.it_IT-3.1.0.1-0.i386
 rsct.basic.msg.it_IT@euro-3.1.0.1-0.i386
 rsct.basic.msg.it_IT.ISO-8859-1-3.1.0.1-0.i386
 rsct.basic.msg.it_IT.UTF-8-3.1.0.1-0.i386
 rsct.basic.msg.ja_JP.eucJP-3.1.0.1-0.i386
 rsct.basic.msg.ja_JP.UTF-8-3.1.0.1-0.i386
 rsct.basic.msg.ko_KR.eucKR-3.1.0.1-0.i386
 rsct.basic.msg.ko_KR.UTF-8-3.1.0.1-0.i386
 rsct.basic.msg.pt_BR-3.1.0.1-0.i386
 rsct.basic.msg.pt_BR.UTF-8-3.1.0.1-0.i386
 rsct.basic.msg.zh_CN.GB18030-3.1.0.1-0.i386
 rsct.basic.msg.zh_CN.GB2312-3.1.0.1-0.i386
 rsct.basic.msg.zh_CN.GBK-3.1.0.1-0.i386
 rsct.basic.msg.zh_CN.UTF-8-3.1.0.1-0.i386
 rsct.basic.msg.zh_TW-3.1.0.1-0.i386
 rsct.basic.msg.zh_TW.Big5-3.1.0.1-0.i386
 rsct.basic.msg.zh_TW.eucTW-3.1.0.1-0.i386
 rsct.basic.msg.zh_TW.UTF-8-3.1.0.1-0.i386
 rsct.core-3.1.0.3-10320.i386
 rsct.core.msg.de_DE-3.1.0.1-0.i386
 rsct.core.msg.de_DE@euro-3.1.0.1-0.i386
 rsct.core.msg.de_DE.ISO-8859-1-3.1.0.1-0.i386
 rsct.core.msg.de_DE.UTF-8-3.1.0.1-0.i386
 rsct.core.msg.es_ES-3.1.0.1-0.i386
 rsct.core.msg.es_ES@euro-3.1.0.1-0.i386
 rsct.core.msg.es_ES.ISO-8859-1-3.1.0.1-0.i386
 rsct.core.msg.es_ES.UTF-8-3.1.0.1-0.i386
 rsct.core.msg.fr_FR-3.1.0.1-0.i386
 rsct.core.msg.fr_FR@euro-3.1.0.1-0.i386
 rsct.core.msg.fr_FR.ISO-8859-1-3.1.0.1-0.i386
 rsct.core.msg.fr_FR.UTF-8-3.1.0.1-0.i386
 rsct.core.msg.it_IT-3.1.0.1-0.i386
 rsct.core.msg.it_IT@euro-3.1.0.1-0.i386
 rsct.core.msg.it_IT.ISO-8859-1-3.1.0.1-0.i386
 rsct.core.msg.it_IT.UTF-8-3.1.0.1-0.i386
 rsct.core.msg.ja_JP.eucJP-3.1.0.1-0.i386
 rsct.core.msg.ja_JP.UTF-8-3.1.0.1-0.i386
 rsct.core.msg.ko_KR.eucKR-3.1.0.1-0.i386
 rsct.core.msg.ko_KR.UTF-8-3.1.0.1-0.i386
 rsct.core.msg.pt_BR-3.1.0.1-0.i386
 rsct.core.msg.pt_BR.UTF-8-3.1.0.1-0.i386
 rsct.core.msg.zh_CN.GB18030-3.1.0.1-0.i386
 rsct.core.msg.zh_CN.GB2312-3.1.0.1-0.i386
 rsct.core.msg.zh_CN.GBK-3.1.0.1-0.i386
 rsct.core.msg.zh_CN.UTF-8-3.1.0.1-0.i386
 rsct.core.msg.zh_TW-3.1.0.1-0.i386
 rsct.core.msg.zh_TW.Big5-3.1.0.1-0.i386
 rsct.core.msg.zh_TW.eucTW-3.1.0.1-0.i386
 rsct.core.msg.zh_TW.UTF-8-3.1.0.1-0.i386
 rsct.core.utils-3.1.0.3-10320.i386
 rsct.core.utils.msg.de_DE-3.1.0.1-0.i386
 rsct.core.utils.msg.de_DE@euro-3.1.0.1-0.i386
 rsct.core.utils.msg.de_DE.ISO-8859-1-3.1.0.1-0.i386
 rsct.core.utils.msg.de_DE.UTF-8-3.1.0.1-0.i386
 rsct.core.utils.msg.es_ES-3.1.0.1-0.i386
 rsct.core.utils.msg.es_ES@euro-3.1.0.1-0.i386
 rsct.core.utils.msg.es_ES.ISO-8859-1-3.1.0.1-0.i386
 rsct.core.utils.msg.es_ES.UTF-8-3.1.0.1-0.i386
 rsct.core.utils.msg.fr_FR-3.1.0.1-0.i386
 rsct.core.utils.msg.fr_FR@euro-3.1.0.1-0.i386
 rsct.core.utils.msg.fr_FR.ISO-8859-1-3.1.0.1-0.i386
 rsct.core.utils.msg.fr_FR.UTF-8-3.1.0.1-0.i386
 rsct.core.utils.msg.it_IT-3.1.0.1-0.i386
 rsct.core.utils.msg.it_IT@euro-3.1.0.1-0.i386
 rsct.core.utils.msg.it_IT.ISO-8859-1-3.1.0.1-0.i386
 rsct.core.utils.msg.it_IT.UTF-8-3.1.0.1-0.i386
 rsct.core.utils.msg.ja_JP.eucJP-3.1.0.1-0.i386
 rsct.core.utils.msg.ja_JP.UTF-8-3.1.0.1-0.i386
 rsct.core.utils.msg.ko_KR.eucKR-3.1.0.1-0.i386
 rsct.core.utils.msg.ko_KR.UTF-8-3.1.0.1-0.i386
 rsct.core.utils.msg.pt_BR-3.1.0.1-0.i386
 rsct.core.utils.msg.pt_BR.UTF-8-3.1.0.1-0.i386
 rsct.core.utils.msg.zh_CN.GB18030-3.1.0.1-0.i386
 rsct.core.utils.msg.zh_CN.GB2312-3.1.0.1-0.i386
 rsct.core.utils.msg.zh_CN.GBK-3.1.0.1-0.i386
 rsct.core.utils.msg.zh_CN.UTF-8-3.1.0.1-0.i386
 rsct.core.utils.msg.zh_TW-3.1.0.1-0.i386
 rsct.core.utils.msg.zh_TW.Big5-3.1.0.1-0.i386
 rsct.core.utils.msg.zh_TW.eucTW-3.1.0.1-0.i386
 rsct.core.utils.msg.zh_TW.UTF-8-3.1.0.1-0.i386
 rsct.opt.stackdump-3.1.0.3-10320.i386
 rsct.opt.storagerm-3.1.0.3-10320.i386
 sam-3.2.1.1-10347.i386
 sam.adapter-3.2.1.1-10347.i386
 sam.msg.de_DE-3.2.1.0-0.i386
 sam.msg.de_DE@euro-3.2.1.0-0.i386
 sam.msg.de_DE.ISO-8859-1-3.2.1.0-0.i386
 sam.msg.de_DE.UTF-8-3.2.1.0-0.i386
 sam.msg.es_ES-3.2.1.0-0.i386
 sam.msg.es_ES@euro-3.2.1.0-0.i386
 sam.msg.es_ES.ISO-8859-1-3.2.1.0-0.i386
 sam.msg.es_ES.UTF-8-3.2.1.0-0.i386
 sam.msg.fr_FR-3.2.1.0-0.i386
 sam.msg.fr_FR@euro-3.2.1.0-0.i386
 sam.msg.fr_FR.ISO-8859-1-3.2.1.0-0.i386
 sam.msg.fr_FR.UTF-8-3.2.1.0-0.i386
 sam.msg.it_IT-3.2.1.0-0.i386
 sam.msg.it_IT@euro-3.2.1.0-0.i386
 sam.msg.it_IT.ISO-8859-1-3.2.1.0-0.i386
 sam.msg.it_IT.UTF-8-3.2.1.0-0.i386
 sam.msg.ja_JP.eucJP-3.2.1.0-0.i386
 sam.msg.ja_JP.UTF-8-3.2.1.0-0.i386
 sam.msg.ko_KR.eucKR-3.2.1.0-0.i386
 sam.msg.ko_KR.UTF-8-3.2.1.0-0.i386
 sam.msg.pt_BR-3.2.1.0-0.i386
 sam.msg.pt_BR.UTF-8-3.2.1.0-0.i386
 sam.msg.zh_CN.GB18030-3.2.1.0-0.i386
 sam.msg.zh_CN.GB2312-3.2.1.0-0.i386
 sam.msg.zh_CN.GBK-3.2.1.0-0.i386
 sam.msg.zh_CN.UTF-8-3.2.1.0-0.i386
 sam.msg.zh_TW-3.2.1.0-0.i386
 sam.msg.zh_TW.Big5-3.2.1.0-0.i386
 sam.msg.zh_TW.eucTW-3.2.1.0-0.i386
 sam.msg.zh_TW.UTF-8-3.2.1.0-0.i386
 sam.sappolicy-3.2.1.1-10347.i386
 src-1.3.1.1-10320.i386
 src.msg.de_DE-1.3.0.3-0.i386
 src.msg.de_DE@euro-1.3.0.3-0.i386
 src.msg.de_DE.ISO-8859-1-1.3.0.3-0.i386
 src.msg.de_DE.UTF-8-1.3.0.3-0.i386
 src.msg.es_ES-1.3.0.3-0.i386
 src.msg.es_ES@euro-1.3.0.3-0.i386
 src.msg.es_ES.ISO-8859-1-1.3.0.3-0.i386
 src.msg.es_ES.UTF-8-1.3.0.3-0.i386
 src.msg.fr_FR-1.3.0.3-0.i386
 src.msg.fr_FR@euro-1.3.0.3-0.i386
 src.msg.fr_FR.ISO-8859-1-1.3.0.3-0.i386
 src.msg.fr_FR.UTF-8-1.3.0.3-0.i386
 src.msg.it_IT-1.3.0.3-0.i386
 src.msg.it_IT@euro-1.3.0.3-0.i386
 src.msg.it_IT.ISO-8859-1-1.3.0.3-0.i386
 src.msg.it_IT.UTF-8-1.3.0.3-0.i386
 src.msg.ja_JP.eucJP-1.3.0.3-0.i386
 src.msg.ja_JP.UTF-8-1.3.0.3-0.i386
 src.msg.ko_KR.eucKR-1.3.0.3-0.i386
 src.msg.ko_KR.UTF-8-1.3.0.3-0.i386
 src.msg.pt_BR-1.3.0.3-0.i386
 src.msg.pt_BR.UTF-8-1.3.0.3-0.i386
 src.msg.zh_CN.GB18030-1.3.0.3-0.i386
 src.msg.zh_CN.GB2312-1.3.0.3-0.i386
 src.msg.zh_CN.GBK-1.3.0.3-0.i386
 src.msg.zh_CN.UTF-8-1.3.0.3-0.i386
 src.msg.zh_TW-1.3.0.3-0.i386
 src.msg.zh_TW.Big5-1.3.0.3-0.i386
 src.msg.zh_TW.eucTW-1.3.0.3-0.i386
 src.msg.zh_TW.UTF-8-1.3.0.3-0.i386

installSAM: The following license is installed:
Product: IBM Tivoli System Automation for Multiplatforms (Try & Buy License) 3.2
Evaluation Period: 90 days (90 days left)
Creation date: Wed 19 Aug 2009 12:00:01 AM CDT
Expiration date: Thu 31 Dec 2037 12:00:01 AM CST

Subsystem         Group            PID     Status
 ctrmc            rsct             1627    active
installSAM: Warning: Must set CT_MANAGEMENT_SCOPE=2

installSAM: All packages were installed successfully.

 

HADR Setup

It’s really just a straight forward HADR setup – you’ll want to make sure you have your HADR peer window set. See my post on HADR for more details on HADR.

Hosts File

TSA is really picky about the hosts file. You either need to have root (sudo is fine) access or have access to someone with root or sudo access so you can edit it if needed. Note that editing a hosts file in certain ways can break other things on the system, so you’ll likely want your Systems Admin or whatever other engineers you’re working with on the project aware that you’re making changes.

What TSA requires is that both hosts for the HADR pair have one line and only one line each in the hosts file. The entries must be of this format:

ipaddress  shortname   longname

or

ipaddress  longname    shortname

Where longname is the fully qualified server name and shortname is a shorter version. You need to have HADR set up with whatever comes first – shortname or longname.

db2nodes.cfg

Make sure that db2nodes.cfg (in $INSTHOME/sqllib) uses the same as hadr(longname or shortname), and that the hostname command returns the same of these two choices.

Longname vs. Shortname

It doesn’t really matter whether you pick the longname or the shortname, you just have to be 100% consistent. IP addresses don’t work. And the errors that db2haicu gives you are in no way helpful to determine you have a mismatch here. The errors are extremely generic.

db2cptsa

Sometimes on Fixpack, the TSA scripts don’t get updated as they should. To make sure you’ve got them up to date, you always should do the following:

Log in as root
cd path_where_fixpack_code_is/universal/db2/linuxamd64/install
OR
cd /opt/IBM/db2/V9.7/install

# ./db2cptsa
DBI1119I  The version of the DB2 High Availability (HA) scripts for the
      IBM Tivoli System Automation for Multiplatforms (SA MP) found in
      /usr/sbin/rsct/sapolicies/db2 is the same version as the version
      of the scripts on the current DB2 install media.

Explanation:

You need DB2 HA scripts to use SA MP with the DB2 HA feature.

SA MP and the DB2 HA feature being installed from the DB2 install media
require DB2 HA scripts with a version the same as or higher than the
version of the scripts also on the DB2 install media.

The version of the DB2 HA scripts currently installed the same as the
version of the scripts on the DB2 install media.

User response:

No action is required.

The above is what we expect to see. If you don’t see the above, you will most likely see it telling you that it has updated the scripts. If this returns an error, you need to investigate further. This must be successful before you can move on.

preprpnode

Ok, this is the last of the prepartion steps. preprpnode is a command that you run as root. I think that all it really does is trade public keys between the two servers. But it can also fail, and it must be run successfully before db2haicu will work. Here’s what it looks like. You don’t have to cd anywhere special to run it.

# preprpnode spp05db01r spp05db02r

Yep, that simple. It doesn’t return any output unless there is an error. The most common things to cause an error here are not having the TSA component installed (in which case you get a command not found), or to have a firewall either on the server or between the servers blocking traffic. The latter looks like this:

# preprpnode svp35db01b svp35db02b
/usr/sbin/rsct/bin/lsrsrc-api: 2612-022 A session could not be established with the RMC daemon on svp35db02b.
preprpnode: Unable to obtain the public key from svp35db02b.

If you see the above, db2haicu will also fail.

You need to check if iptables is running or look in your environment for another firewall. Assuming you are allowed to do so, you can check and stop iptables using this(as root):

# /etc/init.d/iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
3    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
5    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination
1    REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

You have new mail in /var/spool/mail/root

# /etc/init.d/iptables stop
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]

# /etc/init.d/iptables status
iptables: Firewall is not running.
#

Other Posts In This Series

This series consists of four posts:
Using TSA/db2haicu to automate failover – Part 1: The Preparation
Using TSA/db2haicu to automate failover – Part 2: How it looks if it goes smoothly
Using TSA/db2haicu to Automate Failover Part 3: Testing, Ways Setup can go Wrong and What to do.
“Using TSA/db2haicu to automate failover Part 4: Dealing with Problems After Setup

Search this blog on “TSA” for other posts on TSA issues and tips.

References:

Great whitepaper on db2haicu setup:  http://download.boulder.ibm.com/ibmdl/pub/software/dw/data/dm-0908hadrdb2haicu/HADR_db2haicu.pdf

14 comments… add one

  • How do we configure TSA for multiple instances on a server??

    can you provide sample xml file to configure multiple instances

    Reply
    • Unfortunately, I don’t have any experience to share here. My best guess is that you would run db2haicu as each instance owner. (after doing all your prep work) I have done multiple databases on one instance. I haven’t been using the xml method, though I’ve been thinking lately that I should try it.
      We try very hard to stick to a one database one instance one server setup for our production environments, and while I’ve been also doing HADR/TSA in test environments now and then, usually that is a client that is willing to have separate servers even for test to better mimic prod.

      Reply
  • I had also run into the similar installation problem on Red Hat 6.3 while installing the version of TSA that is part of the 9.7 FP5, base code cannot be installed because it does not work with Red Hat 6. The TSA team suggested me to install “3.2.1-TIV-ITSAMP-FP0003, Tivoli System Automation for Multiplatforms”. That worked perfectly for me and I was able to install the TSA sucessfully. One more thing, I had to use the license file, sam32.lic , that comes with the DB2 9.7 install media, since the link provided was a fixpack ( no license included ).

    Hope this information would be useful to someone.

    Reply
  • Hello, I ended with a partial TSA installation , neither ./installSAM nor ./uninstallSAM finish succesfully. The version is bundled with v97fp7 .Can you propose a solution, this is the uninstall log :

    ./uninstallSAM
    /usr/sbin/rsct/bin/lsrsrc-api: 2612-022 A session could not be established with the RMC daemon on “local_node”.
    uninstallSAM: Uninstalling System Automation on platform x86_64
    uninstallSAM: Uninstallin
    sam-3.1.0.6-10046
    uninstallSAM: Error: Failed with return-code 255 : sam-3.1.0.6-10046
    uninstallSAM: Any packages failed uninstallation. See details below
    uninstallSAM: Error: Failed with return-code 255 : sam-3.1.0.6-10046
    error: %postun(sam-3.1.0.6-10046.i386) scriptlet failed, exit status 1

    Uninstalling IBM Tivoli System Automation for Multiplatforms (SA MP) …
    0513-084 There were no records that matched your request.
    0513-084 There were no records that matched your request.
    0513-084 There were no records that matched your request.
    0513-071 The ctrmc Subsystem has been added.
    Cannot add ctrmc to init script structure
    Warning: An error was returned by rmcctrl while attempting to restart the rmc subsystem, exit code=0

    uninstallSAM: No uninstallation was performed

    Reply
    • My first thoughts are to go find a good Linux admin or to open a PMR. I’m sorry I don’t have any good answers on that for you – I have never seen it. If it was AIX, I’d be trying smit to remove it.

      Reply
  • Benefits of using this over setting up ACR(automatic client reroute)?

    Reply
    • ACR and TSA serve different purposes. ACR does not automate failover in any way – with ACR, you have to have someone manually issue the takeover commands for HADR to failover. The only place you really choose between ACR and TSA is when you choose whether to use a virtual IP or not. Using the virtual IP is easier because not all applications do well with caching an alternate server.

      Reply
  • Hi Ember,
    Thanks for this detailed blog. I faced the below issue while installing TSAMP and the prereqSAM failed with this error:

    prereqSAM: Using default prerequisite checking on the following rpm package ‘libstdc++4.-32bit’ ‘x86_64′
    prereqSAM: Error: The rpm package is installed, but the following part was not found ‘libstdc++46-32bit-4.6.9-0.11.38.x86_64
    libstdc++43-32bit-4.6.9-0.11.38.x86_64′ ‘/usr/lib/libstdc++.so.6′

    prereqSAM: Using default prerequisite checking on the following rpm package ‘pam-32bit’
    prereqSAM: rpm package and version installed ‘pam-32bit’

    1 missing package: libstdc++46-32bit

    I struggled for 2 days with it and found the following PMR with IBM:
    https://www.ibm.com/developerworks/community/forums/html/topic?id=e90040ce-2b74-4d80-9b8e-c8d5858adcf3

    So the fix is, install TSAMP with this command and it works fine:
    ./installSAM –noprereqcheck

    Reply
  • Hi,
    I’ve tried to startup de Cluster on DB2 10.1 with a couple of VM on SUSE 10
    Everthing works fins untl I run db2haicu on the Principal server. The last step

    I receive this error:
    2014-07-23-00.18.24.544869-240 E1380534G352 LEVEL: Error
    PID : 32289 TID : 2858981072 PROC : db2haicu
    INSTANCE: db2inst1 NODE : 000
    HOSTNAME: hadrprin
    FUNCTION: DB2 UDB, high avail services, sqlhaUIMain, probe:900
    MESSAGE : ECF=0x90000530=-1879046864=ECF_SQLHA_NOTINITED
    Library uninitialized error

    The only weird thing I found is that cluster finds 4 NICs when I only have 1 eth0 on each one and the lo.
    I also tried to do it with xml file and I get the same error

    Could be a kind of multipath on Linux?

    Best regards

    Reply
    • I have not seen that specific error.

      Reply
  • Hi,
    I have a question regarding the usage of Virtual IP. Concerning I have a HADR nodes in different subnets, I am unable to use the Virtual IP. Nevertheless, I need to connect to my HADR database from withing e.g. a script. Shall I for example catalog the DB2 HADR database on a machine running a script? If not, how shall I use the hadr database? thanks

    Reply
    • ACR(automatic client reroute) still works, and can be used in those situations. It is easy to use if you’re using JDBC type 2 or traditional db2 clients. In those situations, all you have to do is to define an alternate server at the DB2 server, and the clients will pick it up on their next database connection. Functionality should be built in to handle failovers. In fact, we still define an alternate server (of the virtual IP) even when using a virtual IP to get that retry functionality.

      If your application is using type 4 JDBC drivers or some other connection methodology, your app may have a place to specify the alternate server manually.

      Reply
  • I have a problem with additng virutal IP:
    “Cannot create a virtual IP because there are no networks defined for this cluster. Configure the networks before adding virtual IPs.”
    Do you happen to see that error?

    Reply
    • I have not seen that error. When you get it, have you defined at least one network card on each of the primary and the standby through db2haicu?

      Reply

Leave a Comment