Problem with DB2 Federation to an Informix Source when Applying a Fixpack

I can’t say that I work with federation on a daily basis. I have, however, set up federation for an Oracle data source from top to bottom, and support at least one system on a daily basis that uses federation to access Oracle, Informix, and DB2 for AS/400. This database also happens to be a BLU database, and as such, we want to stay really current on the fix pack. Currently with BLU being so new, there can be some drastic differences with just a fix pack.

The Problem

After applying Fix Pack 6 of 10.5, querying any Informix nickname received this error:

SQL1416N  Wrapper library "libdb2informixU.a" is not  compatible with the
release of DB2 installed at the federated server. The wrapper library is
compatible with  the following DB2 releases "10.5.4.0".  SQLSTATE=560CN

All of the other nicknames (Oracle and AS400) worked just fine. I called IBM DB2 support fairly quickly on this issue. After attempting to reinstall the fix pack about 4 times. Luckily this particular data was some that the client only uses about quarterly, so there was little urgency behind solving the problem. Between waiting on IBM DB2 support and waiting on the client to schedule the multiple outage windows needed, the PMR was opened on January 25th, and the problem was not finally solved until March 12th.

Timeline

Things I tried before calling IBM support:

  • Re-installing the fix pack (several times)
  • Running djxlinkInformix and djxlink from /opt/IBM/db2/V10.5/bin, as root, with DB2 down
  • Downloading the fix pack specific to federation, running db2_install -p rcon, to attempt to install Informix federation libraries from scratch.

Support’s fist suggestion included following the shut-down instructions from the IBM DB2 Knowledge Center and trying fix pack 7. While I think I’m pretty decent at stopping DB2 for fix packs and upgrades, I humored support and followed the shutdown instructions in order, and the client was happy to go with the most recent fix pack. When I had originally recommended fix pack 6, 7 was not out yet, and by this time it had been out for enough time that I was comfortable going with it. I didn’t think going with the most recent fix pack was likely to help, but I also didn’t think it would hurt anything. I tend to err on the side of accepting IBM support’s recommendations unless I think they will actually be harmful or if it’s a system-down situation and I think they’re a complete waste of time. These initial suggestions did not solve the problem. On one of the attempts at installing fixpack 7, I took an installation trace using:

./installFixPack -l <log-file-name> -t <trace-file-name>

This generated some additional data, which I sent off to IBM support.

Support also asked for the following information:

$ cd /opt/IBM/db2/V10.5/install
$ ./db2ls -q -b /opt/IBM/db2/V10.5

Install Path : /opt/IBM/db2/V10.5

Feature Response File ID             Level   Fix Pack   Feature Description
---------------------------------------------------------------------------------------------------------------------
BASE_CLIENT                        10.5.0.7          7   Base client support
JAVA_SUPPORT                       10.5.0.7          7   Java support
SQL_PROCEDURES                     10.5.0.7          7   SQL procedures
BASE_DB2_ENGINE                    10.5.0.7          7   Base server support
CONNECT_SUPPORT                    10.5.0.7          7   Connect support
DB2_DATA_SOURCE_SUPPORT            10.5.0.7          7   DB2 data source support
JDK                                10.5.0.7          7   IBM Software Development Kit (SDK) for Java(TM)
LDAP_EXPLOITATION                  10.5.0.7          7   DB2 LDAP support
INSTANCE_SETUP_SUPPORT             10.5.0.7          7   DB2 Instance Setup wizard
ACS                                10.5.0.7          7   Integrated Flash Copy Support
COMMUNICATION_SUPPORT_TCPIP        10.5.0.7          7   Communication support - TCP/IP
DB2_UPDATE_SERVICE                 10.5.0.7          7   DB2 Update Service
REPL_CLIENT                        10.5.0.7          7   Replication tools
DB2_SAMPLE_DATABASE                10.5.0.7          7   Sample database source
ODBC_DATA_SOURCE_SUPPORT           10.5.0.7          7   ODBC data source support
TERADATA_DATA_SOURCE_SUPPORT       10.5.0.7          7   Teradata data source support
JDBC_DATA_SOURCE_SUPPORT           10.5.0.7          7   JDBC data source support
ORACLE_DATA_SOURCE_SUPPORT         10.5.0.7          7   Oracle data source support
SQL_SERVER_DATA_SOURCE_SUPPORT     10.5.0.7          7   SQL Server data source support
CUSTOM_DATA_SOURCE_SUPPORT         10.5.0.7          7   Custom data source support
FIRST_STEPS                        10.5.0.7          7   First Steps
GUARDIUM_INST_MNGR_CLIENT          10.5.0.7          7   Guardium Installation Manager Client
SYBASE_DATA_SOURCE_SUPPORT         10.5.0.7          7   Sybase data source support

After this, support determined that INFORMIX_DATA_SOURCE_SUPPORT was not installed. This is odd, as we had been using it successfully for months. The fix for this was to manually untar and copy some files:

/bin/gunzip -c /db2home/db2inst1/fixpack7/universal/db2/aix/install/../../../db2/aix/FILES/INFORMIX_DATA_SOURCE_SUPPORT_10.5.0.7_aix64_ppc.tar.gz | (cd /opt/IBM/db2/V10.5; /bin/tar xvf -)

This required another outage as it had to be done while this instance was down, and then updates for the instance needed to be run (as root), specifically:

cd /opt/IBM/db2/V10.5/bin
./djxlinkInformix
cd /opt/IBM/db2/V10.5/instance
./db2iupdt -k db2inst1

This is what finally worked. I’m told that there’s a .metadata file where the installed DB2 componnents are listed. In this file, INFORMIX_DATA_SOURCE_SUPPORT was NOT listed. This meant that DB2 did not know to update it when a fix pack was installed. This seems to be chalked up to some sort of silent failure either when DB2 was first installed, when fix pack 4 was installed, or when someone added Informix support. None of these actions were performed by me. The error was limited to only the .metadata file, though, because we have been successfully using Informix support all year.

I can also now see INFORMIX_DATA_SOURCE_SUPPORT in the db2ls output, which I hope means this won’t occur the next time I go to apply a fix pack:

$ ./db2ls -q -b /opt/IBM/db2/V10.5

Install Path : /opt/IBM/db2/V10.5

Feature Response File ID             Level   Fix Pack   Feature Description
---------------------------------------------------------------------------------------------------------------------
BASE_CLIENT                        10.5.0.7          7   Base client support
JAVA_SUPPORT                       10.5.0.7          7   Java support
SQL_PROCEDURES                     10.5.0.7          7   SQL procedures
BASE_DB2_ENGINE                    10.5.0.7          7   Base server support
CONNECT_SUPPORT                    10.5.0.7          7   Connect support
DB2_DATA_SOURCE_SUPPORT            10.5.0.7          7   DB2 data source support
JDK                                10.5.0.7          7   IBM Software Development Kit (SDK) for Java(TM)
LDAP_EXPLOITATION                  10.5.0.7          7   DB2 LDAP support
INSTANCE_SETUP_SUPPORT             10.5.0.7          7   DB2 Instance Setup wizard
ACS                                10.5.0.7          7   Integrated Flash Copy Support
COMMUNICATION_SUPPORT_TCPIP        10.5.0.7          7   Communication support - TCP/IP
DB2_UPDATE_SERVICE                 10.5.0.7          7   DB2 Update Service
REPL_CLIENT                        10.5.0.7          7   Replication tools
DB2_SAMPLE_DATABASE                10.5.0.7          7   Sample database source
INFORMIX_DATA_SOURCE_SUPPORT       10.5.0.7          7   Informix data source support
ODBC_DATA_SOURCE_SUPPORT           10.5.0.7          7   ODBC data source support
TERADATA_DATA_SOURCE_SUPPORT       10.5.0.7          7   Teradata data source support
JDBC_DATA_SOURCE_SUPPORT           10.5.0.7          7   JDBC data source support
ORACLE_DATA_SOURCE_SUPPORT         10.5.0.7          7   Oracle data source support
SQL_SERVER_DATA_SOURCE_SUPPORT     10.5.0.7          7   SQL Server data source support
CUSTOM_DATA_SOURCE_SUPPORT         10.5.0.7          7   Custom data source support
FIRST_STEPS                        10.5.0.7          7   First Steps
GUARDIUM_INST_MNGR_CLIENT          10.5.0.7          7   Guardium Installation Manager Client
SYBASE_DATA_SOURCE_SUPPORT         10.5.0.7          7   Sybase data source support

IBM support tells me that simply running db2_install over again would have fixed this issue, too.

Summary

It took six weeks to get from problem to resolution. Part of that is that it was not critical to the client to have this functionality working, since this client only uses the Informix nicknames about once per quarter. The severity with IBM Support was somewhat lessened because of this. Also, at least two complete weeks of that was waiting to get an outage window that I needed. So the main question this whole thing raises for me is why a very obvious error was not generated either at the time of initial install of fix pack 4 or later with fix pack 6 that something was not right. This is the first time since the bad old days when special builds came as a few files to be carefully copied that I’ve had to mess with manually copying a file to the DB2 install directory. Somewhere along the line there was a silent failure here, and no one can tell me where or why. Hopefully if someone else runs into a similar issue, this blog entry will help them. On the positive side, I now have a new skill in my toolbox.

This is also a reminder to test all functionality including federation after any fix pack. Even if there are no error messages and if core functionality seems to work fine, there can be areas like this where something is wrong.

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *