Vcenter Appliance Troubleshooting

VMware vCenter Server Appliance 5.0 GA with embedded DB2 database fails

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2006812

Interacting with db2:
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2004506&sliceId=1&docTypeID=DT_KB_1_1&dialogID=278854278&stateId=1%200%20278856044
The vCenter Server service fails to start after the service was stopped or restarted when using DB2 database

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021581

vcenter-2:/ # su - db2inst1
db2inst1@vcenter-2:~> db2start
SQL1026N The database manager is already active.
db2inst1@vcenter-2:~> db2ls

Install Path Level Fix Pack Special Install Number I nstall Date Installer UID
-------------------------------------------------------------------------------- -------------------------------------
/opt/db2/v9.7.2 9.7.0.2 2 Tu e Aug 23 17:54:41 2011 UTC 0
db2inst1@vcenter-2:~> db2ilist
db2inst1db2 => connect to VCDB
db2 => set schema vc



db2inst1@vcenter-2:~> db2 get database config for vcdb | grep LOG
Catalog cache size (4KB) (CATALOGCACHE_SZ) = 300
Log buffer size (4KB) (LOGBUFSZ) = 256
Log file size (4KB) (LOGFILSIZ) = 8192
Number of primary log files (LOGPRIMARY) = 16
Number of secondary log files (LOGSECOND) = 112
Changed path to log files (NEWLOGPATH) =
Path to log files = /storage/db/db2/home/ db2inst1/db2inst1/NODE0000/SQL00001/SQLOGDIR/
Overflow log path (OVERFLOWLOGPATH) =
Mirror log path (MIRRORLOGPATH) =
Block log on disk full (BLK_LOG_DSK_FUL) = NO
Block non logged operations (BLOCKNONLOGGED) = NO
Percent max primary log space by transaction (MAX_LOG) = 0
Num. of active log files for 1 active UOW(NUM_LOG_SPAN) = 0
Log retain for recovery enabled (LOGRETAIN) = OFF
First log archive method (LOGARCHMETH1) = OFF
Options for logarchmeth1 (LOGARCHOPT1) =
Second log archive method (LOGARCHMETH2) = OFF
Options for logarchmeth2 (LOGARCHOPT2) =
Log pages during index build (LOGINDEXBUILD) = OFF


db2 => list tables for all to list all tables
=====================================================================================
Error:(vpxa.log)
2012-01-16T15:14:43.269-08:00 [7FFFF3B09700 error 'Default'] [Vdb::LockRepositoryHelper] SQLExecDirect failed: 57011:-438:ODBC error: (57011) - [IBM][CLI Driver][DB2/LINUXX8664] SQL0438N Application raised error or warning with diagnostic text: "Error deleting from VPX_SESSIONL". SQLSTATE=57011
-->
2012-01-16T15:14:43.271-08:00 [7FFFF3B09700 error 'Default'] Unable to get exclusive access to vCenter repository. Please check if another vCenter instance is running against the same database schema.
2012-01-16T15:14:43.275-08:00 [7FFFF3B09700 error 'Default'] Unhandled exception

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021581

We followed the steps, it seems working now. But we need to monitor..


vcenter-2:/ # su - db2inst1
db2inst1@vcenter-2:~> db2start
SQL1026N The database manager is already active.
db2inst1@vcenter-2:~> db2ls

Install Path Level Fix Pack Special Install Number I nstall Date Installer UID
-------------------------------------------------------------------------------- -------------------------------------
/opt/db2/v9.7.2 9.7.0.2 2 Tu e Aug 23 17:54:41 2011 UTC 0
db2inst1@vcenter-2:~> db2ilist
db2inst1db2 => connect to VCDB
db2 => set schema vc
db2inst1@vcenter-2:~> db2 get database config for vcdb | grep LOG

Catalog cache size (4KB) (CATALOGCACHE_SZ) = 300
Log buffer size (4KB) (LOGBUFSZ) = 256
Log file size (4KB) (LOGFILSIZ) = 8192
Number of primary log files (LOGPRIMARY) = 16
Number of secondary log files (LOGSECOND) = 112
Changed path to log files (NEWLOGPATH) =

Path to log files = /storage/db/db2/home/ db2inst1/db2inst1/NODE0000/SQL00001/SQLOGDIR/
Overflow log path (OVERFLOWLOGPATH) =
Mirror log path (MIRRORLOGPATH) =
Block log on disk full (BLK_LOG_DSK_FUL) = NO
Block non logged operations (BLOCKNONLOGGED) = NO
Percent max primary log space by transaction (MAX_LOG) = 0
Num. of active log files for 1 active UOW(NUM_LOG_SPAN) = 0
Log retain for recovery enabled (LOGRETAIN) = OFF
First log archive method (LOGARCHMETH1) = OFF
Options for logarchmeth1 (LOGARCHOPT1) =
Second log archive method (LOGARCHMETH2) = OFF
Options for logarchmeth2 (LOGARCHOPT2) =
Log pages during index build (LOGINDEXBUILD) = OFF


db2 => select * from VPX_SESSIONLOCK

APPL_ID
--------------------------------------------------------------------------------------------------------------------------------
127.0.0.1.52263.120117000832


db2 => delete from VPX_SESSIONLOCK
SQL0100W No row was found for FETCH, UPDATE or DELETE; or the result of a
query is an empty table. SQLSTATE=02000

Verifying the group
db2inst1@vcenter-5:~> cat /etc/group |grep -i db
DBSYSMON:!:111:vc

db2inst1@vcenter-5:~> cat /etc/passwd|grep vc
vc:x:1004:100::/opt/db2/home//vc:/bin/false

Restarting the service.

vcenter-5:~ # service vmware-vpxd status

vmware-vpxd is stopped
tomcat is running

vcenter-5:~ # service vmware-vpxd start

Waiting for embedded DB2 database to startup: .success
Cleaning session lock table: success
Verifying EULA acceptance: success

Shutting down ldap-server..done
Starting ldap-server