Mike's Virtualization Blog

vCenter 4.1 to 5.0 Update 2 database upgrade failure

I’ve been working with a pretty big client the last couple weeks planning and upgrading around 20 vCenter 4.1 servers to 5.0 Update 2. This client runs all their databases on Oracle, currently version 10.2.0.4. I checked out the Interoperability Matrixes and found that 10.2.0.4 is supported for vCenter 5.0 Update 2. Matrix Pic

This client also uses the Oracle 11g client. I remember reading a Gotcha Chris Colotti blogged about so we obtained the client patch.

Now all should be good, but it’s not, upgrading the database failed. Looking through the VCDBUpgradeDatabase and the vpxd logs we’re seeing the same errors as KB 2010529 says were fixed in 5.0 Update 1, we’re installing Update 2 and hitting the bug. I made a comment on twitter last week about upgrading a large environment and dbuhelper.exe taking 22GB of RAM, turns out that’s not a good thing!

What’s happening is that when it crashes dbuhelper is updating all the url records in the database, a table at a time naturally, it does this by trying to load them all into memory in a contiguous block, when it can’t it crashes and the database upgrade fails.

The fix is quite easy but one that’s not documented anywhere I could find, so hopefully this helps someone out. First restore your 4.1 database from the backup you made before you started the upgrade then open up your vpxd.cfg file, on Windows 2008 this is stored in c:\programdata\VMware\VMware VirtualCenter find the <vpxd> section and paste this in somewhere before the closing </vpxd>:

<upgradedb>
     <urlBatchSize>1200</urlBatchSize>
</upgradedb>

This limits dbuhelper to only process 1200 records at a time which in turn keeps its memory usage low. Running this I never saw the RAM get above 25 MB where before I was seeing it range from 15-22 GB and more importantly the database was successfully upgraded.