Mike's Virtualization Blog

vCenter 4.1 to 5.0 Update 2 database upgrade failure

I’ve been working with a pretty big client the last couple weeks planning and upgrading around 20 vCenter 4.1 servers to 5.0 Update 2. This client runs all their databases on Oracle, currently version 10.2.0.4. I checked out the Interoperability Matrixes and found that 10.2.0.4 is supported for vCenter 5.0 Update 2. Matrix Pic

This client also uses the Oracle 11g client. I remember reading a Gotcha Chris Colotti blogged about so we obtained the client patch.

Now all should be good, but it’s not, upgrading the database failed. Looking through the VCDBUpgradeDatabase and the vpxd logs we’re seeing the same errors as KB 2010529 says were fixed in 5.0 Update 1, we’re installing Update 2 and hitting the bug. I made a comment on twitter last week about upgrading a large environment and dbuhelper.exe taking 22GB of RAM, turns out that’s not a good thing!

What’s happening is that when it crashes dbuhelper is updating all the url records in the database, a table at a time naturally, it does this by trying to load them all into memory in a contiguous block, when it can’t it crashes and the database upgrade fails.

The fix is quite easy but one that’s not documented anywhere I could find, so hopefully this helps someone out. First restore your 4.1 database from the backup you made before you started the upgrade then open up your vpxd.cfg file, on Windows 2008 this is stored in c:\programdata\VMware\VMware VirtualCenter find the <vpxd> section and paste this in somewhere before the closing </vpxd>:

<upgradedb>
     <urlBatchSize>1200</urlBatchSize>
</upgradedb>

This limits dbuhelper to only process 1200 records at a time which in turn keeps its memory usage low. Running this I never saw the RAM get above 25 MB where before I was seeing it range from 15-22 GB and more importantly the database was successfully upgraded.

Windows 2008 R2 Templates / Customization Specification / Local Administrator Password

I haven’t done much with templates in quite a while. Last week a client requested I assist with creating a Windows 2008 R2 template. We installed the base OS, did some minor configuration such as installing VMware Tools, enabling RDP and disabling the firewall using netsh, since this template would be used for various server types and we had no Internet access to patch the templates we stopped there. We shutdown the server and converted it to a template.

We then went and created a customization specification, the spec had an Administrator password set and had the server licensing set to per user, everything else was pretty straight forward.

To my surprise when creating a new VM from this template with this specification we couldn’t login, the Administrator password used in the template nor the one set in the specification worked.

When working with the client last week the only thing we got to work was a blank Administrator password in both the template and the customization specification. Not something I was happy about or recommended going to production with.

I came home and worked in my lab and had the exact same issues. After many tests scenarios the only thing that finally worked was enabling “Automatically logon as Administrator”.

2008R2 Template

After enabling this option having a password in the template and a different one in the customization specification the password in the customization specification worked.

I’m sure others have run into this but doing some searches, the only resolution I found was to use the blank passwords so I hope this helps someone who is seeing this same issue.