VMware Validated Design for NSX-T in a Workload Domain

NSX-T is the next major evolution in VMware’s Software Defined Networking. The VMware Validated Design team constantly looks for solutions that will bring value to our customers. As NSX-T is one of those solutions we wanted to bring it into the design in such a way as to not disrupt existing customers deployments, which are based on NSX for vSphere.

We looked at where customers were likely to use the NSX-T features today. This approach lead us to bringing NSX-T into the VMware Validated Design in a workload domain. This allows new and existing deployments to continue leveraging NSX for vSphere in the management and compute workload domains, taking advantage of features such as Cross-Site NSX, while bringing in NSX-T for specific compute workload domains. This gives our customers the flexibility to determine the version of NSX that best suits their needs for their workloads.

Today, I get to announce the fruits of the past few months of my labor. An Early Access look at NSX-T in a Workload Domain. This early access design covers the architecture and design elements for adding NSX-T to a new VMware Validated Design workload domain.

You can download the Early Access Design from our communities site, https://communities.vmware.com/thread/588236.

This is cross posted from the official VVD Blog.

Faithbridge, Project Samuel Zambia Mission Trip Video

One of my friends, Shawn Mansour, that was on the Mission Trip to Project Samuel in Zambia this past July put together this awesome video. It’s a great way for me to remember the trip and I hope it gives anyone who is thinking about doing one in the future a sense of how awesome the experience is for the community you’re serving and also for yourself.




Zambia Update #1 – 30 days till takeoff

Well we’re about 30 days away from heading off on our mission trip to Zambia South Africa. I just wanted to drop you all a quick note and fill you in on what’s been happening.

Since I last posted I’ve met with the group of guys, 7 of us total, a few times. They are a great bunch of guys and we’ve been meeting one on one in-between group meetings sharing our stories and just getting to know each other in general. I’m so excited the Lord has blessed me and pushed me in the direction of going on this trip. It’s going to be life changing, for me as much as the children I hope!

Those of you that have helped both through prayer and financially, Thank You! The prayers have been a huge help, when I committed to this mission I didn’t exactly know how I was going to pay for it. Through your prayers the Lord has provided his help. Those of you who were able to provide financial support you’ve helped me raise exactly half of the cost of the trip. As you can imagine a trip like this isn’t cheap but, the work I’m doing for the Lord is priceless and I can’t thank you all enough for your support.

I’ll post another update before we leave and expect a ton of pictures and a huge update when I return.

In His service,


My chance to serve!

For as long as I can remember I was what you could call a non-believer or agnostic. As some of you know that changed for me around 6 months ago. It’s been a great change in my life & one I’m very excited & passionate about.

I started attending church at Faithbridge & the people there have been nothing short of awesome! I’ve never felt more welcomed in a place as long as I can remember. One individual in particular has become a mentor and even better someone I can call a friend. So why am I telling my social media followers, colleagues, and friends this?

Continue reading My chance to serve!

Making a Move

For the past 18 months I’ve been a Senior Consultant with the Cloud Infrastructure Management Professional Services Organization here inside of VMware. It’s been a great year and half and I’ve grown both professionally and personally by having the opportunity to work with some great customers, some who pushed me and some who made me pull some hair out.

Back at the end of July I interviewed for a Technical Marketing position, I flew out to Palo Alto for 5 interviews packed into 4.5 hours, it was a great experience and everyone I interviewed with from product managers, the manager of the team, up to the VP of Technical Marketing was awesome. I felt I hit it off with everyone and that the interviews went really well. A few days later I received a call from the recruiter handling the position and was offered a transfer onto the team.

So as of 9/1 I moved onto the Technical Marketing team as a Senior Technical Marketing Manager – vCloud Suite. I’m extremely excited to jump in and get to work in my new role within VMware. While at VMworld last week I got to meet and hang out with the rest of the team and it was really good, I think we have a great group all of which seem like real team players which made me even more excited to start working with them.

The remainder of this week I’ll be transitioning the current Professional Services engagement I’ve been on since March to another consultant, once complete I’ll move full time into my new role.

I’m excited for what the future holds, watch this space and the vSphere Blog for a lot more blogging from me.

vCenter 5.1, Multisite SSO, Linked Mode, Custom SSL certificates, protected by vCenter HeartBeat. Part 3

I’ve been a bit distracted lately and wasn’t able to get this written up as soon as I would have liked, but in any case here is the final part in my three part series of setting vSphere 5.1 with vCenter Heartbeat and custom SSL certificates. Part one can be found here and part two here.

The most important part of this is to remember to install all the components and then go back and replace the certificates. If you install a component, say SSO, then replace its certificate then install another component, like vCenter, the Inventory Service or the Web Client, the install will hang when they try to register with the SSO server.

Like I mentioned in part 2 the KB Article on replacing these certificates is pretty spot on, but is a little cumbersome in areas and those are the areas I’m documenting here, otherewise I’ll just link to the corresponding KB.

So getting to it. We should have at least SSO, the vCenter Inventory Service, vCenter Server, and the Web Client installed.

We need to have our Root and Intermediate certificates for this process. More then likely you have these on your Windows server or workstation already, if not you can get them from your Root CA server. Assuming you do have these already installed:

  • Open mmc.exe
  • Add the certificates snap-in
  • Choose computer account when prompted
  • Navigate to Trusted Root Certificate Authorities and open the Certificates folder under it
  • Find your Root and any Intermediate certificates, one by one right click and choose Export under All Tasks
  • Choose Base-64 encoded X.509 (.cer)
  • Save each one, if you only have a Root with no Intermediates save it as Root64.cer
  • If you have Intermediates, once all are saved you need to combine them into one file, this can be done from the command line by running copy file1.cer+file2.cer Root64.cer
  • Copy the Root64.cer file to the certs folder on the machine you’re going to use openssl on

I would recommend using a Mac or a linux box for this, it is possible to do with Windows but the tools are built into to OS X and Linux and just makes the process a little easier. We’ll start by following this KB on creating some configuration files for each component. I create a certs directory in my OS X home folder and then inside of it create a sub-folder for each service (SSO, Inventory, vCenter, etc.). In each of those folders create the cfg file for that service, all cfg files are the same except for the servers name and the organitionalunitname, the organizationalunitname field specifies the service the cfg file is for, use the following values:

  • SSO = vCenterSSO
  • Inventory Service = vCenterInventoryService
  • vCenter = vCenterServer
  • Web Client = vCenterWebClient
  • Log Browser = vCenterLogBrowser
  • Update Manager = VMwareUpdateManager

Copy and paste the following to make your cfg files, replace the italicized text with the correct values for your installation.

[ req ]
 default_bits = 2048
 default_keyfile = rui.key
 distinguished_name = req_distinguished_name
 encrypt_key = no
 prompt = no
 string_mask = nombstr
 req_extensions = v3_req

[ v3_req ]
 basicConstraints = CA:FALSE
 keyUsage = digitalSignature, keyEncipherment, dataEncipherment
 extendedKeyUsage = serverAuth, clientAuth
 subjectAltName = DNS: ServerShortName, IP: ServerIPAddress, DNS: server.domain.com

[ req_distinguished_name ]
 countryName = Country
 stateOrProvinceName = State
 localityName = City
 0.organizationName = Company Name
 organizationalUnitName = Service
 commonName = server.domain.com

Now that we have all of our request templates (cfg files) we need to generate our Certificate Signing Request or CSR. Run the following two commands against each template.

openssl req -new -nodes -out ~/certs/serviceDirectory/rui.csr -keyout ~/certs/serviceDirectory/rui-orig.key -config ~/certs/serviceDirectory/service.cfg

openssl rsa -in ~/certs/serviceDirectory/rui-orig.key -out ~/certs/serviceDirectory/rui.key

In my case the commands for SSO would look like this:

openssl req -new -nodes -out ~/certs/sso/rui.csr -keyout ~/certs/sso/rui-orig.key -config ~/certs/sso/sso.cfg 

openssl rsa -in ~/certs/sso/rui-orig.key -out ~/certs/sso/rui.key 

In the sso folder you’ll see the files rui-orig.key, rui.csr, rui.key, sso.cfg. You’ll take the rui.csr file and use it to get your certificate.

When downloading your certificate from your CA you’ll want to download it in Base-64 format, save it to the service (sso, vCenter, etc..) folder as rui.crt.

Next we need to create a PKCS #12 file for each service, run the following command to generate it:

openssl pkcs12 -export -in ~/certs/serviceDirectory/rui.crt -inkey ~/certs/serviceDirectory/rui.key -certfile ~/certs/Root64.cer -name "rui" -passout pass:testpassword -out ~/certs/serviceDirectory/rui.pfx

In my case the command for SSO would look like this:

openssl pkcs12 -export -in ~/certs/sso/rui.crt -inkey ~/certs/sso/rui.key -certfile ~/certs/Root64.cer -name "rui" -passout pass:testpassword -out ~/certs/sso/rui.pfx 

It’s very important that you do not change the password, it must remain testpassword.

Lastly we need to create the Java Key Store for SSO, run the following commands to generate it:

keytool -v -importkeystore -srckeystore ~/certs/sso/rui.pfx -srcstoretype pkcs12 -srcstorepass testpassword -srcalias rui -destkeystore ~/certs/sso/root-trust.jks -deststoretype JKS -deststorepass testpassword -destkeypass testpassword

keytool -v -importcert -keystore ~/certs/sso/root-trust.jks -deststoretype JKS -storepass testpassword -keypass testpassword -file ~/certs/Root64.cer -alias root-ca

When you have run those two commands run the following to verify all certificates are in the store:

keytool -list -v -keystore ~/certs/sso/root-trust.jks

We now have all the certificates generated and are ready to replace the self signed certificates with our CA generated ones. Copy the folder under our certs folder to the corresponding server, for example the sso folder is copied to the SSO server.

In each of the following KB’s it’s assumed that you don’t have the Root and any Intermediate certificates installed, in most organizations this isn’t true and you can skip the steps about importing Root64.cer into your Trusted Root Certificate Authorities keystore, though verify your servers do have these certificates.

Replace the certificate for the SSO by following the steps in KB 2035011, I’ve found that since our Root64.cer already contains our Intermediate certificates steps 24-28 aren’t required. Also since we haven’t done the vSphere Web Client yet you can’t do step 29-37, instead run the cli command listed under step 37.

Next do the vCenter Inventory service by following the steps in KB 2035009.

Next do vCenter by following the steps in KB 2035005.

Next do the vSphere Web Client and Log Browser by following the steps in KB 2035010.

Lastly do vCenter Update Manager by following the steps in KB 2037581.

Since we used vCenter Heartbeat in our configuration if you want to change the certificate for it you can do so by following the steps in KB 2013041. The one thing with this is it states if you don intend to change the password you need to edit Server.xml. In vCenter HeartBeat 6.5 there is a randomly generated password that is in Server.xml, so you’ll need to open that file and note the password and use it in place of password.

And last but not least if you are planning on using SRM in this environment you must specify a CA generated certificate for it during install or your sites will not pair. KB 1008390 explains the requirements for the certificate but doesn’t give a good step by step, luckily this post in the VMware Communities sums it up pretty well.

And that sums up our install of vSphere 5.1 using vCenter Heartbeat and custom SSL certificates.

Script to use with SRM to clone a Domain Controller

Every time I go into a Site Recovery Manager engagement and we start talking about how the test functionality works I always get the question of how to make a domain controller available to the test network. I always say I have a script for that, so I thought I’d share it with anyone who might need it.

The script below is pretty simple, it just clones an existing domain controller, changes its network connection to the test network’s portgroup and then boots it up.

#Add PowerCLI Snapin
Add-Pssnapin VMware.VimAutomation.Core

#Connect to vCenter
connect-viserver <vCenter Name>

#Clone DC
new-vm -name <NewVMName> -datastore <DataStoreName> -VM <SourceVM> -ResourcePool <ClusterName>

#Change Portgroup
get-vm <NewVMName> | get-NetworkAdapter | Set-NetworkAdapter -NetworkName <IsolatedPortGroupName> -Confirm:$false

#Sleep for 1 Minute
start-sleep -s 60

#Power on DC
start-vm -VM <NewVMName>

<vCenter Name> is the name of the vCenter to connect to. If having SRM run this script make sure the service account SRM is running under has the required vCenter permissions.

<NewVMName> is the name you want to use for the cloned DC.

<DataStoreName> is the name of the datastore you want to store the clone in.

<SourceVM> is the DC you want to clone.

<ClusterName> is the cluster or resource pool you want to cloned DC to run in.

<IsolatedPortGroupName> is the name of the portgroup you want the cloned DC to use.

And that’s it, after running or having SRM run this short script you’ll have an isolated DC from your DR site connected to your isolated SRM test network. Just be sure and delete this DC when you’re done with your tests.

vCenter 4.1 to 5.0 Update 2 database upgrade failure

I’ve been working with a pretty big client the last couple weeks planning and upgrading around 20 vCenter 4.1 servers to 5.0 Update 2. This client runs all their databases on Oracle, currently version I checked out the Interoperability Matrixes and found that is supported for vCenter 5.0 Update 2. Matrix Pic

This client also uses the Oracle 11g client. I remember reading a Gotcha Chris Colotti blogged about so we obtained the client patch.

Now all should be good, but it’s not, upgrading the database failed. Looking through the VCDBUpgradeDatabase and the vpxd logs we’re seeing the same errors as KB 2010529 says were fixed in 5.0 Update 1, we’re installing Update 2 and hitting the bug. I made a comment on twitter last week about upgrading a large environment and dbuhelper.exe taking 22GB of RAM, turns out that’s not a good thing!

What’s happening is that when it crashes dbuhelper is updating all the url records in the database, a table at a time naturally, it does this by trying to load them all into memory in a contiguous block, when it can’t it crashes and the database upgrade fails.

The fix is quite easy but one that’s not documented anywhere I could find, so hopefully this helps someone out. First restore your 4.1 database from the backup you made before you started the upgrade then open up your vpxd.cfg file, on Windows 2008 this is stored in c:\programdata\VMware\VMware VirtualCenter find the <vpxd> section and paste this in somewhere before the closing </vpxd>:


This limits dbuhelper to only process 1200 records at a time which in turn keeps its memory usage low. Running this I never saw the RAM get above 25 MB where before I was seeing it range from 15-22 GB and more importantly the database was successfully upgraded.