Today I was working in my lab, which already has an external harbor instance and some TKG clusters. I wanted to deploy workload management on one of my vSphere clusters, the NSX-T version which uses the ESXi hosts as worker nodes.
I’ve deployed this many times and it works great. Something I hadn’t done, nor could I find documented, was how to use an external harbor instance with this new Kubernetes cluster. The issue being my harbor instance is using a different Root CA from my vSphere environment, neither of which are public, so no trust.
I thought maybe the Pod VM’s that the ESXi host spins up to run pods would just ignore the cert error, not so lucky. Doing a kubectl describe pod showed it couldn’t pull the image due to an untrusted certificate.
In vSphere 7, you can add new Root CA’s to vCenter and subsequently ESXi hosts relatively easy. First go to Administration then Certificate Management and add the new Root CA certificate. Once added you can have each host pull it by going to the Configure tab on the host and navigating to System | Certificate and clicking the Refresh CA Certificates button. I’m sure you could script this for a large cluster, but I only have 4 hosts in this cluster so it was quicker to just go click the button 4 times.
Now I thought I was good to go but low and behold I was still getting the untrusted certificate error. After searching around and asking some colleagues if they’d ever tried this I found a page in our documention about what to do if you change the NSX-T certificate, which obviously wasn’t what I did, but the solution was simple enough, restart the wcp service on the vCenter Server. After wcp restarted life was good and the Pod VM’s were able to pull images from my harbor registry.
