Home > SSL > Upgrading VC 4.x to 5.x while changing the hostname

Upgrading VC 4.x to 5.x while changing the hostname

March 22nd, 2014 Leave a comment Go to comments

Paul Meehan posted a question on twitter earlier this week.

While this was an upgrade from 4.x to 5.1 it would have been possible to simply go to 5.0 first with a new VM, use the migration tool and later change the IP address of that VM, but that would be boring, right?

Disclaimer: This is not a tested or officially approved procedure by VMware at any means, use at your own risks, don’t come to me complaining if everything breaks, DO A BACKUP!

So I quickly spun up a Windows Server 2008 R2 VM, installed vCenter Server 4.0 after quickly checking in the Solution Upgrade matrix that it is indeed supported to go directly from 4.0 U4 to 5.1U2.

Capture

So if we want to upgrade to 5.1 U2 now and change the host name there is a couple of issues using the VC default certificate.

1) Key length, the minimum requirement for vCenter 5.1 is 1024 bit with at least 2048 bit being the recommendation
2) The SAN field only has the old host name in there without an IP address, this certificate would not be valid for the new host system and could lead to many issues with the SSO integration

To move out of this situation you could blow away vCenter, create a certificate and prepopulate the install paths to reinstall against the existing database after a host name change, but I personally really dislike reinstalls, you only do them when you are out of options or an tight time constraints. Therefore the method I will follow here is a double certificate swap, rename of the system and then upgrade the system to vCenter 5.1U2.

As we are doing a double swap I did not care if the intermediate certificate that would be used for a couple of minutes was an CA signed or self signed certificate, I therefore simply created one using openssl (without Inventory Service this would mean on an external box as well).

I used the following openssl config:

[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS: vc.a.com, DNS: vcup.a.com

[ req_distinguished_name ]
countryName = IE
stateOrProvinceName = Cork
localityName = Cork
0.organizationName = fbuechsel.eu
organizationalUnitName = vCenterService
commonName = vc.a.com

This resulted in the following certificate.

Capture2

I replaced the certificate following the official guidelines which is a walk in the park for vCenter Server 4.0. After that I renamed the system and performed a reboot to see if vCenter Server still was able to start with its new certificate which indeed was the case. The next step was to change the name in the vCenter Server settings.

Capture3

A quick search through the registry for the old host name also did not yield any worrying results, so at this point in time I would consider the name change successful. DNS resolution was updated automatically due to my domain settings, depending on the situation this step would need to be done manually.

If you don’t care for the certificates to have additional information in them you could theoretically perform the upgrade to 5.1 U2 now and be done. For those out there actually having some sort of security guys chasing them with sticks if an old host name is still present in the SSL certificate we will go through the same document mentioned above once more and actually replace the certificates before upgrading to 5.1 U2 as it in fact is only a 2 step procedure in 4.0 but far more involved in 5.1.

To test the full impact I also added a host to this vCenter to see if hardware status, search and performance data was still working after the upgrade to 5.1.  As I did add the host after the name change in a real life scenario you might need to update the vpxa.cfg on the hosts to reflect the name change of your vCenter Server.

I noticed that a couple of things were broken on 4.0 since the name change, namely the Inventory Service and all web services relying on the Inventory Service. As we are blowing this instance away anyway during the upgrade process I decided to continue on, otherwise the best way to handle this situation probably would be the reinstall of the binaries to let the installer take care of all the tiny little details, I found at least one stale entry in the proxy.xml alone.

Log snippet from vws.log

[2014-03-22 10:00:22,442 Thread-43 ERROR ‘com.vmware.vim.health.impl.ComponentSpec’] Error while trying to login to https://VCUP.a.com:8443/vws/Login
java.net.UnknownHostException: VCUP.a.com
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:195)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:367)
at java.net.Socket.connect(Socket.java:529)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:562)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:404)
at com.sun.net.ssl.internal.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:121)
at com.vmware.vim.common.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:198)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at com.vmware.vim.security.authenticate.impl.ServiceConnectImpl.loginByCertificate(ServiceConnectImpl.java:213)
at com.vmware.vim.health.impl.ComponentSpec.setupClient(ComponentSpec.java:447)
at com.vmware.vim.health.impl.ComponentSpec.retrieveHealthFromUrl(ComponentSpec.java:298)
at com.vmware.vim.health.impl.ComponentSpec.retrieveHealth(ComponentSpec.java:255)
at com.vmware.vim.health.impl.HealthPollerImpl.retrieveHealthFromUrl(HealthPollerImpl.java:114)
at com.vmware.vim.health.impl.HealthPollerImpl.retrieveHealth(HealthPollerImpl.java:101)
at com.vmware.vim.health.impl.HealthPollerImpl.computeHealth(HealthPollerImpl.java:184)
at com.vmware.vim.health.impl.HealthPollerImpl.retrieveHealth(HealthPollerImpl.java:99)
at com.vmware.vim.health.impl.HealthPollerImpl.pollHealth(HealthPollerImpl.java:82)
at com.vmware.vim.health.impl.HealthPollerImpl.access$100(HealthPollerImpl.java:27)
at com.vmware.vim.health.impl.HealthPollerImpl$PollerThread.run(HealthPollerImpl.java:52)
at java.lang.Thread.run(Thread.java:637)

So to recap we now have a vCenter 4.0 Update 4 on which we changed the name from VCUP.a.com to VC.a.com with a CA signed certificate issues to VC.a.com. The Inventory Service and the Management Web Services seem to be broken after these changes and we will now have a look if the upgrade to vCenter Server 5.1U2 will fix the issue as the binaries and config files should be removed and readded by the installer.

Capture4

I chose the simple install option as I did not see any additional benefit in going through each installer by myself since we are asked for the the FQDN of the system anyway and I was too lazy to type go through the wizard 3 times.

Capture5

The upgrade to 5.1 took around as much time as all steps before combined … I then manually installed the Web Client and updated the Classic Client to have a closer look at the system status now. The first thing I always check is if we have a duplicate certificate in the lookupservice registration, easiest way to do that is by going into the vCenter Server SSL folder and look if a second certificate named sso is present, which is not the case, the installer successfully reused my already existing certificate without running into any issues.

Capture5

The second step is to connect to the vCenter Server using the Classic Client and look if the vCenter Service Status, Hardware Status, Storage Views, Performance Charts and Search are working. This indicates that all tomcat based services are healthy and that the Inventory Service is working as expected. The last and final step is to connect with the Web Client and basically check if I can see an inventory with a user that does have vCenter permissions and if changes done in the Web Client are reflected in the Classic Client and vice versa. If these checks pass I generally deem the system healthy and so far no customer disagreed 🙂

In my installation this was working for the most part, but I would still get 2 alerts in the vCenter Service Status overview. The hardware status also would not display an error anymore but be completely empty.

Capture6

Having a look again at the vCenter Server advanced settings quickly showed the issue related to the vCenter Service status. A quick look into the ADAM instance also confirmed the old entries in there.

Capture7

Editing these settings a following a vCenter Server restart the situation changed.

Capture8

The only thing left to do was getting the hardware status to work now. The cim-diag.log did not show any errors, so Host to VC communication does not seem to be the issue here. The CIM provider script cim-diagnostics.sh on the host also did not show any issues, so this problem must lie between vCenter Server and the Inventory Service. I therefore registered vCenter Server with the Inventory Service. That did not help. I then decided to make a different test and added the host to a different vCenter Server to see if it got a hardware status there as well to rule anything wrong with the host itself as I know this vCenter is set up correctly and can display the hardware status of my physical lab hosts.

And sure I was chasing a ghost, the hardware status also did not work in the known good vCenter Server… guess that is one of the draw backs with old virtualized ESXi versions 🙂

Only step I would change after going through all of this would probably be to set the advanced settings before the upgrade to 5.1 as this might also repair the search etc. in 4.0 already.

Apart from every recommendation out there it seems that it is indeed possible to change the host name even though quite some work is needed. I started this procedure 4 hours ago, if I calculate the write up out the whole procedure probably took me around 2.5 – 3 hours.

Categories: SSL Tags: