This was a fun reminder that time synchronization, like reverse DNS lookup, is critical to infrastructure. tl;dr, verify ESXi host time synchronization before deploying PSC and VCSA.
While deploying the new vCenter Server 6.5 for our “production” demo lab, I ran into issues during Stage 2 of the vCenter Server Appliance (VCSA) installation with an external Platform Services Controller (PSC).
The time on this VMware vCenter Server Appliance is out of synchronization with the external Platform Services Controller with which you are trying to register by 34971.7 seconds.
Initial Diagnosis
My initial thought was to jump into the VCSA via SSH to begin troubleshooting the issue. After gaining shell access, I immediately launched a date command to view the system time. Ah ha! That is not the correct time, or time zone!
Connecting to the PSC via SSH, gaining shell access, and launching the same date command gives me a very different time–one that matches my workstation. Let’s look into the Appliance Management interface of the PSC (https://hostname.domainname:5840).
Everything looks fine with the Platform Services Controller. Upon attempting to enter the same Appliance Management interface for the VCSA, I am presented with the VCSA Installation wizard’s Stage 2 interface that continues to throw the error about time mismatch, so I am unable to get to the NTP settings of the VCSA …
Failed Troubleshooting Attempts
I will streamline this section, as I did not take notes during my flurry of redeployments and tests, but let’s just say I was getting desperate for answers … attempts include forcibly changing the time zone of the VCSA using SSH, manually setting the date and time, validating the ntpd service is running and with the correct NTP servers …
Post-Mortem
After enough troubleshooting, it began working when I had both the PSC and VCSA set to UTC time zone! At this point, I published a tweet and proceeded to head home … Several members of Twitter saw my message throughout the night and reached out to provide assistance; however, I wanted to spend more time testing varying deployment settings to find if the root cause was truly a bug or an issue with our demo lab …
After reviewing logs and testing deployments of PSC and VCSA using varying settings, NTP servers, and firewall policies I am confident I found the cause of the initial time mismatch: time settings of the host server.
Recap of what happened:
- PSC deployed, immediately receives time settings from host.
- Stage 2 of installation wizard for PSC sets up NTP time synchronization.
- VCSA deployed, immediately receives time settings from host.
- By this time, the PSC has synchronized its time with that of the NTP servers.
- Stage 2 of the installation wizard for the VCSA fails when connecting to the PSC because of time mismatch.
I followed the successful deployment with several more, independent PSC+VCSA installations to verify that everything works as expected when the ESXi host has correct time settings.
Leave a Reply