Certificates are always a pain point. So I hope this will help you to just replace the certificates of your vSphere ESXi hosts with one signed by your Enterprise CA.

Generate the Certificate Signing Request (CSR)

First you want to login to a system with OpenSSL installed. After that you need to create a OpenSSL configuration file that will meet the requirements defined by VMware.

[req]
default_bits       = 2048
prompt             = no
encrypt_key        = no
default_keyfile    = nuc-01.benslab.local.key
default_md         = sha256
req_extensions     = req_ext
distinguished_name = dn

[ dn ]
CN = nuc-01.benslab.local
C  = IE
ST = Cork
L  = Cork
O  = benslab
OU = vsphere

[ req_ext ]
subjectAltName       = @alt_names
subjectKeyIdentifier = hash
keyUsage             = digitalSignature, nonRepudiation, keyEncipherment

[ alt_names ]
DNS.1 = nuc-01.benslab.local
DNS.2 = 172.19.92.20
IP.1  = 172.19.92.20
nuc-01.benslab.local.cfg

You just need to change the values to match your environment. After creating all the configuration files for the hosts we can generate the private keys and the CSRs.

root@photon-01 [ ~/ssl ]# openssl req -new -config nuc-01.benslab.local.cfg -out nuc-01.benslab.local.csr

This will generate the private key and the CSR:

root@photon-01 [ ~/ssl ]# ls -la
total 20
drwxr-x--- 2 root root 4096 Sep 25 16:01 .
drwxr-x--- 5 root root 4096 Sep 25 16:01 ..
-rw-r----- 1 root root  529 Sep 25 16:00 nuc-01.benslab.local.cfg
-rw-r----- 1 root root 1167 Sep 25 16:01 nuc-01.benslab.local.csr
-rw-r----- 1 root root 1704 Sep 25 16:01 nuc-01.benslab.local.key
root@photon-01 [ ~/ssl ]#

Getting your Certificate

In this example I will be using Active Directory Certificate Services to sign the Certificate Request of my Host.

First you need to get the CSR of the system that was used to generate it.

root@photon-01 [ ~/ssl ]# cat nuc-01.benslab.local.csr
-----BEGIN CERTIFICATE REQUEST-----
MIIDJTCCAg0CAQAwbjEdMBsGA1UEAwwUbnVjLTAxLmJlbnNsYWIubG9jYWwxCzAJ
BgNVBAYTAklFMQ0wCwYDVQQIDARDb3JrMQ0wCwYDVQQHDARDb3JrMRAwDgYDVQQK
DAdiZW5zbGFiMRAwDgYDVQQLDAd2c3BoZXJlMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEA6cyesaB0+TvyK397pcFmQAlfpV0Up6mHJe2lDQ5rAr3iytiB
pRKCbOZz33k2lHwKR22gJjS1QIU3jnkjCW75mht+u07O3n1LuWVoKoj1f+ABtcVc
wvlP4oshRiqPn6rPklLyvRrfgO44YmkL6npN3EpbEaraFNftLQRaBMzDwTHsMZsa
JvChGPyebe/xrj3wZkIauJrVblDHn4WN9pfh3xOC76OtLfIaVcUXru0XtLsG5sbO
FElyimrAIP676ODxtASKfsgR+cuGJ6i8jHSh/pA6XAx9r6aIDPiuKCqdGXJdnY5h
JeEkD62UqskD9zMlVykt/6a74qQeAjiGiSlOBQIDAQABoHIwcAYJKoZIhvcNAQkO
MWMwYTAzBgNVHREELDAqghRudWMtMDEuYmVuc2xhYi5sb2NhbIIMMTcyLjE5Ljky
LjIwhwSsE1wUMB0GA1UdDgQWBBRyg+aFq5ZPfuuWr7/2OZOsi7sIwDALBgNVHQ8E
BAMCBeAwDQYJKoZIhvcNAQELBQADggEBAGPjFhInYGf3YpHju1XHIHb/xAMjg7av
3UjLHrOIyThmYBg4LTuBLS99Yn6UI8GYSceEWUyptyoTKr10sVtTOwdMiCBb0J54
+AC/3oaw8jAr1MFCGjKEw0i00dVLKj7Uk3FGtHoEJQI9bYI3WGQH0e6fjse6qex0
l6Bd47eJU1uj+nmVgheYOfWBnqnaU+Fql+rnVxGD/FK5E6miByAk/E27n/+2DDCP
e/5cI5gX1VmCYKln2mZ6EELbxop1fYghIZUiDXKYXyOZHwe/wm8iElVEAUHHlGuC
JJP5Vd5D/CgyO1TW/ywUC/2yRGu+lzTJ/rb0sJBbqxQnANQndkkb9gc=
-----END CERTIFICATE REQUEST-----
root@photon-01 [ ~/ssl ]#

After that you need to log in to your CAs web portal.

After selecting "Request a certificate" we have to select the Submit a certificate request option.

And provide our CSR from earlier.

When it comes to exporting we choose Base64 encoded and we want the full chain.

After the Download we open the resulting .p7b file

And export all the certificates in there, again using the Base64 option.

Changing the Host Certificate

With all the preparation out of the way you can now go and replace the certificate on the host.

Now let's copy the files to the host.

root@photon-01 [ ~/ssl ]# scp nuc-01.benslab.local.{cer,key} root@172.19.92.20:/
Password:
nuc-01.benslab.local.cer                           100% 1257     5.7MB/s   00:00
nuc-01.benslab.local.key                           100% 1704    10.8MB/s   00:00
root@photon-01 [ ~/ssl ]#

Before we begin let us create a backup of the current certificates just in case.

root@photon-01 [ ~/ssl ]# scp root@172.19.92.20:/etc/vmware/ssl/rui.* nuc-01.benslab.local-certificate-backup-2019-09-26/*
Password:
rui.bak                                            100%    0     0.0KB/s   00:00
rui.crt                                            100% 1501     6.7MB/s   00:00
rui.key                                            100% 1704     9.1MB/s   00:00
rui.log                                            100% 3284    11.5MB/s   00:00
root@photon-01 [ ~/ssl ]#

The easiest part is the actual replacement:

[root@nuc-01:~] mv nuc-01.benslab.local.key /etc/vmware/ssl/rui.key
[root@nuc-01:~] mv nuc-01.benslab.local.cer /etc/vmware/ssl/rui.crt

Now we just need to reboot the host, and that was it:

root@photon-01 [ ~ ]# openssl s_client -connect 172.19.92.20:443 < /dev/null 2>/dev/null | grep -e subject -e issuer
subject=/C=IE/ST=Cork/L=Cork/O=benslab/OU=vsphere/CN=nuc-01.benslab.local
issuer=/DC=local/DC=benslab/CN=Ben's Lab Root CA
root@photon-01 [ ~ ]#