SDDC Manager Password Rotation Fails due to SSL certificates problems

I’ve had multiple occurencies lately in my lab where by I had to remediate passwords from SDDC Manager on different SDDC components in which I rotated the password manually direcly from the appliance.

One of these instances was on my Workspace One appliance, where I did reset both the System Administrator and Operator admin password directly from CLI (using /usr/sbin/hznAdminTool).

After such event, it is necessary to inform SDDC Manager that you changed the password manually, and you do that by running a Remediate task against the component (process documented here)

Unfortunately for me, the password Remediation task was failing. Tailing /var/log/vmware/vcf/operationsmanager/operationsmanager.log did however show an error unrelated to a password update taks:

2024-07-16T01:01:39.522+0000 DEBUG [vcf_om,0000000000000000,0000] [c.v.v.s.t.DynamicTrustManager,reactor-http-epoll-1] Error checking certificate chain EMAILADDRESS=unknown@vmware.com, CN=workspaceone.vcf-s1.vlabs.local, OU=Horizon-Workspace, O=VMware, L=Palo Alto, S
T=california, C=US for validity.
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:242)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:113)
        at com.vmware.vcf.secure.truststore.DynamicTrustManager.checkServerTrusted(DynamicTrustManager.java:49)
        at io.netty.handler.ssl.util.X509TrustManagerWrapper.checkServerTrusted(X509TrustManagerWrapper.java:69)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
        ... 36 common frames omitted
2024-07-16T01:01:39.526+0000 DEBUG [vcf_om,0000000000000000,0000] [c.v.v.s.t.DynamicTrustManager,reactor-http-epoll-1] Trying to reload trusted certificates and recheck chain EMAILADDRESS=unknown@vmware.com, CN=workspaceone.vcf-s1.vlabs.local, OU=Horizon-Workspace, O=VMware, L=Palo Alto, ST=california, C=US
2024-07-16T01:01:39.532+0000 DEBUG [vcf_om,6695c673f64f913906408b2acd9e4953,09a7] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,om-exec-24] Security config retrieved {"fipsMode":false}

Clearly, there is a problem with the WorkspaceOne certificate, even tho it was perfectly valid. It could also be that the certificate chain isn’t imported in the SDDC Manager trust store. In any case, I decided to regenerate a new certificate (self-signed) for Workspace One. Access your WSA appliance on the 8443 port and head over to Install SSL Certificates

Following the auto-generation, you will have a brand new self-signed certificate chain available to export into a .crt file. Copy the chain from the SSL Certificate Chain field into a file, in my case workspaceone-vcf-s1.certificateChainJul2024.crt

Next, we need to import this new certificate chain into the SDDC Manager trust stores. You can follow this VMware KB How to add/delete Custom CA Certificates to SDDC Manager and Common Services trust stores. For the completeness of this article I’m going to quickly paste the commands I run from SDDC Manager (as root).

Now, going back to SDDC Manager, I was able to perform a Remediate action against the admin account on WSA

I hope this was helpful to someone else other than me! 😉