I’ve had multiple occurencies lately in my lab where by I had to remediate passwords from SDDC Manager on different SDDC components in which I rotated the password manually direcly from the appliance.
One of these instances was on my Workspace One appliance, where I did reset both the System Administrator and Operator admin password directly from CLI (using /usr/sbin/hznAdminTool).
After such event, it is necessary to inform SDDC Manager that you changed the password manually, and you do that by running a Remediate task against the component (process documented here)
Unfortunately for me, the password Remediation task was failing. Tailing /var/log/vmware/vcf/operationsmanager/operationsmanager.log did however show an error unrelated to a password update taks:
2024-07-16T01:01:39.522+0000 DEBUG [vcf_om,0000000000000000,0000] [c.v.v.s.t.DynamicTrustManager,reactor-http-epoll-1] Error checking certificate chain EMAILADDRESS=unknown@vmware.com, CN=workspaceone.vcf-s1.vlabs.local, OU=Horizon-Workspace, O=VMware, L=Palo Alto, S
T=california, C=US for validity.
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:242)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:113)
at com.vmware.vcf.secure.truststore.DynamicTrustManager.checkServerTrusted(DynamicTrustManager.java:49)
at io.netty.handler.ssl.util.X509TrustManagerWrapper.checkServerTrusted(X509TrustManagerWrapper.java:69)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
... 36 common frames omitted
2024-07-16T01:01:39.526+0000 DEBUG [vcf_om,0000000000000000,0000] [c.v.v.s.t.DynamicTrustManager,reactor-http-epoll-1] Trying to reload trusted certificates and recheck chain EMAILADDRESS=unknown@vmware.com, CN=workspaceone.vcf-s1.vlabs.local, OU=Horizon-Workspace, O=VMware, L=Palo Alto, ST=california, C=US
2024-07-16T01:01:39.532+0000 DEBUG [vcf_om,6695c673f64f913906408b2acd9e4953,09a7] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,om-exec-24] Security config retrieved {"fipsMode":false}
Clearly, there is a problem with the WorkspaceOne certificate, even tho it was perfectly valid. It could also be that the certificate chain isn’t imported in the SDDC Manager trust store. In any case, I decided to regenerate a new certificate (self-signed) for Workspace One. Access your WSA appliance on the 8443 port and head over to Install SSL Certificates
Following the auto-generation, you will have a brand new self-signed certificate chain available to export into a .crt file. Copy the chain from the SSL Certificate Chain field into a file, in my case workspaceone-vcf-s1.certificateChainJul2024.crt
Next, we need to import this new certificate chain into the SDDC Manager trust stores. You can follow this VMware KB How to add/delete Custom CA Certificates to SDDC Manager and Common Services trust stores. For the completeness of this article I’m going to quickly paste the commands I run from SDDC Manager (as root).
1) Get the trust store key password
cat /etc/vmware/vcf/commonsvcs/trusted_certificates.key
t8~J_!o7b_K~F-^-BP
2) After copying the certificate file to /tmp inside SDDC Manager, we add the certificate chain to the trusted_certificates store
keytool -importcert -alias workspaceone-vcf-s1 -file /tmp/workspaceone-vcf-s1.certificateChainJul2024.crt -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store
3) we add the same certificate chain to the second trust store, called cacerts
keytool -importcert -alias workspaceone-vcf-s1 -file /tmp/workspaceone-vcf-s1.certificateChainJul2024.crt -keystore /etc/alternatives/jre/lib/security/cacerts --storepass changeit
4) Finally we check the certificate we previously added gets listed during a list operatation using keytool
keytool -list -v -alias workspaceone-vcf-s1 -keystore /etc/vmware/vcf/commonsvcs/trusted_certificates.store -storepass 't8~J_!o7b_K~F-^-BP'
5) Finally, we restart all the SDDC services
/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
Now, going back to SDDC Manager, I was able to perform a Remediate action against the admin account on WSA
I hope this was helpful to someone else other than me! 😉