Please reconfigure the integration according to my blog post here: VMware Identity Manager using Azure AD as 3rd party Identity Provider - Horizon Tech Blog - VMware Blogs
It says AAD but everyting can be done the same with ADFS. Just how ADFS is configured is different.. The most important thing is how the authN methods are setup..