So your trying to import a signed certificate that was created with the certificate signing request (CSR) you get an error in vCenter Orchestrator Configuration about the cert not being the correct format. If your using Microsoft Active Directory Certificate Services here are the exact steps:
- Export a certificate signing request from the VMware vCenter Orchestrator Configuration: Server Certificate area (goto http://vco-server:8282 then click Server Certificate on the left).  If this option isn’t displayed select the option to install a self signed certificate and then you will get the option to export a certificate signing request.
- Copy and paste the contents of the CSR file you downloaded from vCO Config area to your Cert Server web interface (http://CERTSERVER/certsrv).
- Select Web Server from the drop down and submit.
- Now ensure DER encoded is selected and download the certificate chain.
- Change the file extension on the file you just downloaded from .p7b to .csr
- You should now be able to upload it immediately using the “Import certificate signing request signed by CA” option inside the VMware vCenter Orchestrator Configuration: Server Certificate area (again goto http://vco-server:8282 and select Server Certificate on the left)
- You get a green bubble by Server Certificate and everything is happy.