Client and Vendor portal at the same time

Client and Vendor portal at the same time

We have companies that serve both as clients and vendors and they are linked. However, we have a problem giving them access to the vendor/client portal. If they accept the invitation from the client portal, when we try to invite them to the vendor portal we get the error message in attach. When we turn off the client portal and they accept the vendor invitation, we can't invite them to the client portal and we get the same message.  How can we solve this ?