Restrict end users to only one help center

Restrict end users to only one help center

We have multiple help centers, but want to define which end users can access which portal. I can't figure out how to do that. It seems that they have access to all portals, even when inviting them just from one.