Disable ability to add to a custom module on the Contact module only

Disable ability to add to a custom module on the Contact module only

Hi,

We have a custom module that we do want users to add to from the Accounts module but not the Contact module.

Is there a way to restrict the ability to add to the custom module when a user is in the Contact module only?