Can I limit a client users access to specific milestones, task lists, documents, forums?

Can I limit a client users access to specific milestones, task lists, documents, forums?

I am wanting to be able to restrict different client users to specific parts of our projects. For most areas we can flag the area as either internal or external, which is great when we have tasks and forums that the client doesn't need to see. On occasions we have different teams at the client's end and sensitive information that one team doesn't need to see.

Is there a way to provide the permissions to the client users but restrict them?