Territories with Quotes and Access

Territories with Quotes and Access

We have accounts in many states, tracked by a field called "billing state".   How can we define that a specific user only have access to records in specific billing states?

For example, user named Fred, can access accounts and contacts in the state of NY & NJ.   I can get accounts working through territory rules, but how do i add contacts to that?  The contacts are all associated with the accounts.  For example, the contact does not have a billing state, but is associated with the City of ABC, which has a billing state of NY.

And, how can I set that user to only see quotes in their territory verses all quotes in all billing states?