Contact list options from lookup field

Contact list options from lookup field

I have a number of lookup fields in my Potential records, and I'd like to restrict the contact lists that pop up when I access those lookup fields.  Currently, I need to attach the name of an Auditor to an organization's record, but in the Auditor lookup field, I get the default list which is contacts associated with that organization or "All Contacts." Is it possible to create a default list by field that would show me a list of contacts who have been identified as Auditors?  Similarly, when assigning a "Responsible Party" to a Potential, I'd like to restrict THAT contact list to only those inside my organization.  Am I missing an obvious way to do this?  

Andrew