How can I use a lookup field to lookup data in 2 different modules?

How can I use a lookup field to lookup data in 2 different modules?

I have two modules "contacts" and "agents".  I have a lookup field named "referred by" which could be either a contact from the "contacts" module or a contact in the "agents" module.

Is there a way for me to have the lookup field look in both modules?

Thanks!