Merging related records in deduplication issue

Merging related records in deduplication issue

We have a B2C setup.  Our business is largely referral-based, so we have a custom lookup field in Leads for contacts: "Referrer."  The problem arises when we deduplicate contacts and merge them.  All lead records with the contact lookup field for the merged record lose that contact field; it don't get merged along with activities, attachments, etc.

This seems like a significant issue.  Is there a way to do this cleanly without losing data?