Identification of Duplicate Records

What is the most efficient way to identify duplicate records for a large number of records?

It does not seem efficient to compare each record to all others because of the maximum number of executable statements per deluge action.

Note: I did not enable one record per user due to the type information being gathered.

Cheers,
John Whitney