Contact gadget automatically adds contacts based on emails received

It's a tad annoying, as I get a lot of commercial email (as opposed to Junk) and have to spend time deleting contacts that I simply don't want.... any way around this?

Many tks,

Jim Van