Involuntary adding to contact list

Whenever I receive and/or send an email to someone that isn't one of my contacts, they automatically get added to my contact list. Is there any way to keep this from happening?