Multiple Contacts attached to an event

I want to record that multiple Contacts attended an Event. I can do that by adding them as 'Invitees', but then when I look at the Contact record of each Invitee it does not associate them with the event under their Activities list. Am I missing something?