Sort by modified date not working as expected

While going through recent contacts in my custom contacts view, sorted by modified date I noticed that adding a call note to a a contact does not move it to the top when sorting by date, where modifying a contact field such as phone number does. Contacts should be able to be sorted by last contact date.