Thanks for your continuous support to Zoho and our services.
We would like to apologize for a minor inconvenience caused to some of you, where you might have received duplicate copies of the same notification or alert emails triggered from our services.
Zoho TransMail is the transactional email service used by Zoho Services, to send transactional emails (notifications/ alerts/ service messages) to our users. These Zoho services use the APIs of TransMail to trigger emails via the Transmail servers. Please find the details of the incident and the rootcause below.
Transmail triggered duplicate copies of emails instead of sending a single email. We found that some emails were duplicated due to a technical issue in one of our servers. We identified the root cause of the issue and fixed it and also rolled out the fix to minimize the impact.
Issue duration: 1 hours 25 minutes (01:50 to 03:17PDT)
Around 100 K transactional emails/ notification emails were duplicated to the recipients including all the above services. We are deeply sorry for the trouble.
A total of around 1.6 M emails were duplicated and placed in the SMTP out queue. However we acted immediately and blocked the emails triggered from Desk to external servers. Around 1.5 M emails were blocked and the duplicate emails from Desk were blocked. When blocking, some normal emails were also blocked and removed from the queue to minimize the impact.
So some users may have missed certain notification emails due to this fix.
We were unable to perform updates in one of our servers due to a memory issue affecting a dependent application. So even after an email was sent, we were unable to mark the email as processed. Hence the emails were being retried again, until the error is corrected and the email is marked as sent.
So the emails were duplicated as they were repeatedly processed.
We apologize profusely for the inconvenience caused for you due to the duplication of notification emails from our services. We are also taking required precautionary steps to ensure that similar issues will not recur in future.