EU DC Partial Outage Resolved: A Detailed RCA

EU DC Partial Outage Resolved: A Detailed RCA

Incident Summary

Due to an overload on one of the nodes in the EU DC for Zoho Desk, the system was unable to handle the heavy load, causing a slowdown in requests and resulting in a partial outage for customers with data residing in that node.

On May 2, 8:26 AM, CEST , the incident was identified by our team. Our engineers determined the root cause and initiated efforts to address it by adjusting our system configurations. During this time, we began transferring high-traffic organizations to other nodes to minimize impact on the affected node. Although our secondary adjustments mitigated the issue and enabled the portals to load and work by May 2, 03:19 PM, CEST, it was not completely resolved.

After completing the scheduled movement of top-traffic generating organizations, we deployed a bug fix to prevent the system from holding connections for extended periods. Stability was fully restored on May 3, 12:40 PM, CEST. The incident and its history were also captured on the service availability page (status.zoho.eu).
 
Technical Breakdown

We identified that our system was experiencing slowness, leading to a significant number of users experiencing difficulty in connecting to Desk. This issue directly impacted the availability and performance of our service.
 
Further analysis using our monitoring system and log data indicated that the high traffic was legitimate and not due to a DDoS attack. Also, this incident was solely related to a load-surge issue, and there was no data loss or data impact caused by the partial outage. Our engineering team worked to optimize our system configuration to handle the traffic, but it did not completely resolve the issue. In addition, we moved top traffic generating organizations from the impacted node to minimize traffic, resulting in some improvement. 

Furthermore, we identified a code bug that held connections for an extended period of time and quickly deployed a live build to rectify the issue.
 
Timeline (in CEST)
 
May 2, 09:26 AM
Incident identified
May 2, 11:40 AM
Root cause identified - high number of connections
May 2, 12:15 PM
System configurations tuned
May 2, 01:51 PM
Started moving top-traffic generating orgs
May 2, 03:54 PM
Second-level system configuration tuning
May 3, 12:12 PM
Preparation of build to fix code bug began
May 3, 12:29 PM
Movement of top-traffic generating orgs completed
May 3, 02:40 PM
Bug fix build went live, stabilizing the system.
 
Future Preventive Measures to Avoid Recurrence of the Issue
  • Relocating selected organizations to other nodes to keep the connection count to the affected node at a minimum.
  • Monitoring the system connections proactively and re-balancing them as necessary.
  • Setting a lower connection threshold to receive early notifications when breached and take prompt action to avoid customer impact.
  • Incorporate a code-check configuration rule to prevent code from holding connections for extended periods of time before being shipped to production.

Regards,
Zoho Desk Team
    • Sticky Posts

    • Zoho Desk Partners with Microsoft's M365 Copilot for seamless customer service experiences

      Hello Zoho Desk users, We are happy to announce that Zoho Desk has partnered with Microsoft's M365 to empower customer service teams with enhanced capabilities and seamless experiences for agents. Microsoft announced their partnership during their keynote
    • WhatsApp pricing changes: Pay per message starting July 1, 2025

      Starting July 1, 2025, WhatsApp is shifting from conversation-based pricing to per-message billing. That means every business-initiated message you send will count. Not just the first one in a 24-hour window. Pricing updates on the WhatsApp Business Platform
    • Live Webinar - Work smarter with Zoho Desk and Zoho Workplace integration

      Hello customers! Zoho Desk and Zoho Workplace are coming together for a webinar on 14th May, 2024. Zoho Workplace is a suite of productivity apps for email, chat, docs, calls, and more at one single place. Zoho Desk is closely integrated with a few tools
    • Apple iOS 17 and iPadOS 17 updates for Zoho Desk users

      Hello Zoho Desk users! Apple recently announced the release of iOS 17 and iPad OS 17. These latest OS updates will help you stay productive and efficient, through interactive and seamless user experiences. Zoho Desk has incorporated the updates to help
    • Zoho Desk Cheat Sheet For The Year-End

      Check out these Zoho Desk best practices to end this year on a high and have a great one ahead! #1 Set Business (Holiday) Hours - If you have limited working hours, please make sure you restrict your business hours or set them as holidays for the coming days. Let your customers know when you will, and won't, be available. #2 Update the Annual Holiday List - Check the holidays for the new year and update the holiday schedule. Usually, holidays from the current year will be carried over for the next
    • Recent Topics

    • Native QuickBooks integration for Zoho CRM: Connecting sales and finance

      Greetings, I hope all of you are doing well. We're excited to announce Zoho CRM's integration with QuickBooks Web, which is designed to synchronize your CRM data with your QuickBooks accounting records and bridge the gap between sales and finance. This
    • Edit a previous reconciliation

      I realized that during my March bank reconciliation, I chose the wrong check to reconcile (they were for the same amount on the same date, I just chose the wrong check to reconcile). So now, the incorrect check is showing as un-reconciled. Is there any way I can edit a previous reconciliation (this is 7 months ago) so I can adjust the check that was reconciled? The amounts are exactly the same and it won't change my ending balance.
    • BUG: Related List Buttons with Client Script action now erroring

      There appears to have been a bug introduced over the last few days with Related List buttons that invoke a Client Script action. Button configuration: Configured Client Script: Results: The default loader is presented at the top of the page, and an error
    • Zoho Writer extension is now available in Zoho Books!

      The Zoho Writer extension is now available in Zoho Books. With this, you can design documents your way. Create custom templates with Zoho Writer in Zoho Books. Instantly generate multiple templates for invoices, estimates, and purchase orders. Easily
    • Paste Options don't work

      I've always wondered about this, as I've experienced this issue for quite some time now. Why don't the right-click Paste options work properly in Zoho Writer? I can use Ctrl + V without any issue, but if I right-click and use one of the Paste menu options,
    • What's New in Zoho Analytics - April 2026

      Hello Users! April brings a fresh set of updates and enhanced capabilities designed to make your analytics more intuitive and efficient. Explore What's New! Zia Insights in Dashboards We’re bringing the power of Zia Insights directly into dashboards.
    • Issue with Resume Parsing and Storage Limit in Zoho Recruit

      Hello Team, We are currently facing an issue with resume parsing in Zoho Recruit. While parsing resumes, we are receiving a message indicating that the storage is full. We would like to delete multiple old resumes from the system to free up storage space.
    • Introducing Custom Columns in Forecasts in Zoho CRM

      Release Plan: Enabling in Phased Manner, Enabled for JP DC Hello all, Forecasts in Zoho CRM help sales representatives, managers, and business stakeholders evaluate performance and plan future sales activities. While standard metrics such as Target, Achieved
    • Integrate QuickBooks with Bigin and streamline your sales and accounting!

      If your business relies on Bigin for customer management and QuickBooks for accounting and invoicing, this new integration is here to make your operations more efficient. By connecting these two platforms, you can now manage your CRM and financial processes
    • What's New in Zoho Billing | April 2026

      April 2026 brings a wide set of updates to Zoho Billing, from updated Payment Links layout and AI-powered billing workflows to smarter subscription quoting, better compliance tools for German Edition users, and more. Here's everything that's new. Connect
    • No Ability to Rename Record Template PDFs in SendMail Task

      As highlighted previously in this post, we still have to deal with the limitation of not being able to rename a record template when sent as a PDF using the SendMail Task. This creates unnecessary complexity for what should be a simple operation, and
    • Need Native Support for docx files in Zoho Writer

      Absolutely love Zoho Writer, but often need to share files by email with people who are in the Office ecosystem. Downloading a file as docx, then sending it by email, getting the comments back, converting it to Zoho format, editing it, then converting
    • Invalid value passed for line_item_category

      duplicating a previous used invoice and trying to save it (new invoice number / po number used) I keep encountering this error when trying to save the invoice Invalid value passed for line_item_category
    • What is the different between Zoho invoice and Zoho book

      Hi, both product do invoice and Zoho book having all function / feature Zoho invoice, please explain more, thanks
    • Mastering Zia Match Scores | Let's Talk Recruit

      Feeling overwhelmed by hundreds of resumes for every job? You’re not alone! Welcome back to Let’s Talk Recruit, where we break down Zoho Recruit’s features and hiring best practices into simple, actionable insights for recruiters. Imagine having an assistant
    • Option for - CSV Export from Pipeline Deals by Stage (Including Products, Companies, and Contacts)

      I would like to know when we will be able to export a simple CSV file from pipeline deals, with the option to select a specific stage within the pipeline. This export should include data for products, companies, and contacts, all in a single view. For
    • What is the difference between workflows, journeys, and blueprints?

      I semi-understand what they are individually but they all say they can be used to automate processes in your CRM. What makes these three different? What are the benefits and cons of using each?
    • Free webinar! Simplify hiring and HR workflows with Zoho Sign for Zoho People & Zoho Recruit

      Hello! Managing recruitment, onboarding, and employee paperwork doesn’t have to be complex or time-consuming. Discover how Zoho Sign, integrated with Zoho People and Zoho Recruit, helps you digitize and streamline your document workflows from hire to
    • Multiple Blueprints on different fields at the same time.

      It looks only 1 Blueprint can run at the same time, it makes sense for many Blueprints on the same field (Eg. Stage). But what about multiple Blueprints on "different" fields? the multiple options must be available. (Eg. Stage, Documents Status, Contract
    • Edit 'my' Notes only

      The permissions around Notes should be more granular, and allow to user to be able to edit the notes he created only. The edit Notes permission is useful as it allows the user to correct any mistakes or add information as needed. However, with this same
    • Need to make a specific canvas my default view for contacts

      Need to make a specific canvas my default view for contacts How do I do it?
    • Add Zia matching jobs on the main screen of candidates module

      It will be good if it is added in the main screen as a column so that we can quickly hover over and see if they match for any job openings. That will save from two additional clicks
    • Domain Disclaimer: A standardized footer for your entire organization

      Every email sent from an organization represents its identity externally. Most teams require consistent line of text at the bottom of outgoing messages. It can be a confidentiality notice, a legal statement, a compliance requirement, or a uniform sign-off.
    • Zia flags the deal as at risk - but leaves my customers figuring out the rest themselves

      I implement Zoho for many businesses. Team sizes vary, some clients have 3 reps, some have 40. But I keep hearing the same complaint across all of them and I figured it's worth raising here. Zia's deal scoring has genuinely improved over the past year.
    • How do we change system field names?

      I found some very old discussions, but looking for more recent. Very confused on mapping the addresses correctly, due to different names for some reason between. for example: leads: city, state, zip etc... as normal contacts: Mailing adddress & Other
    • Remove "Subject" as a required field on Quotes

      Currently, when you create a quote in CRM, the field "Subject" is mandatory. The properties of a system defined field cannot be edited which means we cannot de-select the mandatory requirement. A 'subject' on a quote is a little vague and not something
    • Adding Multiple Products (Package) to a Quote

      I've searched the forums and found several people asking this question, but never found an answer. Is ti possible to add multiple products to a quote at once, like a package deal? This seems like a very basic function of a CRM that does quotes but I can't
    • Unattended - Silent

      How can I hide the tray icon / pop up window during unattended remote access for silent unattended remote access?
    • What is the Potential field for in expense submissions?

      I'm trying out Zoho Expense in Zoho Project so I can record project expenses which aren't time related. On the expense form there is an option called Potential but I don't understand what this is for. When I click the dropdown it just shows the name of
    • Pasting Images in Zoho Desk ignores cursor location

      My team has reported an issue which started recently where when we paste an image into a new or existing reply or comment, the pasted image seems to ignore the current cursor location instead paste itself at the last character present in the reply/comment,
    • how do i add more than one google my business location?

      they are connected to one account, but while connecting social channels it makes me pick one location. I have 3 and growing.
    • Unable to switch existing AWS RDS connection to DataBridge after moving RDS behind VPN

      Hi everyone, I’m facing a problem with an existing Zoho Analytics setup and would like to know the best migration path. Originally, my Zoho Analytics connection to AWS MySQL RDS was configured using direct public access to the RDS endpoint. Everything
    • Hotmail

      I am sending an email to a hotmail, and this guy does not receive the email, either in his SPAM nor inbox. Can you help me? thanks!
    • Updating Sales orders on hold

      Surely updating irrelevant fields such as shipping date should be allowed when sales orders are awaiting back orders? Maybe the PO is going to be late arriving so we have to change the shipment date of the Sales order ! Not even allowed through the api - {"code":36014,"message":"Sales orders that have been shipped or on hold cannot be updated."}
    • How do I change the Subject header when I reply please, it contains Re which I want to remove.

      Hi Zohodesk, When a customer logs a call we have amended the Acknowledge on new Ticket template so the subject header has "Ticket Id" at the start of it.  When we reply the customer gets Re: and then the Id and I can't see a template for this? Can you
    • Zia Agent built in ChatKit UI does not render markdown

      Hi, You have a major shortcoming in the Zia Agent UI. The test UI that is embedded in agents.zoho.com allows you to test the agent has full support for rendering markdown, but your ChatKit UI does not have support for rendering markdown. If I embed it
    • Automated entries past the current month in a calendar report

      Hi all, I have an automation problem. I have a form which on successfull entry adds either 5 or 10 more of these entries with a slight change so our customers can see it throug a calendar report on the webiste. The entry put in manually shows up perfectly
    • [Bug] WebAuthn passkey registration blocked on rpIds with TLDs longer than 6 characters (.accountant, .technology, etc.) — isValidDomain regex too strict

      Hi, Filing on behalf of an enterprise customer where Zoho Vault is deployed across the company. The Chrome extension blocks WebAuthn passkey registration on legitimate sites whose Relying Party ID (rpId) has a TLD longer than 6 letters. This affects every
    • Get Files Associated to Data Template via API

      I have a data template with multiple files associated to it, and trying to write a Deluge script that will fetch files associated with this data template. I created the script below based on the WorkDrive API documentation, one request uses the data templates
    • ZOHO CRM User management or role

      I need guidance regarding Zoho CRM licensing and user management. I want to purchase one Zoho CRM license and create multiple team users under the same account with the following hierarchy: Super Admin User Manager User Executive Users (with limited access)
    • Next Page