Unresponsiveness in Zoho Cliq

Time of post: 6:56 am PDT, 19 July 2023

Dear Cliq users,

We have encountered an unprecedented issue in our messaging servers. The details of the issue are as follows:

Impacted users : 1/4th of users in US DC (users who access cliq services through cliq.zoho.com)
Issue detection time : 4:22 AM PDT [4:52 pm IST], 19 July 2023
Issue : Unresponsive UI, Inaccessability of chats
Cause : Slow responsivenes in two of our messaging servers due to an unforeseen surge in requests.

We are currently working on resolving it with high priority. We will provide an update here in 30 mins. We apologize for any inconvenience caused.

Update 1 [8:39 am PDT, 19th July]: We have identified specific functions within the system that were causing issues in sending message. To ensure proper functionality, we have implemented restrictions on these functions. We are pleased to report that we have observed improvements in message delivery as a result. We will continue monitoring the situation and provide you with an update in the next 60 minutes. Thank you for your patience.

Update 2 [10:31 am PDT, 19th July]: We are pleased to inform you that the issue with our servers has been successfully resolved, and the Zoho Cliq services are now operating smoothly for all users. Our team is actively monitoring the system to ensure the absence of any further issues.

We sincerely apologize for any inconvenience this may have caused and extend our gratitude for your patience while we resolved the matter.

Update 3 [2: 53 am PDT, 20th July]: We encountered a recurring issue with one of our servers, which persisted from 11:41 pm PDT, 19th July, to 12:14 am PDT, 20th July. However, thanks to our diligent server monitoring team, the problem was swiftly addressed, and services have now been fully restored to their normal operation.

Update 4 [5:17 am PDT, 20th July]: We regret to inform you that we experienced a similar issue with one of our servers, which occurred between 4:02 am PDT, 20th July, and 5:01 am PDT, 20th July. The good news is that our dedicated team is actively working on resolving this matter, and as a result, Cliq services should now be stable for most users. However, we want to be transparent about the situation, and some users may still encounter partial slowness in message processing. Rest assured, our team is diligently addressing this remaining concern, and we will provide you with another update on the progress within the next hour.

Update 5 [6:59 am PDT, 20th July]: Our team has been diligently working on a patch to address the issue with our servers. As a result, Cliq services are in the process of being restored to normal functionality, and we expect everything to be up and running smoothly very soon. We sincerely apologize for any inconvenience this may have caused and appreciate your understanding and patience during this time.

Update 6 [7:19 am PDT, 20th July]: I am pleased to inform you that our team has successfully resolved the issue with our servers. For the past 20 minutes (starting from 7:00 am PDT, 20th July), our systems have remained stable. We are actively monitoring the servers to ensure that this incident doesn't recur in the future. As of now, Cliq services are expected to work normally for all users. You can proceed with your tasks and collaborations without any concerns.

Analysis report:

On July 19, 2023, at 3:49 AM PDT, there was a problem in our network which caused a drop in live connections between clients and servers. As a result of this, our clients were creating new connections, where as the backend servers were trying to repair the

existing connections. This caused a surge in two of our servers, which resulted in partial downtime.

During the partial downtime the followings functionalities would have been affected.

Chat related functionalities
Propagation of user status
Notifications

To fix this issue temporarily, some of the internal functionalities were throttled. This allowed us to control the surge of connections and process the load.

Counter measures :

To prevent this from happening again, the logic for repairing failed connections has been modified, so that no surges should be observed in any of our servers. We hope to push it to production as soon as possible.
We are also working on segregated processing of functionalities to prevent the availability issues due to heavy load.

Thank you once again for your understanding.

Best regards,

Poorvik

Zoho Desk Resources

Desk Community Learning Series
Digest
Functions
Meetups
Kbase
Resources
Glossary
Desk Marketplace
MVP Corner
Word of the Day

Sticky Posts
New in Zoho Chat: Threaded conversation at its finest best
Perform effective team communication in Zoho Chat with our new 'Reply' option. Converse and stay focussed on the parent conversation, rather than getting entangled in the web of several, never-ending sub threads. To reply to a certain message, all you need to do is hover to the left hand side of the message. Then, click on the three dots to open a pop up menu. Here, click on the Reply button and type the reply to the message in the compose box and press Enter. Voila, that was pretty simple.
Changes in Cliq iOS app notification due to iOS 13 and Xcode 11
Hello everyone! With the iOS 13 update, Apple has updated its policy on usage of VoIP push notifications. Over the past few months, we tried our best to provide a similar experience with the updated policy. Changes in iOS 13: With iOS 13, Apple mandates all VoIP push notifications to be reported to the CallKit framework as a new call. If a VoIP push notification is not reported to the CallKit within a designated time window, iOS will terminate the app. If enough VoIP push notifications are not
What's new in Zoho Cliq - June 2020 updates
Hello again, everyone! I'm back to share with you the recent feature improvements and updates that we've pulled together for enhancing your experience in Cliq. Here's what's new this June for you all in Cliq's web and iOS app! New on Cliq Web: Drag and drop files to a chat in your left side panel Now you can drag and drop attachments from your open conversation window to a specific chat or channel in the left side menu without opening it. Swift up actions and collaborate efficiently with Cliq's
4 Things You Should Do Once You Get Started with Cliq
Hey there, new user! You've successfully logged in and set up your organization and you're all set to start working. What's next? Buckle up because here are 4 essential things you need to do first in order to get the most out of your Cliq experience: 1. Invite your colleagues Now that you've set up your Cliq for business, you need to bring in all your employees, of course, because how else can you collaborate with them? To invite your colleagues to Cliq, head on over to the Admin Panel which
New in Zoho Chat : Search for contacts, files, links & conversations with the all new powerful 'Smart Search' bar.
With the newly revamped 'Smart Search' bar in Zoho Chat, we have made your search for contacts, chats, files and links super quick and easy using Search Quantifiers. Search for a contact or specific conversations using quantifiers, such as, from: @user_name - to find chats or channel conversations received from a specific user. to: @user_name - to find chats or channel conversations sent to a specific user. in: #channel_name - to find a particular instance in a channel. in: #chat_name - to find