Chat API

Details of the CometChat incident on December 30, 2019

Information

At CometChat, we have always emphasized on providing seamless customer experience and a robust chat solution.

At the same time, to achieve this, we regularly focus on the optimization of services we are providing, and our developers are continuously working towards improving our performance markers. On 30th December 2019 at 18:00 UTC, we released a solution for a database query optimization, by which we were aiming to reduce the latency for our APIs while fetching messages. This resulted in some unexpected behavior for some users- the messages fetched using API were returning incorrect results under certain conditions. These results, although incorrect, did NOT allow the user to see any messages not intended for them. They were ONLY able to see messages they’ve sent or received.

However, as soon as we noticed this issue, we moved quickly to rollback the entire release and everything was returned back to normal. The duration of this incident was approximately 90 minutes.

We completely understand how much this hurts our valuable customers. However, at the same time, we are assuring our customers that during this entire duration no customer data was compromised (and users were not able to see any messages not intended for them), and we will strive even harder starting today to make sure something like this never happens again, for same we are restructuring our release process to ensure such updates never reach production.