The Incident:

On 22nd June at 7:25 MDT our monitoring system alerted due to increased response times from database clusters serving the chat system in the US region.

Root Cause:

On initial investigation, the CometChat team determined that the increased response time was due to a sudden and significant increase in the memory utilization on the affected clusters. From further investigation, the team discovered a bug in the load balancer application which caused all the read queries to be directed to the master instead of the read replicas. Due to the bug mentioned above all read queries were directed to the master causing exhaustion of buffer memory which in turn led to degraded performance for fetch operations.

Resolution:

As soon as the bug was identified, a patch for the same was deployed on the load balancer application. After the patch was applied the load balancer behaved as expected, the database cluster memory utilization returned to nominal and response times returned to within expected limits.