Monitor and CareDelivery Degraded Performance

Incident Report for Person Centred Software

Postmortem

On the 12th and 19th of January 2026 an incident with mCare impacted a subset of customers due to a SQL recompilation resulting in a degraded query plan impacting performance, this lead to excessive resource usage due to the query plan being 100x less efficient.

In the first case on the 12th our DevOps team identified the impacted query and attempted to clear the bad plan while also upscaling resources to take into account the increased usage. Scale up in this case took 4 hours due to the SQL resource being at 100% disk utilisation preventing efficient transfer to new resources. Once the scale up completed and the bad plan was cleared the servers were left at their higher resource levels to prevent recurrence.

On the 19th the same issue occurred despite the increased resources, at this time a known good query plan was implemented through a SQL Force Plan process which cleared the issues and was left in place while investigations took place.

Further detailed investigation identified a little used index that SQL was deciding to switch to using and resulting in the decreased performance, work was undertaken to remove the usage of that index leading to its deletion and ensuring that SQL would no longer create query plans based off it, this permanently resolved the issue and was rolled out to all shards over the week of the 26th January.

Posted Mar 06, 2026 - 16:13 GMT

Resolved

This incident has been resolved.
Posted Jan 12, 2026 - 22:10 GMT

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jan 12, 2026 - 21:59 GMT

Update

We are continuing to work on a fix for this issue.
Posted Jan 12, 2026 - 21:37 GMT

Update

We are continuing to work on a fix for this issue.
Posted Jan 12, 2026 - 20:36 GMT

Identified

The issue has been identified and a fix is being implemented.
Posted Jan 12, 2026 - 19:24 GMT

Investigating

We are currently investigating an issue affecting performance for data shard A.
Posted Jan 12, 2026 - 18:36 GMT
This incident affected: mCare (Care Delivery, Monitor, Relatives Gateway, Action Plans).