Monitor Degraded Performance

Incident Report for Person Centred Software

Postmortem

mCare Service outage on August 8th, 2025

Summary

On 08.08.2025 Person Centred Software experienced a critical incident that affected all mCare users. We recognise the disruption this caused and want to share a transparent overview of what happened, how we responded, and what we’re doing to prevent recurrence.

Root Cause

The root cause of this was a change in a shared library changing how configuration variables are read, the change was to support nested configuration. This change added an additional overhead to reading configuration which didn’t show up in low throughput and testing situations.

Timeline of Events

Resolution

To resolve the issue:

·         We restored the system by rolling back to a stable version.

·         We kept customers informed through our status page and in-app banner.

·         We have rescheduled the related release to include stronger risk controls.

Full service was restored by 16:00 and systems have remained stable since.

Customer Communication

 During the incident, we communicated via introducing the banner for those that were able to access the home page. The support team were also able to give updates and inform users that the issue was being looked in to and resolved.

We acknowledge the importance of timely, accurate updates and are reviewing how we can improve this further.  

Preventative Measures and Next Steps

To reduce the risk of recurrence, we are:

·         Investigating increased load testing in test environments to identify high throughput specific issues in advance

·         Ensure release planning explicitly flags and monitors high-risk changes.

Posted Aug 28, 2025 - 11:52 BST

Resolved

This incident has been resolved.
Posted Aug 08, 2025 - 16:03 BST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Aug 08, 2025 - 14:10 BST

Identified

The issue has been identified and a fix is being implemented.
Posted Aug 08, 2025 - 13:40 BST

Investigating

We are investigating reports of poor performance with Monitor for some customers on Web Shard C
Posted Aug 08, 2025 - 13:35 BST
This incident affected: mCare (Monitor).