Incident Report (Resolved)
Event Description: Intermittent failure to display call history and traces.
Event Start Time: 2021-03-16 12:55 AM EST
Event End Time: 2021-03-17 05:25 PM EST
RFO Issue Date: 2021-03-17
Affected Services
Call history and traces were intermittently unavailable on all cores. No disruption to call services occurred.
Event Summary
On March 16th, 2021 at 12:55 AM EST, our NOC received alerts that the service responsible for generating call detail records and displaying them in the portal was crashing across all cores. This affected the display of call history in the Manager Portal and access to call traces to assist in troubleshooting.
Root Cause Analysis
The service responsible for handling call detail records suffered a memory leakage resulting in persistent crashes. Our self-healing would restart the service which would result in the restoration of call history in the portal until the next crash.
Resolution
A review of the core dump files allowed us to isolate the cause of the issue and effect a solution via a patch. We also increased the allotted memory assigned to the CDR service.