28/09/2016 – Outage on number routing

Our carrier partner for UK numbering suffered an outage this morning from approximately 10am to 12am. This affected all calls inbound calls to the UK numbers we provide and some outbound calls were affected. This is a large operator using carrier grade equipment, outages are rare. The cause is currently under investigation a full report will be issued and we will pass on the findings.

We apologise for any inconvenience this has caused.
We will post more details here when we have them.

Update 29/09/2016 – Carrier Incident Update (note this refers to the carrier infrastructure, not our infrastructure)

Total Length of outage – 2 hours 53 Minutes

“At approximately 08:45 on 28/09/2016 there were reports of significant issues within our voice infrastructure which resulted in the loss of services across multiple voice platforms, including our own Service Desk telephony solution which impacted our ability to receive calls from Customers or to proactively call out.

**Carrier** initiated troubleshooting activities and discovered that our Voice Session Border Controllers (SBC’s) could not perform the required lookup activities against one of our resilient lookup servers.

This server was not immediately removed from the pool as it was still accepting network traffic and overall looked to be responsive. However, the process that controls the lookup server was not responding.

As the lookup server was unavailable there was a resultant backlog of calls which caused our SBC’s to hit connection limits.

This was escalated to the 3^rd line systems team who upon extensive diagnosis discovered the failed process within the lookup cluster and restarted the failed process, at which point the backlog cleared and normal service resumed.

We have escalated this software fault with our 3^rd party software vendor and are working on monitoring improvements to allow us to detect and respond to this type of failure more efficiently in future.

We continue to monitor the environment. The incident will not be set to resolved until such time as we are satisfied with the responses of our software vendor, and **Carrier** have conducted a full architectural review.

Sincere apologies for the disruption this has caused you.”