28/09/2016 – Outage on number routing Posted on September 28, 2016October 25, 2017 Our carrier partner for UK numbering suffered an outage this morning from approximately 10am to 12am. This affected all calls inbound calls to the UK numbers we provide and some outbound calls were affected. This is a large operator using carrier grade equipment, outages are rare. The cause is currently under investigation a full report will be issued and we will pass on the findings. We apologise for any inconvenience this has caused. We will post more details here when we have them. Update 29/09/2016 – Carrier Incident Update (note this refers to the carrier infrastructure, not our infrastructure) Total Length of outage – 2 hours 53 Minutes “At approximately 08:45 on 28/09/2016 there were reports of significant issues within our voice infrastructure which resulted in the loss of services across multiple voice platforms, including our own Service Desk telephony solution which impacted our ability to receive calls from Customers or to proactively call out. **Carrier** initiated troubleshooting activities and discovered that our Voice Session Border Controllers (SBC’s) could not perform the required lookup activities against one of our resilient lookup servers. This server was not immediately removed from the pool as it was still accepting network traffic and overall looked to be responsive. However, the process that controls the lookup server was not responding. As the lookup server was unavailable there was a resultant backlog of calls which caused our SBC’s to hit connection limits. This was escalated to the 3rd line systems team who upon extensive diagnosis discovered the failed process within the lookup cluster and restarted the failed process, at which point the backlog cleared and normal service resumed. We have escalated this software fault with our 3rd party software vendor and are working on monitoring improvements to allow us to detect and respond to this type of failure more efficiently in future. We continue to monitor the environment. The incident will not be set to resolved until such time as we are satisfied with the responses of our software vendor, and **Carrier** have conducted a full architectural review. Sincere apologies for the disruption this has caused you.” Share this:Click to share on Twitter (Opens in new window)Click to share on Facebook (Opens in new window)Click to share on Google+ (Opens in new window)Click to share on LinkedIn (Opens in new window)Click to share on Reddit (Opens in new window)Click to share on Tumblr (Opens in new window)Click to share on Pocket (Opens in new window)Click to share on Telegram (Opens in new window)Click to share on WhatsApp (Opens in new window)Share on Skype (Opens in new window)Click to print (Opens in new window)Like this:Like Loading... Related