Connectivity Issues
Incident Report for Vonage Video API
Postmortem

What happened

Between July 8th, 2020 18:54 pm UTC, and July 8th 20, 2020 20:43 UTC, we experienced an incident impacting some of our Enterprise signaling servers.

During this time, users on all client SDKs would not have been able to connect to a session running in the Enterprise environments. The time of disruption varies depending on the location of the data centers.

Users who had already joined the sessions in the Enterprise environments were not impacted by this incident. Ongoing and new sessions in Mumbai, India, and Tokyo, Japan Enterprise servers were not impacted by the incident.

The Standard environment was not impacted by this incident.‌

Causes

A bug in a server management configuration file resulted in the misconfiguration of several servers. As a result, it prevented the connectivity of the impacted servers.

When the root cause was identified, a fix was applied to all impacted Enterprise environments The service started to recover at 20:31 UTC and was completely recovered at 20:43 UTC.‌

Preventive Actions

  • Improve internal processes on the operation and maintenance of the platform.
  • Review and update the configuration and management processes and tools.
  • Review monitoring and alerting tools to improve the detection of intra-component communication issues.
  • Review and improve the incident management processes to reduce the time to announce incidents to the customers, once an incident has been identified.
Posted Jul 09, 2020 - 22:53 UTC

Resolved
This incident is now resolved. Please contact support@tokbox.com if you have any questions.
Posted Jul 08, 2020 - 23:52 UTC
Monitoring
We have implemented a fix that should have resolved the issue and we are monitoring the situation closely. We will update this status if anything changes or once it is fully resolved.
Posted Jul 08, 2020 - 20:54 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Jul 08, 2020 - 20:45 UTC
Update
We continue to investigate the connectivity issues in our video API platform. Customers may be experiencing an increased number of connection errors at this time.
Posted Jul 08, 2020 - 20:33 UTC
Update
We are currently experiencing issues with the video API platform. As a result, we are seeing an increase in connection errors, we are investigating.
Posted Jul 08, 2020 - 19:59 UTC
Investigating
We are currently experiencing connectivity issues. During this time, some services may not be performing optimally. If you have any questions please reach out to us at support@tokbox.com.
Posted Jul 08, 2020 - 19:58 UTC
This incident affected: Enterprise (Enterprise Video) and Standard (Standard Video).