On September 10th, while adding capacity to the enterprise environment, three of our new enterprise video servers experienced a network configuration issue. This caused sessions allocated to these three servers to experience connectivity issues. This incident started at 13:40 PST and was resolved by 14:50 PST.
While adding capacity to the enterprise environment, three of our new video servers experienced a network configuration issue. This was due to a specific port not being open to the internet. As a result, any session allocated to one of these three video servers would have experienced connectivity issues.
|Time Period (PST)||Major Incident Milestones|
|2019-09-10 13:48 PST||Issue reported to Support by customer.|
|2019-09-10 14:36 PST||Issue escalated to Engineering by Support team.|
|2019-09-10 14:50 PST||All impacted server configurations updated and issue resolved.|
|2019-09-10 15:21 PST||Incident posted on TokBox Status Page.|
|2019-09-10 16:33 PST||Incident closed on TokBox Status Page.|
Before adding these three enterprise video servers into rotation, we ensured that our internal spec tests were all successful. However, these tests were run from within the OpenTok infrastructure and did not emulate all possible customer configurations. In lieu of this, we will take adequate measures to broaden our testing infrastructure to emulate additional configurations.
As a result of our post-mortem investigation we have identified the following areas of improvement: