Issues with HLS broadcast
Incident Report for TokBox
Postmortem

What happened

On April 28th 2019, at 4:54am (PST), it was reported that all HLS streaming was not working in both the standard and enterprise environments.

A 502 (bad gateway) error was received when the HLS broadcast streams were viewed. As a result, broadcast viewers were unable to view the HLS streams.

A fix was made and deployed on April 29th 2019, at 7:41am (PST), and no further reports have been identified since.

Causes

The incorrect cookbook was deployed, resulting in an incorrect configuration on the broadcaster machine. This was triggered by the assignment of a new block of IP addresses for a datacenter that required updating the server configuration.

Corrective Actions

A temporal solution was taken to rely on existing security groups on AWS machines.

Since then, our engineering team has updated the configurations. The engineering team is also actively working on improving the alerts in place to prevent this from occurring in the future.

Posted 15 days ago. May 06, 2019 - 16:28 PDT

Resolved
This incident has been resolved.
Posted 19 days ago. May 02, 2019 - 15:13 PDT
Update
This issue is fully resolved.
Please contact support@tokbox.com if you have any questions.
Posted 22 days ago. Apr 29, 2019 - 14:13 PDT
Monitoring
We have identified the root cause of this issue, and it should be resolved now. Users should be able to view the HLS broadcast streams at this time.
We are still running final tests and monitoring the platform to ensure full resolution.
Posted 22 days ago. Apr 29, 2019 - 09:44 PDT
Investigating
We are experiencing problems with HLS broadcast service. This impacts specifically HLS viewers.
We are working to resolve this issue.

RTMP/S broadcast and OpenTok sessions should not be affected by the issue.

We will update this status as soon as there are updates or when this is resolved. Please contact support@tokbox.com if you have any questions in the meantime
Posted 23 days ago. Apr 29, 2019 - 07:08 PDT
This incident affected: Standard (Standard Broadcast) and Enterprise (Enterprise Broadcast).