On April 28th 2019, at 4:54am (PST), it was reported that all HLS streaming was not working in both the standard and enterprise environments.
A 502 (bad gateway) error was received when the HLS broadcast streams were viewed. As a result, broadcast viewers were unable to view the HLS streams.
A fix was made and deployed on April 29th 2019, at 7:41am (PST), and no further reports have been identified since.
The incorrect cookbook was deployed, resulting in an incorrect configuration on the broadcaster machine. This was triggered by the assignment of a new block of IP addresses for a datacenter that required updating the server configuration.
A temporal solution was taken to rely on existing security groups on AWS machines.
Since then, our engineering team has updated the configurations. The engineering team is also actively working on improving the alerts in place to prevent this from occurring in the future.