At approximately 12:00 pm PST, some Enterprise media servers were put into rotation in the SJS Datacenter after an OS upgrade and the servers not configured correctly.
A manual sanity test was made before the servers were put into rotation, but failed to notice that audio was in fact not working.
At approximately, 3:30 pm PST, the misconfigured servers were taken out of rotation.
Due to a human error, an incorrect configuration (chef recipe) was used, which was not fully compatible with the binary deployed. This triggered a bug in which the misconfigured media server would reject the audio codec in the SDP. As a result, the clients would not publish any audio.
We fixed the server configuration and after new sessions were generated, worked as required.
The team is discussing improvements needed to ensure this won't happen again.