Incident: Videos not found
· 6 min read
Early on 10/4, teams began reporting that some videos on the website and Smartwork were not loading. After investigating, we discovered that the NGINX proxy pointing to Liquid Web was not resolving requests properly. This happened because the NGINX proxy pod restarted in the middle of the night, pulling a new image with code that made videos in Liquid Web unreachable. We rolled back to the previous image to restore service.
While we later determined that routing to most resources on Liquid Web were affected, we received no indication that any other service or asset type was affected, which is why this incident focuses on the videos.