Claire Kruse (Open LMS Support)
Nov 15, 2022, 12:22 EST
Hi Brian,
Here is the summary from our admin team...
Open LMS experienced an outage with the underlying storage supporting the Davidson College solution bringing the site offline. The issue was resolved and the site was brought back online shortly after.
Start time of outage: 8:39 AM ET
End time of outage: 10:00 AM ET
Open LMS administrators were alerted to an issue involving the Davidson College system not responding to web requests and instead issuing a 504 Gateway Timeout from the load balancer supporting the solution. Upon investigation, our team found that the storage solution on which the web servers mount content volumes had become unavailable, preventing the application from accessing network cache stores, Moodle content, and other critical file paths. Further, systems were experiencing issues connecting to some AWS services via the hypervisor they were located on.
System administrators worked to quickly resolve the underlying issue, and redeployed the web nodes supporting Davidson College to bring the site back into operating status. Along with resolving the issue, we’ve deployed additional measures to the infrastructure configuration that will prevent this issue from occurring in the future.
The incident and its resolution did not cause any form of data loss and we can confirm that complete stack functionality has returned. We will be debriefing our team internally to discuss the incident and continue to work on delivering the best possible Moodle experience.