Core wireless network instability

Incident Report for Davidson College

Resolved

Several weeks have passed since technicians implemented any changes to the wireless system. Through a combination of technical measures, surveys, reports, and individual conversations, T&I can now say with a high degree of confidence that the root causes of this semester's wireless issues have been identified and resolved. T&I staff are developing plans to improve our wireless infrastructure and increase its overall resilience so that an outage of this magnitude never happens again. Thank you to all campus constituents who helped us with our troubleshooting or were simply patient with our staff as we worked to overcome these problems.
Posted Nov 09, 2023 - 12:24 EST

Update

The problematic wireless controller returned to a normal state at around 2:20 PM. The wireless connectivity issues that began at 1:45 PM should be resolved. We are working with our wireless vendor's emergency response team to determine the root cause.
Posted Oct 24, 2023 - 15:00 EDT

Update

We are experiencing a partial outage in our wireless controller infrastructure, starting at around 1:45 PM. Faculty, staff, and students may be unable to connect to the wireless network until the problem has been diagnosed and resolved.
Posted Oct 24, 2023 - 14:19 EDT

Update

T&I staff have continued to work to diagnose and resolve the wireless networking problems that have been reported on campus this semester. This has included extensive troubleshooting and consulting with Aruba, our vendor; our Aruba reseller/support partner; and wireless engineers at NC State (which had noted their own issues with eduroam and WiFi throughout the early weeks of the term.)

Based on this work, our collective team identified a potential root cause as a change in the Aruba code/firmware in June 2023 which affected how storage works on the virtual conductor servers that help manage the network. With this change, it is possible for log files to overwrite a system database that now shares file storage space, creating network instabilities.

During fall break, T&I staff and our consultants adjusted settings and resources on our virtual conductor environment, with work concluding late in the day on Tuesday, October 10. Since these changes completed, several community remembers who had reported network difficulties have informed us that their service has improved, and the T&I Support Center received no network service difficulty reports on Wednesday, October 11.

As we continue analyzing, it would be very helpful if those affected by these problems in recent months could let us know if their WiFi service has improved. Please take a moment to complete this survey, which has only two required questions: https://go.davidson.edu/wifisurvey

Thank you again for your patience during this frustrating outage.
Posted Oct 11, 2023 - 19:02 EDT

Update

The network team will be performing urgent maintenance on the wireless infrastructure and network core this evening. Multiple rolling outages of 15 minutes or more are possible.
Posted Oct 09, 2023 - 16:58 EDT

Monitoring

Core network maintenance was successful. No outages were detected. We are monitoring the network for signs of improvement.
Posted Sep 21, 2023 - 23:10 EDT

Identified

Further testing has revealed that Saturday's maintenance did not resolve the wireless issues that Davidson users are experiencing, because the changes we intended to make were not fully implemented. We are trying again tomorrow, Thursday, September 21 at 8:00 PM. While network changes of this nature do increase the probability of a widespread network outage, most users connected to the network during the maintenance window should experience nothing more than a short lapse in connectivity lasting no longer than 30 seconds.
Posted Sep 20, 2023 - 16:19 EDT

Monitoring

T&I engineers made the configuration changes intended to resolve this issue just prior 8 AM. The network appears to be in a healthy state. Engineers are testing the network to see if the changes have resolved the network transmission problems that are contributing to this incident.
Posted Sep 16, 2023 - 08:14 EDT

Identified

On Saturday, September 16 from 7:00 - 11:00 am, Davidson T&I will be performing required maintenance to the Davidson network. Users may experience a loss of internet service beginning at 7:00 am. While no widespread outage is expected, if issues should occur during the maintenance, network service interruptions could extend past the allotted time window. This outage will affect both wired and wireless networks, including eduroam, DavidsonDevice, DavidsonGaming, and all associated services that are connected to these networks. If you have questions about this outage and maintenance period, please contact our support team at ti@davidson.edu or 704-894-2900.
Posted Sep 15, 2023 - 16:13 EDT

Update

Engineers are getting closer to identifying the cause of the wireless issues. We will be doing some work in Watson this evening to gather more data.
Posted Sep 11, 2023 - 15:51 EDT

Investigating

Over the last two weeks, T&I has observed a sharp uptick in reports from Mac OS users regarding performance issues on the wireless network. Affected users report intermittent periods of very high latency or a complete loss of service, despite their device reporting full bars and a strong connection to the closest wireless access point. T&I has engaged an outside consultant to visit the campus and begin an advanced diagnostic and troubleshooting process. If you are experiencing persistent wireless issues on your device, no matter the type or operating system, we encourage you to contact ti@davidson.edu for assistance. Please provide as much detail as possible, including the behavior you are seeing on the device, and the location, date, and time that the issue occurred.
Posted Sep 07, 2023 - 09:17 EDT
This incident affected: Network & Internet Access (Wireless: eduroam, Wireless: DavidsonDevice, Wireless: DavidsonGuest, Wireless: DavidsonGaming).