‘Unusual’ IT error affects multiple pages at USU
September 29th, 2015 Posted in USU LifeBy Brenna Kelly
bkellyjcom1130.blogspot.com
Several pages on the Utah State University server were down the weekend of Sept. 19 and 20 as a result of an unusual error.
The main index page was dead, affecting at least the counseling, student services and access center pages.
IT system administrator Blake Rich said www.usu.edu has a front-end load balancer through which pages from different areas are served. The issue came as a result of a failure between the load balancer and one of the back-end servers. As a result, the pages would not configure correctly.
“We have extensive monitoring for a variety of failure modes to try to ensure a solid web presence,” Rich said. “This particular issue was a strange one, which didn’t trigger any of our alerts.”
The IT Service Desk has a variety of alerts, from basic alerts if the web server stops responding to metrics that help see how the server is performing. Usually, if a system stops responding, a network monitor will check every minute and alert IT administrators via email or text. Additionally, Rich said sometimes students will call the service desk but that did not happen this weekend either.
The failure this weekend was the first time that this particular problem has been seen and it was related to the load balancer software put in place when the new USU template was launched in July. The template is a framework in which web content is managed. The university server supports three different templates but many departments are switching over to the newest version.
Student involvement director Linda Zimmerman was one of the first to notice the error Monday morning.
“A student called and made us aware that our website was down, and we called John Reynolds, who does our back-end on our website,” Zimmerman said. “In the meantime, the same student contacted me and let me know the whole system was down. Within two hours, it was up and running again.”
At the same time, student services system administrator Brent Colledge noticed something was off, as well.
“First thing in the morning, I was getting on to do a webpage update,” Colledge said. When he was checking his changes, he received an error code. “I started digging around and found it was on multiple pages, multiple sites… it was unusual.”
Other than the URL, the error was consistent — the same orange-yellow box appeared on the screen on all the different pages. Colledge said it was new but not remarkable. This was the first error he had seen since the new template was uploaded to the university server.
“It’s been really stable, I think – at least from what I’ve seen,” Colledge said. He hasn’t put any of the student services sites on the new template yet but will do so this year.
Rich said the IT Service Desk is adding additional checks to trigger alerts in case this problem happens again. He encourages anyone who sees an issue that persists more than five or 10 minutes to call the service desk at (435)797-HELP.
“We’re still evaluating what would be the best way to add metric and monitors to ensure that this doesn’t happen in the future,” Rich said.
-mdl/mr
Sorry, comments for this entry are closed at this time.