Wingspan fails temporarily

Confusion hit campus the first day of the spring 2010 semester when students, faculty and staff discovered Citrus College’s student database system, WingSpan, was not working.

WingSpan was down from Feb. 17 at approximately 10 a.m. until Thursday around 4:30 p.m., when functionality was finally restored.

Linda Welz, chief information services officer, said officials know some of the symptoms of the mishap, but not the exact cause. They are still investigating it, she said.

“Two things happened at once, and it was kind of the perfect storm environment,” Welz said.

“We were having a huge number of users, but that was a normal occurrence at the peak of user times. But, at the same time, we had a memory failure. We don’t know why that occurred, but that’s why we ended up with the system crashing,” she said.

WingSpan is a SunGard banner system that runs on an Oracle database. Oracle produces the database, while SunGard produces the applications, and together they create WingSpan.

Leigh Buchwald, network central computing and telecommunications systems supervisor, and Welz, along with other staffers in the TeC Services department, took immediate steps to fix the problem.

“The first thing we did was try our common fixes, and that didn’t work,” Welz said. “We then called SunGard and put it in as a critical one problem, which means that they respond right away, which they did. They were very good at responding to us.”

SunGard directed Citrus staffers to work through several recovery steps, which also didn’t work. SunGard determined that the crash had to be the result of a failure at the database level, and TeC Services was referred to Oracle.

Oracle worked through their recovery process, and that approach also failed to fix the problem.

“We were unable to fix the failure with the recovery process,” Buchwald said, “so what we were able to do was use a restore process. We were able to restore the database to a point in time—I believe 15 minutes prior to the failure.”

During the time WingSpan was down, many students and faculty were affected by the failure.

Jeanne Hamilton, vice president of student services, said faculty could not pull up current roll sheets, no one could add or drop classes and students were unable to print their class schedules.

“I needed to drop a class. I wasn’t able to and I had to wait. It was a slight inconvenience,” business major Daisy Rios said.

Nursing major Angela Cheng, 20, said she was unable to purchase her parking permit online.

“After the system went back up, the lines were very long to get your parking permit, as well as your ASCC sticker,” Cheng said.

Citrus College has a 10-year license with SunGard for their services yearly. For 2009-2010, Citrus paid $175,000. Those campus services cover financial aid, Human Resources and the scanning system.

Citrus has a five-year license with Oracle. This year the college paid $120,000 for its services.

No other school that contracts with either company, such as Mt. San Antonio College and Pasadena City college, was affected, Welz said. It was entirely an internal issue, he said.

“We can’t really predict or manage memory failures, but we can take steps to increase the number of users that can be on at one time,” Welz said.  “If you put a lot of people in the system, it really slows, but we’ve been able to increase that so we won’t hit that at the same peak.”

Hamilton is already anticipating the next registration cycle.

“My advice to students is to print out your schedule immediately after registering,” Hamilton said. “You can’t wait until last minute when you know the system is at its peak use time.”