{"id":514,"date":"2015-01-16T13:54:06","date_gmt":"2015-01-16T21:54:06","guid":{"rendered":"http:\/\/www.wou.edu\/westernjournal\/?p=514"},"modified":"2015-01-16T13:54:06","modified_gmt":"2015-01-16T21:54:06","slug":"perfect-storm-incapacitates-campus-computer-network","status":"publish","type":"post","link":"https:\/\/wou.edu\/westernhowl\/perfect-storm-incapacitates-campus-computer-network\/","title":{"rendered":"\u2018Perfect storm\u2019 incapacitates campus computer network"},"content":{"rendered":"
By ALLISON OPSON-CLEMENT\r\nNews Editor<\/pre>\n
 <\/p>\n
Western\u2019s network was down from 9:30 a.m. to 2 p.m. Wednesday, Jan. 14, because of a router overload due to increased traffic, partly because of an external hacking attempt; the campus system was restored by University Computing Services (UCS) workers, and diagnostics are ongoing.<\/p>\n
\u201cThere\u2019s a whole bunch of \u2018don\u2019t know\u2019 right now,\u201d Bill Kernan, director of University Computing Services, said, adding that he and UCS are taking a forensic look into what happened.<\/p>\n
The focus was on getting Western\u2019s computers going again. Kernan said his entire team worked continuously, not stopping for lunch at all, and stayed clear until the end: many left only at 9 p.m. that night, after almost twelve hours of non-stop work.<\/p>\n
The network interruption was noted at 9:30 a.m., and Kernan and his team were contacted.<\/p>\n
They spent the next hour troubleshooting.<\/p>\n
\u201cThe typical issues weren\u2019t there,\u201d Kernan said.<\/p>\n
He started calling in help from off-site backup. By the end, UCS was on the phone with, off and on, up to three engineers simultaneously, all coordinating and working on the problem.<\/p>\n
\u201cI got as many resources thrown at it as I could,\u201d Kernan said.<\/p>\n
He called what happened a \u201cperfect storm.\u201d Two things happened nearly simultaneously, but either one alone could have been sufficient to bring down the network, because both resulted in traffic flow beyond what the main router on campus has had to deal with before.<\/p>\n
He said it was like two fire hoses of information: the streams were too strong, even alone, but together, it was tremendous.<\/p>\n
Increased usage overwhelmed the router. In addition to the increase of normal traffic, it also had net flow logs which were running. These help in diagnostics for determining the types and amounts of usage when that can help UCS.<\/p>\n
\u201cIt\u2019s not like we did something new recently,\u201d he said. \u201cNet flow shouldn\u2019t have done this to us.\u201d The whole network had been stable up until this incident, but in this case, the net flow logs happened to be the tipping point on the scales.<\/p>\n
The other thing that happened was that the main host server for the campus system experienced an attack from external sources. The hackers\u2019 IP addresses were traced back to computers in China.<\/p>\n
\u201cThey used the server as a launching pad for an attack against the network,\u201d Kernan said. The attacks took the router down via the compromised host server. He called this a malicious compromise of the system, a directed denial of service attack.<\/p>\n
No data was compromised, Kernan said. Only the one server was affected, and it is currently out commission.<\/p>\n
Kernan said they made the choice to get campus back up and running. The system was restored to operation by temporarily taking it out from behind the protective firewall. This was done with fewer than half of the most important of the 22 campus networks, and only between 1 p.m. and 8 p.m. on Wednesday.<\/p>\n
Without the fire wall, there was less stress on the router, and service resumed. During the time the fire wall was down, UCS decided that it was necessary to temporarily function without the net flow logs, and removed those to keep the system operational.<\/p>\n
At 8 p.m. the system was returned behind the firewall. There were no ill effects of operating without the firewall, Kernan said, partly because it was such a short time frame.<\/p>\n
UCS also attempted to reintegrate the compromised server, but within the two minutes that it was on, it was the target of 430,000 attacks. It is currently off the system.<\/p>\n
Western\u2019s system is up and running. A forensic investigation is taking place, according to Kernan, but this is only secondary to keeping the campus computer network functioning.<\/p>\n
\u201cIt was a complicated problem,\u201d Kernan said. He will be posting more details on his blog in the next couple of days as they learn more.<\/p>\n
For more information as it becomes available, visit wou.edu\/wp\/underthehood<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
By ALLISON OPSON-CLEMENT News Editor   Western\u2019s network was down from 9:30 a.m. to 2 p.m. Wednesday, Jan. 14, because of a router overload due to increased traffic, partly because of an external hacking attempt; the campus system was restored by University Computing Services (UCS) workers, and diagnostics are ongoing. \u201cThere\u2019s a whole bunch of […]<\/p>\n","protected":false},"author":367,"featured_media":190,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_seopress_robots_primary_cat":"","_seopress_titles_title":"","_seopress_titles_desc":"","_seopress_robots_index":"","_lmt_disableupdate":"","_lmt_disable":"","_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":"","_links_to":"","_links_to_target":""},"categories":[3],"tags":[],"class_list":["post-514","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news"],"modified_by":null,"_links":{"self":[{"href":"https:\/\/wou.edu\/westernhowl\/wp-json\/wp\/v2\/posts\/514","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wou.edu\/westernhowl\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wou.edu\/westernhowl\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wou.edu\/westernhowl\/wp-json\/wp\/v2\/users\/367"}],"replies":[{"embeddable":true,"href":"https:\/\/wou.edu\/westernhowl\/wp-json\/wp\/v2\/comments?post=514"}],"version-history":[{"count":0,"href":"https:\/\/wou.edu\/westernhowl\/wp-json\/wp\/v2\/posts\/514\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/wou.edu\/westernhowl\/wp-json\/wp\/v2\/media\/190"}],"wp:attachment":[{"href":"https:\/\/wou.edu\/westernhowl\/wp-json\/wp\/v2\/media?parent=514"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wou.edu\/westernhowl\/wp-json\/wp\/v2\/categories?post=514"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wou.edu\/westernhowl\/wp-json\/wp\/v2\/tags?post=514"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}