Moodle Journey

Posted on by

Late spring term, 2014, Moodle would periodically reach a load level that made it effectively unusable.  During the early part of summer term, the OS load levels would go from a reasonable level of 2 and would spike at 50 or 60.  During these high load events, both CPU utilization and memory utilization would spike, along with disk caching. From a user perspective the system was down.

In the early part of the diagnostics, Munin, a systems monitoring tool, collected detailed systems performance data from both the Moodle server and the MySQL back-end database.  It showed that there was memory paging occuring between RAM and disk.  After identifying the paging issue, the system admin increased memory(from 16G to 64G), CPUs(from 2 to 6) and increased the buffer size of the MySQL server.  This fixed the paging issue, but the same performance symptoms were observed.

Something drastic needed to occur to determine the cause of the underlying performance issue.  Moodle was split into two instances, one running fall, winter and spring, while the second ran summer only.  Performance issues were still observed.  At this point, a focus was placed on large embedded video files and large course backups, both of which caused excessively large load levels.  The large embedded video files were identified as a possible cause, moved out of Moodle and replaced with a link to the videos.  At the same time a memory leak was identified within Flowplayer.  Flowplayer was updated to the version used in Moodle 2.7 and the system admin put a cron job in place that frees unused memory every 3 – 5 minutes.  At this point, CPU, load average, memory and network utilization were greatly reduced.

What is the current status of Moodle?

  • Moodle 2.5, with upgraded Flowplayer has not performed poorly since week 28, 2014
  • Moodle 2.7 in currently being configured for fall term
  • The memory cleanup script will remain in place for Moodle 2.7
  • Load testing will be performed, utilizing LoadStorm.com and results will be posted here
  • The location of large video files will be dependent upon the results of load testing
  • Test performance utilizing SSD system drives

August 8, 2014 Update

  • Moodle 2.7, the lastest version, is available for Fall term course development. It is located at https://ginger.wou.edu — its final name will be https://moodle.wou.edu
  • All Moodle data on 2.7 have been moved to SSD drives (solid state disk)
  • A performance test was run, utilizing spinning disk followed by SSD drives.  The graph can be seen here:  (the large double spikes represent a 580 user load test utilizing spinning drives. The small bump in the afternoon is a 5,125 user load test utilizing solid state drives.)  What this means to you is, the system can handle 10 times the users at 1/20 the processing time.
  • A second Moodle web server was added, doubling the capacity.  Testing will commence shortly.
  • online3.wou.edu and online2.wou.edu are meant for archival purposes only.  New courses must be developed on moodle.wou.edu  For help exporting from old versions of Moodle into the 2.7 version of Moodle, call Elayne Kuletz.

August 28, 2014

Today, there are 4 web servers residing behind the load balancer.  The load balancer hands out initial web-requests utilizing a round-robin algorithm.  This means that if there are a total of 100 active users utilizing Moodle, each web-server will be handling 25 users.  To access the load-balanced Moodle, go to https://moodle.wou.edu  Each web-server can handle about 410 users and still maintain a good user experience.  This will give us a capacity of 1,240 concurrent users with a good user experience.  This past academic year, 250 concurrent users were the maximum number of users that were observed.  All load testing was performed while playing large videos.  As our load grows, additional web servers can be easily cloned from the existing web servers and placed behind the load balancer.  By the end of next week, we will have a consultant available to provide systems administration support.

The load-balanced schematic can be found here.

Final hardware configuration:

    • NetApp EF550 solid state storage array
    • Cisco UCS server platform
    • Cisco ACE 4710 load balancers
    • 10Gb network infrastructure at the core
    • Four load balanced web servers running on RedHat
    • One database server running on RedHat
    • Moodle version 2.7.1

Best / Worst Analysis

  • What is the best outcome that can occur, continuing with Moodle?
  • What is the worst outcome that can occur, continuing with Moodle?
  • What is the best outcome than can occur, by not continuing with Moodle?
  • What is the worst outcome that can occur, by not continuing with Moodle?

WOU Computing Infrastructure

Posted on by
Reply

Where to start…?  Maybe a summary from the ten thousand foot level, then expand out from there, yet keep a rein on going too far into details.  🙂

 

Summary:

WOU supports a variety of computer labs, smart classrooms, faculty and staff workstations and residential computing.  A variety of software packages are supported, both in computer labs, virtually and on faculty/staff workstations.  High speed internet is provided to campus as well as the residence halls through both wired and wireless connections.  NERO is our internet provider.  There is both a primary and redundant data center.  The network infrastructure includes routing, switching, firewalls, intrusion detection, network access control, wireless access points, etc.  A variety of enterprise applications are supported, including purchased, open source and developed.

Detail:

There are 25+ computer labs in a variety of locations on campus.  This includes Hamersly Library, APSC, Education, DeVolder Family Science Center, Natural Science, Creative Arts and Werner University Center, along with many others.  There are approximately 400 lab stations available for students.  This includes a mixture of PC’s, Mac’s and VDI (virtual) stations.  VDI can also be accessed remotely by a resident hall student from their own computer, providing access to a variety of software titles.  There are 40 Xerox 8870 color cube computers spread across all of these labs.

The lab workstations contain a variety of software including the Adobe Creative Suite, MS Office, Google Apps, SPSS, ARC GIS, Autodesk suite, Matlab, SAS, MSDN, Sophos and many others.  Not all applications are found at all locations.  All the applications contained on VDI, can be accessed remotely from anywhere, utilizing your WOU Pawprint account credentials.

There are 110 smart classrooms on campus.  Every room has one or two installed video projectors.  The instructor station has a computer, document camera, touch panel control system, AV interface, Blu-Ray DVD play and a sound system.  In addition some rooms contain an annotating or touch screen monitor, lecture capture system and IP-video conferencing.  Some of the larger rooms are dual-platform, PC, Mac or VDI.  Several lab-classrooms include the ability to display any of the lab stations on the projection screen.  About 20 of the classrooms have Apple-TV capability, giving you the ability to display the contents of an iOS device on the projection screen.

There are approximately 800 faculty and staff workstations on campus.  The current workstation life cycle is four years. There is a combination of PC’s, Mac’s and VDI stations, depending on need.  Access to Google Apps and MS-Office is provided on each station.  Other software is provided on an as-need basis.  In addition there are approximately 200 general-use stations.

The network backbone is a switched 10 Gb/sec network.  Network from the core to a building is typically two 10Gb/sec network ports that are aggregated, effectively providing 20 Gb/sec of bandwidth, plus redundancy.  In most cases, there is 1 Gb/s to the desktop.  The connection from WOU to the internet is a 500 Mb/sec connection.  NERO provides WOU internet services.  There is a redundant core project under way.  Both the campus and residence halls has wireless connectivity provided through 600 Cisco wireless access points, centrally managed by redundant wireless lan controllers.

The data center includes UPS power, backed up by a diesel generator.  The core server infrastructure resides on Cisco UCS Blade servers.  NetApp is our primary storage vendor, while SuperMicro provides secondary storage.  The data center includes both traditional spinning hard drives and solid-state drives.  The majority of the servers are virtualized.  There are 100 plus servers installed in the virtualized environment, including Windows Server 2008, 2012, RedHat, Oracle Linux and OpenIndiana.   More information on the redundant data center can be found here.

Examples of enterprise software run in the data center include, DNS, DHCP, Moodle course management system, Astra course and event scheduling, HVAC control system, Dimensional Data Warehouse, Portal, Oracle Enterprise Database, Oracle ODI, Cognos Connection and Framework manager, wouTV and many more.

Digital Media Productions resources include a studio with LED lighting, a TriCaster, 4 Sony HD video cameras, a Mac Pro editing station running Final Cut Pro X.  DMP broadcast its’ projects on the local community cable channel and wouTV as well as campus digital signage.

 

E-mail Security

Posted on by

E-mail is not secure.  E-mail can be intercepted on the path from sender to receiver.  If the e-mail is sent to/from an open list-server, Google can scan it and post the results in Google searches.  These are just a couple of the many ways your e-mail can be viewed.  If you don’t want the contents of your conversations showing up in the newspaper, don’t put it in an e-mail.

If you need to send HIPPA, FERPA or other confidential information using e-mail, then encrypt the e-mail.  In the subject line use the keyword encryption or #secure#.  This only works when sending e-mail to an e-mail address outside of @wou.edu.  Only the body of the e-mail will be encrypted.  The subject line will be in clear text, available to the public.

More on encryption can be found here.

 

Desktop Support

Posted on by

A new web site has been created, dedicated to providing information regarding Desktop Support Services.  Lori, Megan and John will each provide frequent updates to the Desktop Support site.

I would like to introduce you to John Rushing.  He is the latest addition to our desktop support team.  Most recently he was a Analyst Programmer / Support Technician for Dining Services and Resident Halls.  Previously John provided staff and student support services for Dining and the Resident Halls, including the Landers student lab.  Previous to coming to WOU in 2005, John was a Programming Engineer at McAfee.com, providing virus programming support.  John started his first position at WOU in 1997.  He provided hardware support for the computer labs, managing student workers and ordering of supplies and inventory.  John was in this position through 1999.  John brings a vast level of knowledge to this position.  John will be combining both his Desktop Support experience and programming experience to automate many of the current manual processes.

Megan provides both printer and desktop support.  As noted in a previous entry, she is a certified Xerox printer technician.  Megan will continue to provide services in all areas of desktop and network support services.  Megan is managing the Service Request and Computer Lab student employees.

Lori continues to provide Desktop Services, Network Access Control(NAC) services and Windows Server administration.  Lori also manages the Desktop Support student employees.  Lori is responsible for re-imaging all the labs over the summer, preparing them for fall term classes.  Lori is Dell hardware certified.

Each technician will have primary responsibility for specific buildings.  After analyzing Desktop support data, each technician will be assigned a specific area of campus.  The assignment process will be completed in August.

Each technician will be responsible for the following tasks and more…

  • imaging new computers for faculty, staff and student computer labs
  • re-imaging student computer labs
  • configuring and monitoring Windows update server
  • configuring and monitoring Sophos anti-virus server
  • researching and deploying OSX and iOS deployment server
  • updating Windows applications
  • research and deploy new technology
  • maintaining Network Access Control (NAC appliance)
  • support MathTest, CompTest, IME, SOAR, MelBrown, etc.
  • provide useful information on the Desktop Support blog site
  • provide web-site with recent top 10 service requests, with solutions
  • face-to-face communication with faculty/staff on a daily basis
  • follow-up with faculty/staff after tasks have been resolved
  • provide training for lab attendants
  • backup Nathan when he is not available for classroom support tasks
  • support all aspects of computer replacement deployments, including imaging, bios changes, software installs, old data migration, installation, etc.
  • support printer deployments, maintenance and repair
  • maintain inventory
  • support PDR process, including DoD wipe, storage and disposal
  • interview and hire student employees, including Desktop Support, Service Request Desk and Student Computer Labs
  • maintain Active Directory (AD), including changes, moves and adds
  • maintain folder permissions in LDAP
  • research, recommend and implement streamlining and optimization of services processes

A summary of what is new:

  1. Move John into Desktop Support services, increasing support by one FTE. (equipment systems specialist)
  2. Distributed primary responsibility of new computer deployments to all three technicians
  3. All three technicians will be able to perform the full complement of duties required of an Equipment Systems Specialist
  4. Each technician will provide primary support for a specific area of campus
  5. Provide users with additional information through Desktop Support web site

Purchasing / Inventory

Posted on by

Tricia, Joanie and Sarah (a student employee), make up the inventory and purchasing team.

All IT purchasing, as defined in the WOU Purchasing policy, is purchased centrally.  Please submit your purchase request/price request, for either software or hardware, through the purchase request system.

Tricia is the entry point for all purchases.  She would be glad to answer your questions or help you configure your next purchase.  She is available by e-mail at flahert@wou.edu

When there is contract language associated with the purchase, I will work with Darin Silbernagel, Director of Business Services, to modify/confirm an appropriate contract.

Joanie sends the order out for bid.  Three bids are required for all orders over $5,000.  For those vendors that we have not used in the past, a Substitute W-9 form must be filled out.  Lucinda in the Business Office, then enters the new vendor into Banner.  Joanie creates a Purchase Order in Banner.  She will use the funding source you specified in the Purchase Request system.

Sarah un-boxes all the equipment when it arrives, creates and affixes inventory labels and updates the inventory system.

Tricia assigns delivery or installation of software to a technician, using the Service Request system.

Printers-R-Us

Posted on by

Megan Eichler is a Xerox certified printer technician.  Don’t hesitate to call Megan regarding any of your printer needs.  She maintains HP printers also. 🙂  You can often find her at her desk with a printer completely pulled apart, replacing a failed part in the core of the printer.  With the Xerox certification, Megan has back-line support with the Xerox technicians and the parts department.

Xerox printers that utilize PagePak support, come with an endless supply of ink and consumer replaceable parts.  The PagePak contract includes a $0.0108 per page charge for mono and $0.0355 per page charge for color + a base charge of $13/month.

megan

Megan repairing Xerox printer

Redundant Data Center Overview

Posted on by

DeVolder Family Science Center houses a small redundant data center.  This location contains servers, storage and core network gear.

Both the main data center in ITC006 and the redundant data center in DeVolder utilize the NetApp SnapMirror technology.  With this product enabled, data that is written in one location, is then replicated at the redundant site.  Currently, only core services such as DNS, DHCP, core web server and A/D are duplicated at the redundant site.  Over time additional services will be added to the redundant site, including such applications as Moodle.

In addition, there is cheaper storage in DeVolder and Ackerman Hall that is used as secondary storage.  All primary storage is transferred to secondary disk storage, a minimum of one time per day.

redundentPwr

Redundant power

storageAndBladeSrv

NetApp storage and Cisco UCS blade servers

redundentCore

Redundant core

Creativity — Utilizing 3-D Printing

Posted on by

Sharp bends in fiber optic cables can negatively impact the data flowing across the fiber.  After years of fiber hanging from a network device, the cable can become deformed.  Dave McEvilly (lead windows sys admin) designed and printed a cable management structure utilizing Vectorworks Designer and a Makerbot Replicator 2.

The cost of materials for printing this project was approximately $1.00.  A commercial cable management product is in the range of $30 – $50.  The locally produced product exactly meets our needs, whereas the commercial product is “made to work”.  Below is a view of the product in production.  Each waterfall took approximately four hours to print.

3D_4587

McEvilly Waterfall — 2014

Data Warehouse

Posted on by

The new dimensional data warehouse was launched June 2, 2014.  The WOU model uses the Kimball approach to modeling.  The current warehouse contains data from Finance, Student and Human Resources.  Both canned reports and ad-hoc query capability are available.  The data is refreshed on a nightly basis.

The model was developed using a product called Oracle ODI.  ODI is the same tool that is used to perform the nightly data refresh.  Historical data is preserved during the refresh process.  The modeled data is then transformed into a business friendly format utilizing Cognos Framework Manager.  Cognos Connection is the web-based end-user reporting tool.

The warehouse team includes:

  • Richard Kavanagh — Kimball modeling expert
  • Shea Hawes — SIS expert
  • Nick Miller — FIS/HR expert
  • Max Chartier — Reports guru
  • Jared Petersen — Reports, web docs, etc.  (student employee, honors student)
  • Michael Ellis — Project development

For access to the data warehouse contact datasecurity@wou.edu

Data warehouse resources can be found here.

Available warehouse reports can be found here.

The warehouse can be found here.

Domain Name Services (DNS)

Posted on by

WOU operates a primary DNS, along with several secondary DNS servers.  The primary server runs on a Sun Microsystems server running Solaris 10.  The secondary servers run on Cisco blade servers running RedHat Linux on top of VMware, a virtualization engine.  The servers are configured and maintained by Dave Diemer, unix systems administrator.

DNS servers allow you to access a destination on the internet using words, rather than numbers.  As an example, you can use https://wou.edu instead of https://140.211.127.171.  There is a tool called nslookup that you can use to determine the ip address associated with a domain name.  Another tool, dig, returns additional information, including which DNS servers can authoritatively respond with the ip address associated with the domain name.

There is a good document on the history and usage of Domain Name Servers here.

IMG_4608

Lead Unix Systems Administrator