Our Journey to DevOps
CIOREVIEW >> Education >> NEWS

Seton Hall University

Dr. Stephen G. Landry, CIO, and Matthew Stevenson, Executive Director, IT Architecture

Our Journey to DevOps

Dr. Stephen G. Landry, CIO, and Matthew Stevenson, Executive Director, IT Architecture
Dr. Stephen G. Landry, CIO, and Matthew Stevenson, Executive Director, IT Architecture, Seton Hall University

Nicholas Carr observed that what electricity was to the industrial age, cloud computing is to the information age. While most campuses now use cloud services such as Gmail and Microsoft Office365, many colleges and universities are moving all or part of their infrastructure to the cloud to take advantage of the flexibility, resiliency, and security the cloud affords. This move to “Infrastructure-as-a-Service” is likely to increase as we emerge from the current pandemic, as campuses that were further along in cloud adoption found it easier to pivot to remote operation in March 2020.

Seton Hall University began our move to Amazon Web Services (AWS) in 2015 out of necessity. We did a full replacement of our on-premises data center in 2012, but by 2015 the server requirements for part of a new release for our Banner ERP system exceeded any of our servers. By 2018 we had moved most of our systems to AWS, and we gained some immediate benefits in efficiency, security, and reliability. We no longer had to build servers or install operating systems and were able to load balance servers across availability zones to build redundancy. But many of our systems are still built on older IT architectures, and we were not able to take full advantage of the cloud.

Many colleges and universities face similar challenges. Most higher education Student Information Systems are running “on-premises,” with the campus IT team responsible for applying software updates and making customizations to support campus business processes. At Seton Hall, this meant we had a very traditional, “siloed” IT organization, with different teams responsible for developing the ERP customizations, managing the database, and running the servers and operating systems. Desired enhancements to our Banner ERP system took considerable time and effort to develop, and once deployed, required ongoing maintenance and delayed applying software updates and security patches from the vendor.

“Change can be difficult; it is much easier to replace legacy technology than legacy thinking”

Hosting systems in AWS offloads a lot of routine tasks from campus IT professionals, such as building servers and load balancers, installing and updating operating systems, and scaling to meet demand.  The cloud fosters innovation and speed; rather than being built on a massive code base, “cloud native” applications are built on micro services that are easy to build, deploy, and change. Devops –a portmanteau of “development” and “operations” - is an approach to rapidly building applications in the cloud. Devops seeks to break down organizational silos; it fosters collaboration among cross-functional teams which span the traditional work processes of developers and systems administrators. Devops combines lean and agile methodologies with a systems-oriented approach. Using devops processes and tools leads to faster software development, easier deployment, and greater reliability. But devops practices can be uncomfortable to traditional IT leaders and organizations that evolved to manage traditional ERP systems.

Seton Hall University’s adoption of devops practices started slowly. Our systems support teams had initial responsibility for managing our cloud infrastructure in AWS. As their skills with AWS matured, they began automating more operations processes and learning AWS’s development tools. Several members of the operations team were looking to increase their skills and responsibility, so in 2018 we created a small, dedicated devops group. The devops team became trained and certified in agile methodologies, such as SCRUM, and began using new tools for project tracking, such as Jira and Git. The devops team changed their main development environment from Java to Python and implemented Ansible for automation control and scheduling. While we continued some traditional application development inside our Banner ERP, the team’s focus shifted to applications development and integration using REST APIs, especially as vendors began requiring API integration with our Banner ERP system rather than file feeds. As part of the evolution of agile in the IT organization, the devops team met with every department in IT, including our PC support and instructional technology teams, to demonstrate the value of agile concepts and tools.

The 2020 pandemic increased the demand for IT services while reducing campus revenues. In response, the University imposed an across-the-board staff and budget reduction which resulted in laying off 10 percent of our IT workforce. The devops team was a key resource as the Universitypivoted to emergency remote teaching and operations. The devops team automated the deployment of Microsoft Teams rooms for each course, rapidly deployed AWS Appstreams to provide remote access to the University workforce, and used AWS Connect to create call centers accessible from anywhere for our Help Desk and other critical student services.

When the pandemic receded in 2021, IT salaries began increasing, making it more difficult for us to retain IT talent. Discussions revealed that many of our IT professionals desired to learn devops skills and advance their responsibilities. This led to a major reorganization, expanding the devops teams. By eliminating some open positions, we were able to increase the salaries of this larger devops team, recognizing both their increased value to the University and the increased marketability of their new skills. As part of our devops adoption process, we created “Cloud Fridays” where twice a month we formed new ad-hoc teams, based on interest and skills, to address University IT needs using open tools in AWS while learning new techniques such as sprints and backlogs. Work teams outside of IT were further exposed to devops concepts and tools through their SaaS vendors, which helped increase awareness and acceptance of devops across campus.

While still relatively new, this approach has already created significant value for the University. The devops team has created a data lake in AWS that combines data from our ERP system with other externally hosted systems, including our Blackboard learning management and our EAB student retention (CRM) systems to help track student progress and student success. They created a self-service portal for onboarding new students and employees into our MFA system and automated moving faculty phone lines from our legacy PBX to Microsoft Teams as we migrate the campus to a new phone system.

Everyone’s journey toward devops will be different. Critical success factors for our implementation of devops included: We started with a small, willing team interested in progressing their careers by learning agile programming methodologies while solving important problems for the campus. The team began with ruthless automation of as much of the operations as they could to free up their time for addressing campus technology needs while developing new skills. Grass roots buy-in was critical, but so was support from our leadership team, who inevitably felt discomfort and disorientation by their teams’ empowerment. Transparency with all the IT staff, whatever their level of involvement with devops, was important, as everyone wonders what their future in an agile organization might be and whether devops is going to automate them out of a job. We had to plan for changes to our organization; devops led us to become a smaller, more agile, but more highly skilled IT organization. Change can be difficult; it is much easier to replace legacy technology than legacy thinking. We encouraged our IT professionals to shift their thinking from products to functions, for example, thinking about becoming the virtualization expert rather than the VMware tech. Finally, we anticipate devops will expand into IT areas beyond the traditional notion of a developer; telecom, for example, will move to software defined networking. Perhaps the most important lesson is that in any organization devops must start sometime, and if you haven’t started yet, the time is now.

 

The articles from these contributors are based on their personal expertise and viewpoints, and do not necessarily reflect the opinions of their employers or affiliated organizations.