Head of AI and Quantum at Moody's Analytics

In today's rapidly evolving technological landscape, generative AI transforms entire industries by enabling users to quickly generate useful content, such as text, images, or videos. AI personal assistants are an example of the transformative potential of generative AI. These personal assistants can dramatically increase the productivity of users tasked with generating reports, websites, and even code.

Moody's is at the forefront of exploring innovative technologies to enhance the productivity and efficiency of our workforce. As part of our effort to embrace and drive advancements in generative AI, Moody's recently conducted a month-long internal trial with GitHub Copilot, a generative AI personal assistant tool that serves as a companion for coders. Led by Sergio Gago Huerta, Head of GenAI & Quantum at Moody’s, this trial sought to assess the impact of GitHub Copilot on coders' performance and overall coding experience. "The impressive results revealed a significant improvement in both productivity and enjoyment among the participating coders," – said Sergio.

GitHub Copilot Experiment

GitHub Copilot (GHC) is similar to your phone's 'autocomplete' function - except it's for programmers. Coders can write a comment or start typing a line of code, and GHC will automatically complete the line and even suggest entire blocks of code and complete functions. Although it's important to note that the programmer is still responsible for ensuring the code it produces is correct and crafting the high-level logic so the pieces fit together.

Coders across Moody's were invited to participate in the GHC trial – as a result, over 300 Moody's employees were granted access to GHC. As part of the trial, David Bressler from Moody's Generative AI Team designed a controlled experiment to test the impact of using GHC. Participants were given a timed (1-hour) coding test consisting of difficult coding questions. Partakers were randomly selected to be part of the Control group or the Experimental Group. The Control group filled out the coding test before they were granted access to GHC - which created a baseline of performance without the assistance of the tool. In contrast, the Experimental group filled out the coding test after they finished one month of using GHC in their daily tasks.

“AI personal assistants are an example of the transformative potential of generative AI. These personal assistants can dramatically increase the productivity of users tasked with generating reports, websites, and even code.”

We found that using GHC improved performance on the test across the board. We designed an array of various test cases to assess whether the code was operating correctly directly. Using this generative AI tool was associated with an increase in the average number of test cases passed (from 42% to 75% of test cases passed) (FIGURE 1), as well as an increase in code quality scores (as assessed by an independent group of experienced coders).

FIGURE 1: Copilot improves the performance of a sequence generation task. Users completed a HackerRank sequence generation task, either without (Control Group) or with (Experimental Group) the aid of GitHub Copilot. For this task, HackerRank automatically scored code performance based on the percentage of test cases passed.

Moreover, these improvements were apparent across multiple different subgroups. Copilot improved performance in the coding test across multiple different age ranges, across the three coding languages we tested (Javascript, Python, and C#), and regardless of a number of years of prior coding experience (FIGURE 2).  

FIGURE 2: Improvement in the sequence generation task by coding   experience, age, and coding language.

Perhaps more important than the results of this particular test, however, we asked trial participants to assess their own experience using GHC in their daily tasks. We found that users overwhelmingly enjoyed working with GHC and found it helped them be more productive in their work at Moody's. More than 75% of trial participants reported that it helped them "stay in the flow" of coding, reduced unnecessary search time on the internet, and made daily coding more productive and enjoyable. (FIGURE 3).

Nearly 80% of users said figuring out how to make GHC useful for their daily coding tasks was easy. A majority of users also reported that GHC helped them learn new code syntax and techniques, as well as helped them learn completely new technology stacks.

We also asked users to quantify the benefit they experienced using GHC. These self-reported measures are astounding. Of the 86% of users who reported that GHC sped up their daily coding tasks, more than half reported a productivity speed-up of greater than 20%, and more than a tenth reported a productivity speed-up of greater than 50%! We also asked users to assess what proportion of GHC suggestions proved useful. Nearly all the users found at least some of the suggestions to be useful to their coding tasks, and two-thirds of users found that GHC produced useful suggestions more than 50% of the time. We see the large magnitude of these self-reported benefits to be very encouraging for the potential of GHC and other generative AI tools we are developing to improve productivity across the organization at Moody's.

And maybe the best sign of GHC's generative AI benefits – is that nearly 90% of those involved in the trial requested access to a license to continue using the tool.

What Does It Mean to Moody's?

As a result of this successful trial, Moody's intends to move forward with an enterprise license to GitHub Copilot. This experiment shows the practical relevance and efficiencies we can leverage using AI in our day-to-day work. GHC is just one of the generative AI tools we are testing and developing at Moody's. We have already released an alpha version of an internal generative AI assistant - "Moody's CoPilot" - which is enabling users across the organization to harness the power of generative AI without traditional coding experience. Given a prompt, users can use Moody's CoPilot to ask questions, summarize text, evaluate code, and generate images or articles from scratch. In addition, Moody's CoPilot has provided an opportunity - repeatedly requested in internal surveys - for users to connect and collaborate on generative AI topics and use cases in MCO-wide events, training sessions, open office hours, and Slack channels, including #app-moodys-copilot which has over 2,000 users and growing.