How to Weather the Perfect Storm Brewing in Your Developer Team and Database
Database problems sometimes seem to appear without any warning. Whether you run into a sudden server stall or find your resources devoured by a set of poorly performing queries, a database issue can have effects that ripple through your entire organization. When they do, no one feels the pain more than a database administrator. DBAs regularly bear the brunt of the question, “Why is the database slow again?” Alas, even the best-prepared DBAs will sometimes find themselves fighting to maintain order in the middle of difficult circumstances. And when these circumstance overlap, they can amplify into dangerous effects—a perfect storm of database operations.
The Perfect Storm
The term "perfect storm" describes a rare but exceedingly powerful weather system, when three normal-size storms converge and create a single massive storm, much more dangerous and lethal in comparison. For DBAs and developers, the equivalent of a perfect storm is when multiple operational elements create a difficult and complex combination of working conditions. Unlike an actual perfect storm, which is rare, the following three elements converge frequently at tech-driven companies and present a serious challenge for the team taking care of the database:
For DBAs and developers, the equivalent of a perfect storm is when multiple operational elements create a difficult and complex combination of working conditions
● Rapid release cycles making development more intense. Engineering teams release new software versions dozens of times per day, putting database performance at risk if queries are not optimized. Meanwhile, environments grow increasingly complex, with modern apps often using 3-5 different solutions in their data tier alone.
● Increased system performance expectations. Databases often support customer-facing applications, so performance and availability are even more crucial than they were only a few years ago. For example, high-volume e-commerce companies aim for site performance standards measured in milliseconds—and downtime is simply unthinkable.
● Increased reliance on a small group of experts. DBAs provide the critical expertise required to size, build, and maintain highly scalable database environments. Simply put, they are outnumbered at many organizations. During the busiest times, they might feel as if they're under siege. Many companies have hundreds of developers deploying to production multiple times daily, supported by only a handful of DBAs. It’s not uncommon to find DBAs who manage anywhere from 100 up to 3,000 servers per person.
It can be hard to retain DBAs in this demanding and crisis-prone environment. And losing a DBA can be a real blow to an organization—they're in high demand and short supply. Recent data from the U.S. Bureau of Labor Statistics cite the overall unemployment rate for tech workers at 2.4 percent, with employment opportunities for DBAs projected to grow 11 percent from 2014 to 2024–a pace that would add 13,400 jobs.
So, how can you create the conditions that will allow DBAs to thrive while continuing to innovate and deliver better performance? You need to have the right number of staff, with the right skills, who are effectively sharing the workload with their developer brethren.
Weathering the Storm
In data-driven, high-performance organizations, the DBA’s role are a critical interface between vital IT teams. The key to happier and more productive teams is making that team collaboration work.
● Right-size your staff. The first challenge is simply staffing the right number of DBAs. Right-sizing staff is dependent upon the size and complexity of your environment, and the number of developers serving it. The ratio of developers to DBAs can get out of balance in an atmosphere that encourages speed: pushing more releases, faster, naturally tends to result in a big bump in developers' numbers. Companies often hire more but keep the DBA staff at the same size.
In our recent research report “The Hidden Costs of Data Engineering,” we found that 80% of respondents still rely on teams of six or fewer to serve an ever-growing number of developers and data. Faced with this imbalance, many teams can only act reflexively as ad hoc requests multiply. Getting your DBA-to-developer ratio right is essential to making DBAs more proactive and your whole team more productive overall.
● Take a skills inventory. Beyond having the right number of staff, you need to make sure that your staff has access to the right skill sets. The specific duties of a DBA depend greatly on the organization's technologies deployed, team structure, and company culture. However, the fundamental objectives of database administration remain constant and typically include installing, configuring, securing, and maintaining the database server; designing the data’s logical and physical schema; validating rules and business logic; and developing, monitoring, and optimizing queries running against the data.
You will also find DBAs responsible for monitoring the database server’s performance, security, and availability. To become more scalable and flexible you need to understand what database skills you also have on your dedicated engineering team and within your team of developers.
● Use talent wisely. Perhaps your strongest storm-avoidance mechanism is sharing the database workload between developers and DBAs—DevOps principles are powerful! Once you have a clear skills inventory, you can look to allocate or share tasks to relieve the pressure on DBAs and help speed deployments and improve time-to-resolution in production.
By offloading some tasks to developers, you relieve the pressure on DBAs—they no longer need to feel like they are the single point of failure or the team's bottleneck. In this new shared model, developers take on some of the more traditional operational tasks like measuring their code's effect on the database, reading their own monitoring and metrics, and anticipating—from development to production—how their changes to the system might impact performance across your entire tech stack.
Your objective is not to just weather the storm, but to avoid it altogether. You need to understand the skill sets of your team and allocate tasks, to share the workload more effectively, reducing the pressure on your DBAs. The result will be improved team productivity, boosted employee satisfaction and retention, and better overall system performance. So, the next time someone wants to know why the “database is slow again,” you will have an integrated team, collectively responsible for answering that question. More importantly, you'll be confident that your team can navigate any tricky weather patterns together, quickly and safely.