The SRE Journey Continues…
G-Research has been growing rapidly over the last several years. In 2017 and again in 2018, our software engineering team grew by 50%. As discussed in our previous SRE Journey post, we began building an SRE team during this timeframe. What began with one team member has expanded to seven SRE plus a manager. This quick growth has been a challenge and did not come without some adjustments to the hiring process along the way. Here is how we’ve approached the puzzle of finding the best SRE candidates for our engineering team.
The Challenge of Finding a Great SRE
The first position was filled by an internal candidate, who had previously been an SRE at another organisation. This engineer expressed interest in moving from a more traditional Software Engineering role at G-Research to the SRE team we were building. We were happy to make that happen. However, we knew that attracting high-quality SRE candidates from outside would be a more difficult task. Great SREs are hard to come by and are often incentivised to stay in their current roles. With many other well-known organisations in London clamouring for these talented people, our in-house recruiters have had their work cut out for them.
After pouring over many CVs, and holding many phone screens, sadly, we were struggling to find the perfect SRE candidates. However, we began to spot patterns. We spoke to plenty of fabulous software engineers, but most were interested in hands-on keyboard time. We spoke with amazing infrastructure engineers, who had deep knowledge in Chef, Terraform, ansible, linux kernel implementations and operations, but they didn’t read or write object oriented code, nor could they readily see inefficiencies in the code they reviewed. While these candidates were brilliant at what they did, they did not fit what we needed in our SRE team, especially for the first few hires.
Continuous Improvement in SRE Hiring
Knowing that something needed to change, we took a page out of the continuous improvement playbook and opted to adjust our process. We began by working closely with the internal recruiters to really hone in on qualities we wanted to see in this role. Ultimately, we shifted the focus of the interview and hiring process to create better matches.
What we wanted in SRE candidates (and, I should say, we still want!) were engineers with broad experiences. Engineers that could help engineers implement monitoring solutions within their services or help engineers build out their CI/CD pipeline. We wanted people who were as keen on debugging networking schemes as they were promoting best practices in telemetry. So we updated the questions we asked in the phone screen. We binned highly technical, gotcha questions and focused more on problems at scale. This change reduced the number of candidates that passed through the phone screen, but the calibre of the candidates that made it to the next stage was much higher.
The modified process, which includes a mix of technical, design and scale questions resulted in our first external hire. The candidate, who was already an SRE at another organisation, liked how the role read and was aligned with our vision from the start. Had we applied our previous dev-specific questions to his screening process, this person may not have progressed through. Ultimately, we were able to put together the right comp package and demonstrate how we were going to grow the team and this candidate came on board.
We continued to apply this modified approach with our next two hires, taking time to review and tweak the process as necessary. We settled on a few standardised scale interview questions that are now given to all candidates. This technique helps us to easily compare candidate answers against the baseline of what we are seeking. With this process at the ready, we continued to hire one engineer at a time throughout 2018 and 2019.
A Similar But Different Challenge
During this time, we began looking for an SRE manager. Hiring for this role proved to be even more challenging. In the beginning, when the team was very small, we sought a very hands-on manager, but couldn’t find any individuals with the right balance of leadership and tech. Again, many of the candidates we interviewed were either too hands-on, or at the time, too far away from code. Unfortunately, some of the managers we brought in failed the interview because they didn’t meet the highly technical expectation set. Similarly, other highly technical managers failed the interview process due to lack of leadership experience or vision.
As the team grew, we began to re-evaluate our expectations for the team manager role. It was becoming clear that we didn’t need a hard-core programmer or systems engineer. What we needed was a strong manager with SRE experience. While finding someone to fit that category was also difficult, it turns out it wasn’t impossible. Ultimately, we found our SRE manager in America. And after a few video chats, a very coincidental meet-up at SRECon Americas 2018, and a couple of transcontinental plane rides, this person is happily leading the team forward.
Our takeaway from these experiences is that hiring for SRE is hard, but not impossible. Applying our own best practices in the vein of continual improvement, and being intentional about the hires we make has meant bringing on a team of quality SREs to G-Research in the last year. And when a candidate popped up in Eastern Europe, we wasted no time in applying our process and eagerly snatching them up.
We are still looking for talented engineers to join our SRE team. If this is something you would be interested in finding more about, please email the Recruitment Team.