Big Data Platform Engineer
G-Research is Europe’s leading quantitative finance research firm. We hire the brightest minds in the world to tackle some of the biggest questions in finance. We pair this expertise with machine learning, big data, and some of the most advanced technology available to predict movements in financial markets.
Our business centres on forecasting financial markets. To achieve this, G-Research uses an ever growing amount of data and processing. The Big Data Platform team builds the primarily Spark and Hadoop-based platforms that are key to enabling these business-critical functions.
The team is responsible for cutting-edge, petabyte-scale clusters which underpin diverse use-cases such as quantitative research, risk analysis and cyber security. The team works closely with various development teams, quantitative researchers, IaaS Engineering, other PaaS teams, and security. We are keen to be close to these users, understand their use-cases and challenges, and help them get the most out of our platforms.
At such a scale, automation is key, and with this role there is a significant focus on configuration management, orchestration, Infrastructure as Code and CI/CD. In addition to this, G-Research are embarking on a transformational journey in how we delivers infrastructure using open source technologies. We are investing heavily in a hybrid cloud platform on which to build the next generation of applications and distributed platforms and deliver development efficiencies. This provides a unique opportunity to re-invent how we operate our big data ecosystem, and continue to modernize and improve our tooling and practices.
Key responsibilities of the role include:
- You will help shape and engineer the big data platform, ensuring it is scalable, stable, performant, and easy to use and maintain
- Help our users work at pace and get the most out of the platform by providing metrics, documentation, and self-service infrastructure
- Define automation standards, frameworks and reporting
- Manage a complex ecosystems of distributed technologies working together
- Enable the company to expand its big data capabilities using Infrastructure-as-Code principles
- Build infrastructure automation
- Develop Python libraries
- Develop CI/CD pipelines
- Use your advanced troubleshooting skills to diagnose and fix problems
Who are we looking for?
The successful candidate will be an enthusiastic Platform Engineer who is able to build an automated, scalable, reliable and high performing Big Data platform. They will work well as part of a team, but also be able to propose and run your own projects and improvement initiatives.
The team has a mixed set of skills that complement each other. Typically, team members will have strong abilities in one of infrastructure automation, Big Data technologies or software development, and be keen to expand their expertise in other areas.
The ideal candidate will have strong skills and experience in at least one of the following:
- Automation tooling: exposure to a majority of the following tooling or the ability to quickly pick-up new tooling:
- Jenkins (CI/CD)
- Linux OS core principles, performance and tuning
- Cloud technologies, e.g. Terraform, AWS, OpenStack
- Big Data technologies: exposure to a number of the following big data components, including building, tuning, troubleshooting clusters:
- Hadoop ecosystem, e.g. HDFS, Yarn, Zookeeper, Hive
- Batch and streaming job frameworks, e.g. Spark, Storm, Flink
- NoSQL databases, e.g. HBase
- Security components, e.g. Kerberos, SSL certificates
- Cloud technologies, e.g. AWS EMR, CDP Public Cloud
- Experience with scripting or programming languages, Python is preferred, but capability in any high-level languages is acceptable
- Understanding of unit testing
Who are we looking for?
- Highly competitive compensation plus annual discretionary bonus
- Informal dress code and excellent work/life balance
- Comprehensive healthcare and life assurance
- 25 days holiday
- 9% company pension contributions
- Cycle-to-work scheme
- Subsidised gym membership
- Monthly company events
- Central London office close to 5 stations and 6 tube lines