Big Data Platform Engineer
G-Research is Europe’s leading quantitative finance research firm. We hire the brightest minds in the world to tackle some of the biggest questions in finance. We pair this expertise with machine learning, big data, and some of the most advanced technology available to predict movements in financial markets.
G-Research’s work centres on forecasting financial markets. To achieve this, we use an ever-growing amount of data and processing. The Big Data Platform team builds the (primarily Spark- and Hadoop-based) platforms that are key to enabling these business-critical functions.
The team is responsible for cutting-edge, petabyte-scale clusters that underpin diverse use-cases such as quantitative research, risk analysis and cyber security. We work closely with various development teams, quantitative researchers, IaaS Engineering, other PaaS teams, and security, understanding their use-cases and challenges and helping them to get the most out of our platforms.
At such a scale, automation is key and with this role there is a significant focus on configuration management, orchestration, Infrastructure as Code and CI/CD. In addition to this, G-Research is embarking on a transformational journey in how we deliver infrastructure using open source technologies. We are investing heavily in a hybrid cloud platform on which to build the next generation of applications and distributed platforms and deliver development efficiencies. This provides a unique opportunity to re-invent how we operate our big data ecosystem.
Key responsibilities of the role include:
- Help shape and engineer the Big Data platform, ensuring it is scalable, stable, performant, and easy to use and maintain
- Help our users work at pace and get the most out of the platform by providing metrics, documentation, and self-service infrastructure
- Define automation standards, frameworks and reporting
- Manage a complex ecosystem of distributed technologies
- Enable the company to expand its Big Data capabilities using Infrastructure-as-Code principles
- Build infrastructure automation
- Develop Python libraries
- Develop CI/CD pipelines
- Use your advanced troubleshooting skills to diagnose and fix problems
Who are we looking for?
The ideal candidate will be an enthusiastic Platform Engineer who is able to build an automated, scalable, reliable and high performing Big Data platform. They will work well as part of a team but also be able to propose and run their own projects and improvement initiatives.
The team has a mixed set of skills that complement each other. Typically, team members will have strong abilities in one of infrastructure automation, Big Data technologies, or software development, and be keen to expand their expertise in other areas.
The ideal candidate will have strong skills and experience in at least one of the following areas:
- Automation tooling (exposure to a majority of the following tooling or the ability to quickly pick-up new tooling):
- Jenkins (CI/CD)
- Linux OS core principles, performance and tuning
- Cloud technologies, e.g. Terraform, AWS, OpenStack
- Big Data technologies (exposure to a number of the following big data components, including building, tuning and troubleshooting clusters):
- Hadoop ecosystem, e.g. HDFS, Yarn, Zookeeper, Hive
- Batch and streaming job frameworks, e.g. Spark, Storm, Flink
- NoSQL databases, e.g. HBase
- Security components, e.g. Kerberos, SSL certificates
- Cloud technologies, e.g. AWS EMR, CDP Public Cloud
- Experience with scripting or programming languages, Python is preferred, but capability in any high-level languages is acceptable
- Understanding of unit testing
Why should you apply?
- Highly competitive compensation plus annual discretionary bonus
- Informal dress code and excellent work/life balance
- Comprehensive healthcare and life assurance
- 25 days holiday
- 9% company pension contributions
- Cycle-to-work scheme
- Subsidised gym membership
- Monthly company events
- Central London office close to 5 stations and 6 tube lines
G-Research is committed to cultivating and preserving an inclusive work environment. We are an ideas-driven business and we place great value on diversity of experience and opinions.
We want to ensure that applicants receive a recruitment experience that enables them to perform at their best. If you have a disability or special need that requires accommodation please let us know in the relevant section.Apply