Qubole, the leading cloud-agnostic, big data-as-a-service provider, is passionate about making data-driven insights easily accessible to anyone. Qubole delivers the industry’s first autonomous data platform. The cloud-based data platform, Qubole Data Service (QDS), removes the burden of maintaining infrastructure of multiple big data processing engines, and enables customers to focus on their data. Qubole customers process nearly an exabyte of data every month. Qubole investors include Charles River, Institutional Venture Partners, Lightspeed, Norwest, Harmony and Singtel Innov8.
We are looking to hire an experienced big data performance engineer.
What you'll be doing: As part of the Performance Team at Qubole, you will
· Develop testing and workload frameworks for running big data benchmarks such as TPCDS against various big data engines supported by Qubole (Hive, Hadoop, Spark and Presto).
· Work on automating benchmark runs using frameworks such as Jenkins
· Work on tools and scripts for analyzing performance results.
· Compare performance of Qubole big data engines against other big data offerings from competitors.
· Be also responsible for running these benchmarks against new Qubole releases and monitoring any deviations in performance.
· Work closely with various backend engine teams and solution architects to fine tune engine configurations and performance runs.
· Talk and write about your work in blogs and conferences.
Must Haves :
Prior experience in running and using big data benchmarks such as TPCDS.
Prior experience is performance engineering.
Prior experience in using or creating solutions involving big data systems such as Hive, Hadoop, Spark and Presto.
Past experience of using one or more clouds - Amazon Web Services (AWS), Azure and Google Compute Cloud (GCP).
Some experience of developing code in Java and Python.
5-6 years of work experience.
Good to Have:
Good understanding of query processing and database internals.
BS in Computer Science or related fields.