Senior Engineer, Competitive Benchmarking & Performance

Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale , unleashing the potential of businesses and people.

We are seeking a Senior Software Engineer to lead competitive benchmarking initiatives within the Elasticsearch Performance team. You will develop representative workloads, automate reproducible experiments, and investigate how Elasticsearch compares with alternative systems.

Responsibilities

• Own prioritized competitive benchmarking initiatives across search, vector and hybrid retrieval, and analytical and observability workloads.
• Build and maintain automated benchmark harnesses, datasets, query sets, and load profiles that reflect real-world usage, including concurrent ingestion and querying.
• Design fair comparisons with documented system versions, configurations, tuning decisions, resource budgets, and functional differences.
• Evaluate latency distributions, throughput, resource consumption, and cost efficiency. For retrieval workloads, measure performance at comparable recall or relevance targets.
• Adapt and extend workload-appropriate tools such as Elasticsearch Rally, ClickBench, and vector search benchmark frameworks.
• Investigate performance differences using profiling, system metrics, and controlled experiments; collaborate with engineering teams to identify causes and validate improvements.
• Evaluate relevant self-managed and managed cloud deployments, including serverless offerings, accounting for scaling behavior and service constraints.
• Work with engineering and technical content partners to publish reproducible benchmark artifacts and clear technical reports that explain methodology, findings, limitations, and tradeoffs.
• Partner with engineering and product management to prioritize benchmark coverage and translate findings into actionable product improvements.

Requirements

• Solid software engineering skills, with experience building maintainable automation and analysis tools in Python or another relevant programming language.
• Hands-on experience evaluating and diagnosing performance in distributed systems, databases, or search infrastructure.
• Depth in at least one relevant area (search, vector retrieval, analytical databases, or observability systems) and the ability to learn unfamiliar systems.
• Experience designing representative datasets and workloads, including query patterns, concurrency, and ingestion behavior.
• Sound experimental judgment, including repeated measurements, variability analysis, warm-up and cache effects, and the distinction between meaningful changes and measurement noise.
• Experience building reproducible test environments and documenting configurations, procedures, and results.
• Clear technical writing and communication skills, with the ability to explain complex findings and their limitations.
• Solid collaboration skills and the ability to turn performance evidence into applicable engineering recommendations.

Nice to Have

• Experience with Elasticsearch, Lucene, Rally, ClickBench, or vector search benchmark frameworks.
• Experience measuring retrieval quality alongside latency, throughput, and resource usage.
• Familiarity with cloud performance and cost analysis, including autoscaling and serverless systems.
• Public technical writing, open-source contributions, or published benchmark studies.

Benefits

• Competitive pay based on the work you do here and not your previous salary
• Health coverage for you and your family in many locations
• Ability to craft your calendar with flexible locations and schedules for many roles
• Generous number of vacation days each year
• Increase your impact - We match up to $2000 (or local currency equivalent) for financial donations and service
• Up to 40 hours each year to use toward volunteer projects you love
• Embracing parenthood with minimum of 16 weeks of parental leave