Database Systems Evaluation Expert

Accountabilities

• Design one challenging, self-contained database engineering task that evaluates an AI agent's ability to solve realistic problems in database systems.

• Develop clear problem statements and provide reproducible repositories or technical environments that enable consistent testing and evaluation.

• Implement a working reference solution that demonstrates the expected engineering approach and establishes a reliable benchmark for assessing submissions.

• Create comprehensive tests to verify correctness, error handling, concurrency behavior, retry mechanisms, and regression prevention, where applicable.

• Evaluate system performance using relevant metrics, including latency, throughput, memory consumption, and other measures appropriate to the task.

• Develop detailed grading rubrics that distinguish robust engineering solutions from implementations that appear correct but fail under realistic conditions.

• Assess technical trade-offs involving database internals, storage engines, query execution, optimization, transaction processing, replication, and other related systems.

• Collaborate with technical reviewers to refine task specifications, validate reference solutions, and ensure that evaluation criteria are technically sound.

• Deliver a complete, well-documented task that meets the required quality standards and project deadline.

• Incorporate technical feedback where necessary to improve the clarity, reproducibility, and reliability of the evaluation process.

Requirements:

• At least five years of hands-on experience working with database engine internals, or a PhD in database systems or a closely related field.

• Deep technical knowledge of one or more database systems areas, such as storage engines, query execution, query optimization, transaction processing, replication, or low-latency systems.

• Strong software implementation skills and the ability to develop reliable, testable solutions to complex engineering problems.

• Demonstrated understanding of correctness, performance optimization, concurrency, failure handling, and the trade-offs involved in designing database systems.

• Ability to construct meaningful technical tests, establish measurable performance criteria, and evaluate solutions against rigorous engineering standards.

• Strong analytical and problem-solving skills, with the ability to break down complex technical challenges into clear, self-contained tasks.

• Excellent written communication skills, including the ability to explain implementation decisions, expected behavior, grading criteria, and performance trade-offs clearly.

• Experience with Go is preferred. Familiarity with at least one compiled programming language, such as C, C++, Rust, Go, or Java, is necessary for the technical assessment.

• Ability to work independently, manage a defined project scope, and deliver high-quality technical work within an agreed timeframe.

• Willingness to complete a self-paced technical assessment consisting of written responses and a coding exercise, using a personal laptop or desktop with a camera, microphone, and full-screen sharing capabilities.

• Access to a development environment with Python 3.8 or later, a suitable compiler, and a code editor and terminal.

• Commitment to completing the assessment independently, without external AI assistance, as required by the assessment guidelines.

• Experience limited to database administration, SQL analytics, ETL processes, or application-level database usage alone does not meet the technical requirements for this opportunity.

Benefits:

• Competitive project compensation: Earn $2,200 for completing the assigned evaluation task, subject to selection and the applicable project terms.

• Defined project scope: Complete one specialized database engineering task requiring approximately 25 hours of work.

• Remote work arrangement: Contribute from your preferred location in India without the need for a traditional office setup.

• Flexible assessment schedule: Complete the self-paced technical assessment at your own pace, with control over when you advance between sections.

• Technical feedback: Receive feedback following the technical review of your completed task.

• Advanced AI exposure: Apply your database engineering expertise to evaluate the capabilities of AI agents on demanding, real-world technical problems.

• Opportunity to influence AI development: Help establish benchmarks that measure the correctness, robustness, and performance of AI-generated engineering solutions.

• Potential future projects: If the pilot expands, completed assessments may be considered for additional opportunities, subject to project requirements and availability.

• Referral rewards: Earn up to $250 for each successful referral through an eligible referral link, subject to applicable referral limits and policies.