Senior Data Engineer

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior Data Engineer based in the United States.

We are seeking an experienced Senior Data Engineer to help build, operate, and optimize the data infrastructure powering analytics and business intelligence across global markets. In this individual contributor role, you will take ownership of critical data synchronization pipelines, ensuring reliable and timely data delivery from market-level databases into Snowflake. Working within a modern AWS-based technology environment, you will leverage tools such as Kafka, Spark Structured Streaming, Apache Iceberg, and Amazon S3 to manage large-scale data flows. You will play a key role in improving system performance, scalability, reliability, and cost efficiency while maintaining high standards of data quality. You will also drive engineering best practices, strengthen monitoring and deployment processes, and lead production incident investigations. This is an exciting opportunity to solve complex distributed systems challenges within a collaborative, international, and mission-driven technology environment.

Accountabilities:

Own, maintain, and continuously improve change data capture (CDC) and data synchronization pipelines, ensuring accurate, reliable, and timely data delivery from market-level databases into Snowflake.
Optimize pipeline performance, scalability, and infrastructure efficiency by reducing processing latency, controlling operational costs, and supporting increasing data volumes across multiple markets.
Design, develop, and evolve data infrastructure by onboarding new data sources, tables, and markets while managing schema changes and ensuring compatibility across systems.
Establish and maintain service level objectives (SLOs), implement monitoring and alerting mechanisms, and develop automated data quality checks to ensure data accuracy, availability, and reliability.
Strengthen engineering standards by developing unit and integration tests, maintaining CI/CD workflows, implementing infrastructure-as-code, and introducing safe deployment strategies such as dry runs and blue-green deployments.
Troubleshoot production incidents, lead root cause analyses, and translate findings into improvements in monitoring, testing, technical documentation, and operational runbooks.
Participate in architectural and technical design discussions, prepare and review technical design documents, and contribute to code quality through constructive peer reviews.
Collaborate with Data Science, Analytics, Data Platform, Product, Credit, and Business Development teams to understand data requirements and provide dependable access to critical information.
Take end-to-end ownership of production data systems, from development and deployment through ongoing maintenance and operational improvement.
Requirements:
At least 4 years of professional experience in backend engineering or data engineering, ideally supporting consumer-scale applications.
Minimum 2 years of hands-on professional experience developing with Python.
Proven ability to independently build, operate, troubleshoot, and optimize production-grade data pipelines.
Practical experience with change data capture (CDC) and stream processing technologies such as Kafka, Amazon MSK, AWS Database Migration Service (DMS), or comparable solutions.
Strong proficiency in Apache Spark, particularly Spark Structured Streaming.
Experience with modern data lakehouse architectures and table formats, including Apache Iceberg, Delta Lake, or Apache Hudi, as well as Parquet.
Strong database knowledge across relational and analytical platforms, including MySQL, PostgreSQL, Amazon RDS, and Snowflake.
Hands-on experience with AWS cloud infrastructure, ideally including EMR, S3, MSK, DMS, RDS, and EKS.
Experience with infrastructure-as-code and CI/CD technologies such as Terraform, Terragrunt, Helm, and Jenkins.
Familiarity with data pipeline orchestration and job scheduling tools such as Apache Airflow or Python-based services running on Kubernetes.
Strong understanding of distributed systems, scalable algorithms, and large-scale data processing architectures.
Excellent problem-solving and analytical skills, with the ability to investigate complex technical challenges and deliver reliable solutions.
Strong written and verbal communication skills, including the ability to explain complex technical concepts to engineering teams and business stakeholders.
Ability to work independently, take ownership of technical initiatives, and collaborate effectively across multidisciplinary teams.
Nice to Have:
Experience with Trino or Presto query engines.
Practical knowledge of Kubernetes and containerized applications.
Familiarity with AWS Glue.
Experience with additional stream processing technologies such as Debezium, Amazon Kinesis, or Apache Flink.
Exposure to modern data lakehouse technologies, including Delta Lake and Apache Hudi.
Advanced degree in Computer Science, Mathematics, or another relevant technical discipline.
Benefits:
Annual salary: $130,000–$200,000, depending on qualifications, experience, and other relevant factors.
Fully remote, remote-first working environment.
Opportunity to collaborate with an international team across multiple global markets.
Exposure to modern cloud infrastructure, AI-driven technology, and large-scale distributed data systems.
Opportunity to contribute to innovative financial technology solutions with meaningful global impact.
Inclusive and diverse working environment that values different backgrounds, perspectives, and professional experiences.
Opportunities to influence technical architecture, engineering standards, and the evolution of critical data infrastructure.