Senior Data Platform Engineer
Chennai, TN, India12h ago
Key details
- Location
- Chennai, TN, India
- Work style
- Onsite
- Level
- Senior
- Job type
- Full-time
- Experience
- 8–11 yrs
- Function
- DevOps / Infrastructure
- Focus area
- Data engineering & Analytics
- Posted
- 12h ago
Skills
About the role
We are seeking a highly skilled, hands-on Senior Data Platform Engineer to design, build and manage complex, high-performance enterprise data services supporting data integration, reconciliation, data controls, data quality and operational intelligence.
The role combines strong software engineering expertise with distributed data processing, cloud-native architecture and modern lakehouse technologies. The successful candidate will develop reusable, scalable and resilient data services supporting high-volume batch and real-time processing across enterprise applications.
Key Responsibilities:-
- Enterprise Data Services Engineering: Architect, develop and maintain complex, reusable data services for data ingestion, transformation, standardisation, validation, reconciliation and data processing using Scala, Spark, Python and Java.
- Data Integration & Processing: Build scalable integration frameworks supporting files, databases, APIs, Kafka, CDC and enterprise messaging, including batch, near-real-time and streaming ingestion.
- Reconciliation & Matching Engines: Develop high-performance reconciliation services supporting complex matching rules, multi-pass matching, tolerance handling, one-to-one and many-to-many matching, balance proofing and exception generation.
- Data Controls & Integrity: Engineer reusable control frameworks for completeness, accuracy, timeliness, record counts, control totals, source-to-target validation, auditability, data lineage, replay and recovery.
- Data Quality Engineering: Develop configurable data quality services supporting schema validation, business rules, profiling, anomaly detection, data quality scoring, exception handling and automated remediation.
- Databricks & Lakehouse Engineering: Build and optimise distributed processing workloads using Databricks, Apache Spark, S3, Apache Iceberg, Delta Lake, Unity Catalog and Lakeflow orchestration.
- Streaming & Event-Driven Services: Develop Kafka/MSK-based event-processing services, CDC integration, stateful streaming, checkpointing, deduplication and reliable event delivery.
- Software & API Development: Design and develop REST APIs, microservices, metadata-driven processing frameworks, configuration services and reusable libraries using Scala, Java/Spring Boot and Python.
- AI & Intelligent Data Services: Integrate AI/ML capabilities for intelligent matching, anomaly detection, data quality remediation, metadata discovery and AI-assisted data processing.
- Cloud Engineering, DevOps & Reliability: Implement AWS infrastructure, Terraform, CI/CD, automated testing, observability, performance optimisation, FinOps, production support and disaster recovery.
Essential Technical Skills:-
- Software Engineering: Strong hands-on Scala, Java or Python development, object-oriented design, concurrency, distributed systems and software design patterns.
- Distributed Processing: Apache Spark, Scala Spark, PySpark, Spark SQL, Structured Streaming and performance tuning.
- Cloud & Lakehouse: AWS, Databricks, S3, Apache Iceberg, Delta Lake and Unity Catalog.
- Data Integration: Kafka/MSK, CDC, REST APIs, JDBC, file processing and enterprise messaging.
- Data Services: Data transformation, reconciliation, rule engines, data validation, data controls, data quality and exception processing.
- Application Development: Microservices, REST APIs, Spring Boot/FastAPI, metadata-driven frameworks and service integration.
- Data Engineering: SQL, data modelling, schema evolution, partitioning, incremental processing and data optimisation.
- DevOps & SRE: Terraform, Docker, Git, CI/CD, automated testing, CloudWatch, OpenTelemetry and production troubleshooting.
Desirable Skills:-
- Experience developing enterprise reconciliation or financial data processing platforms.
- Real-time matching, balance proofing and stateful stream processing.
- AI/ML, MLflow, vector search, RAG and agentic AI integration.
- MongoDB Atlas, PostgreSQL/Aurora and enterprise workflow integration.
- Financial services data governance, regulatory controls and operational resilience.
Experience
- 8+ years of hands-on software development and data platform engineering.
- Proven experience building complex, high-throughput distributed data services.
- Strong development experience with Scala/Spark and cloud-native architectures.
- Experience designing reusable frameworks rather than only implementing ETL pipelines.
- Demonstrated ownership of production systems, performance, reliability and scalability.
- Experience delivering enterprise-scale platforms within regulated environments preferred.
Expected Outcomes:-
- Scalable, reusable enterprise data services.
- High-performance batch and streaming reconciliation processing.
- Automated, auditable data quality and control frameworks.
- Reliable and cost-efficient Databricks/AWS workloads.
- Production-ready services with strong observability, resilience and automated recovery.
- Accelerated onboarding of new data sources and business use cases through configurable, self-service capabilities.
Education:
- Bachelor’s degree/University degree or equivalent experience
- Master’s degree preferred
------------------------------------------------------
Job Family Group:
Technology------------------------------------------------------
Job Family:
Applications Development------------------------------------------------------
Time Type:
Full time------------------------------------------------------
Most Relevant Skills
Please see the requirements listed above.------------------------------------------------------
Other Relevant Skills
For complementary skills, please see above and/or contact the recruiter.------------------------------------------------------
Citi is an equal opportunity employer, and qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran, or any other characteristic protected by law.
If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi.
View Citi’s EEO Policy Statement and the Know Your Rights poster.
How well do you match this job?
Free account needed to apply or save. It takes a minute, and we'll bring you right back to this job.