Sr Data Engineer- Databricks
Bengaluru, KA, India16mo ago
Key details
- Location
- Bengaluru, KA, India
- Work style
- Onsite
- Level
- Senior
- Job type
- Full-time
- Experience
- 5–8 yrs
- Function
- Data & ML
- Focus area
- Data engineering & Analytics
- Posted
- 16mo ago
Skills
access controlawsazureci/cdcloud computingdata governancedata lakesdata pipelinesdatabricksdebuggingdelta lakedevopsgcpgenerative aigithub actionsinfrastructure automationjenkinskafkallmmachine learningmetadata managementpysparksparksqlsystem designterraform
About the role
Exp- 5+ yrs
Location- Remote (Preferred candidates to be in bangalore)
Notice- Looking candidates with to be joining within 30 Days
Key Responsibilities:
- Design, implement, and optimize scalable data pipelines using Databricks and Apache Spark.
- Architect data lakes using Delta Lake, ensuring reliable and efficient data storage.
- Manage metadata, security, and lineage through Unity Catalog for governance and compliance.
- Ingest and process streaming data using Apache Kafka and real-time frameworks.
- Collaborate with ML engineers and data scientists on LLM-based AI/GenAI project pipelines.
- Apply CI/CD and DevOps practices to automate data workflows and deployments (e.g., with GitHub Actions, Jenkins, Terraform).
- Optimize query performance and data transformations using advanced SQL.
- Implement and uphold data governance, quality, and access control policies.
- Support production data pipelines and respond to issues and performance bottlenecks.
- Contribute to architectural decisions around data strategy and platform scalability.
Requirements
Required Skills & Experience:
- 5+ years of experience in data engineering roles.
- Proven expertise in Databricks, Delta Lake, and Apache Spark (PySpark preferred).
- Deep understanding of Unity Catalog for fine-grained data governance and lineage tracking.
- Proficiency in SQL for large-scale data manipulation and analysis.
- Hands-on experience with Kafka for real-time data streaming.
- Solid understanding of CI/CD, infrastructure automation, and DevOps principles.
- Experience contributing to or supporting Generative AI / LLM projects with structured or unstructured data.
- Familiarity with cloud platforms (AWS, Azure, or GCP) and data services.
- Strong problem-solving, debugging, and system design skills.
- Excellent communication and collaboration abilities in cross-functional teams.
How well do you match this job?
Skills
Experience
Level
Title
Location
Salary
Freshness
Similarity
Free account needed to apply or save. It takes a minute, and we'll bring you right back to this job.