Senior data engineer
Budapest, Hungary5mo ago
Key details
- Location
- Budapest, Hungary
- Work style
- Onsite
- Level
- Senior
- Experience
- 6–8 yrs
- Function
- Data & ML
- Focus area
- Data engineering & Analytics
- Posted
- 5mo ago
Skills
About the role
• Data
Pipeline Design & Development (ETL/ELT)
o Develop
ETL/ELT Pipelines: Build scalable ETL/ELT pipelines using Azure Databricks,
integrating multiple data sources like Azure Blob Storage, Azure Data Lake, SQL
Databases, etc.
o Automation
of Data Workflows: Automate data ingestion, transformation, and loading
processes through Azure Data Factory (ADF), Databricks workflows, and Azure
Functions.
o Real-time
Data Processing: Implement real-time streaming data pipelines using Databricks
Structured Streaming for use cases such as IoT or event-driven architectures.
• Data
Transformation & Modeling
o Data
Cleaning & Transformation: Leverage Apache Spark in Databricks to process
large datasets efficiently, performing data cleansing, transformation, and
enrichment.
o Data
Modeling: Design and implement dimensional data models (e.g., star schema)
optimized for performance and querying in Azure Synapse Analytics or other
reporting layers.
o Delta
Lake Implementation: Use Delta Lake for reliable and scalable ACID-compliant
data storage and to optimize data for batch and stream processing.
• Optimization
& Performance Tuning
o Optimize
Data Processing: Tune Databricks notebooks and jobs for performance, leveraging
Databricks' autoscaling features and optimizing Apache Spark configurations for
specific workloads.
o Data
Partitioning & Indexing: Implement best practices for partitioning large
datasets, managing table storage formats (Parquet, Delta), and indexing data
for faster querying.
o Cluster
Management: Manage Databricks clusters (autoscaling, sizing, and costs),
ensuring efficient resource utilization on Azure.
• Collaboration
& Integration with Azure Services
o Azure
Integration: Integrate Databricks with other Azure services like Azure Data
Lake Storage, Azure SQL, Azure Synapse, Azure Key Vault (for security), and
Power BI for seamless data flow and analysis.
o Monitoring
& Alerts: Set up monitoring, alerting, and logging of data pipelines using
tools like Azure Monitor, Databricks Jobs, and Azure Log Analytics to ensure
smooth operations.
• Data
Governance & Security
o Data
Security: Ensure security measures such as data encryption, role-based access
control (RBAC), and compliance with GDPR and other regulations using Azure
Active Directory (AAD) and Databricks Secrets.
o Data
Quality Management: Implement data quality checks in Databricks pipelines,
ensuring consistency, accuracy, and validity of data.
o Version
Control & CI/CD: Use version control systems like Git and implement CI/CD
pipelines using Azure DevOps for Databricks notebooks and data workflows.
• Collaboration
& Documentation
o Cross-functional
Collaboration: Collaborate with data scientists, analysts, and business users
to develop insights and solutions that drive business objectives.
o Documentation:
Maintain thorough documentation of data architectures in DF Confluence,
processes, and pipelines in Databricks for future scalability and team
collaboration.
• Advanced
Analytics & Machine Learning
o Machine
Learning Integration: Collaborate with data science teams to build and deploy
machine learning models using Databricks MLflow and integrate with Azure's
machine learning services for operationalization.
o Data
Exploration: Support exploratory data analysis and business intelligence needs
using Databricks Notebooks and integrate with Azure Power BI or other
visualization tools.
Requirements
How well do you match this job?
Free account needed to apply or save. It takes a minute, and we'll bring you right back to this job.