Data Engineer
Overland Park, KS, United StatesYesterday
Key details
- Location
- Overland Park, KS, United States
- Work style
- Onsite
- Level
- Mid level
- Experience
- 2–4 yrs
- Function
- Data & ML
- Focus area
- Data engineering & Analytics
- Posted
- Yesterday
Skills
airflowapi gatewayavroawsaws lambdabusiness intelligenceci/cddata architecturedata pipelinesdata qualitydata securitydata validationdata visualizationdatabricksdelta lakedockerecsetlexcelfastapigitgithubgithub actionsgitlabgrafanainfrastructure as codekafkalakehousemicroservicesmlflowmongodbnosqlperformance tuningpostgresqlpower bipysparkpythons3sparksqlsql serverterraform
About the role
About the Role
Shamrock Trading Corporation is looking for a Data Engineer who wants to utilize their expertise in modern data platforms, microservices architectures, application programming interfaces, data pipeline development/support and analytical reporting skills by joining our Data Services team. This role is responsible for gathering and analyzing data from several internal and external sources, designing a cloud-focused data platform for analytics and business intelligence, reliably providing data to our analysts.
This role requires a significant understanding of data engineering and analytical techniques. An ideal candidate will have strong technical capabilities, business acumen, and the ability to work effectively with cross-functional teams
What You’ll Do
Develop & Maintain Scalable Data Pipelines: Build, optimize, and maintain ETL/ELT pipelines using Databricks, Apache Spark, and Delta Lake.
Optimize Data Processing: Implement performance tuning techniques to improve Spark-based workloads.
Cloud Data Engineering: Work with AWS services (S3, ECS, Lambda, Glue, etc.) to design and implement robust data architectures.
Real-time & Streaming Data: Develop streaming solutions using Kafka and Databricks Structured Streaming.
Data Quality & Governance: Implement data validation, observability, and governance best practices using Unity Catalog or other tools.
Cross-functional Collaboration: Partner with analysts, data scientists, and application engineers to ensure data meets business needs.
Automation & CI/CD: Implement infrastructure-as-code (IaC) and CI/CD best practices for data pipelines using tools like Terraform, and GitHub Actions.
What You’ll Bring
Bachelor’s degree in computer science, data science or related technical field, or equivalent practical experience
2-4 years of experience in data engineering, with a focus on cloud-based platforms.
Strong hands-on experience with Databricks (including Spark, Delta Lake, and MLflow).
Experience building and maintaining AWS based data pipelines: currently utilizing AWS Lambda, Docker / ECS, MSK, Airflow, Databricks, Unity Catalog
Development experience utilizing one or more of the following:
Git ecosystem (GitHub, GitLab, etc.)
Python
Databricks (PySpark, SQL, Spark Declarative Pipelines/DLT, Databricks Asset Bundles)
Kafka and the Kafka Connect ecosystem (Schema Registry,Avro, etc.)
Familiarity with CI/CD for data pipelines and infrastructure as code (GitHub Actions, Terraform)
Strong SQL skills for data transformation and performance tuning.
Understanding of data security and governance best practices.
Enthusiasm for working directly with customer teams (Business units and internal IT)
What Will Set You Apart
Proven experience with relational and NoSQL databases (e.g. Postgres, Microsoft SQL Server, MongoDB, etc.)
Familiarity with data lakehouse architectures and optimization strategies.
Familiarity with API architectures such as FastAPI, AWS API Gateways, and OpenAPI Specifications
Familiarity with data visualization techniques using tools such as Databricks Dashboards, Grafana, PowerBI, and Excel.
Shamrock Trading Corporation is looking for a Data Engineer who wants to utilize their expertise in modern data platforms, microservices architectures, application programming interfaces, data pipeline development/support and analytical reporting skills by joining our Data Services team. This role is responsible for gathering and analyzing data from several internal and external sources, designing a cloud-focused data platform for analytics and business intelligence, reliably providing data to our analysts.
This role requires a significant understanding of data engineering and analytical techniques. An ideal candidate will have strong technical capabilities, business acumen, and the ability to work effectively with cross-functional teams
What You’ll Do
Develop & Maintain Scalable Data Pipelines: Build, optimize, and maintain ETL/ELT pipelines using Databricks, Apache Spark, and Delta Lake.
Optimize Data Processing: Implement performance tuning techniques to improve Spark-based workloads.
Cloud Data Engineering: Work with AWS services (S3, ECS, Lambda, Glue, etc.) to design and implement robust data architectures.
Real-time & Streaming Data: Develop streaming solutions using Kafka and Databricks Structured Streaming.
Data Quality & Governance: Implement data validation, observability, and governance best practices using Unity Catalog or other tools.
Cross-functional Collaboration: Partner with analysts, data scientists, and application engineers to ensure data meets business needs.
Automation & CI/CD: Implement infrastructure-as-code (IaC) and CI/CD best practices for data pipelines using tools like Terraform, and GitHub Actions.
What You’ll Bring
Bachelor’s degree in computer science, data science or related technical field, or equivalent practical experience
2-4 years of experience in data engineering, with a focus on cloud-based platforms.
Strong hands-on experience with Databricks (including Spark, Delta Lake, and MLflow).
Experience building and maintaining AWS based data pipelines: currently utilizing AWS Lambda, Docker / ECS, MSK, Airflow, Databricks, Unity Catalog
Development experience utilizing one or more of the following:
Git ecosystem (GitHub, GitLab, etc.)
Python
Databricks (PySpark, SQL, Spark Declarative Pipelines/DLT, Databricks Asset Bundles)
Kafka and the Kafka Connect ecosystem (Schema Registry,Avro, etc.)
Familiarity with CI/CD for data pipelines and infrastructure as code (GitHub Actions, Terraform)
Strong SQL skills for data transformation and performance tuning.
Understanding of data security and governance best practices.
Enthusiasm for working directly with customer teams (Business units and internal IT)
What Will Set You Apart
Proven experience with relational and NoSQL databases (e.g. Postgres, Microsoft SQL Server, MongoDB, etc.)
Familiarity with data lakehouse architectures and optimization strategies.
Familiarity with API architectures such as FastAPI, AWS API Gateways, and OpenAPI Specifications
Familiarity with data visualization techniques using tools such as Databricks Dashboards, Grafana, PowerBI, and Excel.
How well do you match this job?
Skills
Experience
Level
Title
Location
Salary
Freshness
Similarity
Free account needed to apply or save. It takes a minute, and we'll bring you right back to this job.