Databricks Subject Matter Expert

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Databricks Subject Matter Expert based in the United States.

We are seeking a highly experienced Databricks Subject Matter Expert to lead enterprise-level data engineering, analytics, and data management initiatives. In this role, you will leverage your technical expertise to design and implement scalable solutions for collecting, validating, transforming, and visualizing complex datasets. You will work extensively with Databricks, Python, SQL, and distributed processing technologies to turn large volumes of structured and unstructured data into actionable insights. You will also contribute to machine learning initiatives, develop advanced analytical methodologies, and solve complex technical and business challenges. Working independently in a fast-paced, entrepreneurial environment, you will collaborate with technical and business stakeholders to deliver high-quality information systems and data solutions. This is an opportunity to apply your expertise to impactful projects while supporting mission-critical data and analytics operations.

Accountabilities:

Lead and provide subject matter expertise for data collection, validation, visualization, and analytics initiatives.
Apply structured methodologies to plan, analyze, design, and develop enterprise-wide information systems and data engineering solutions.
Develop advanced analytical techniques and methodologies to address complex business and technical challenges.
Conduct strategic systems planning, business information planning, and business analysis to support organizational objectives.
Organize, process, and analyze large volumes of structured and unstructured datasets using advanced data analytics tools and technologies.
Locate, access, integrate, merge, clean, and standardize data from multiple sources while developing meaningful derived metrics.
Design, develop, and implement data collection, processing, and analysis tools using Python, Databricks, SQL, Scala, R, and Java.
Design, script, debug, test, and analyze data engineering solutions to ensure accuracy, reliability, and performance.
Develop and implement machine learning-based tools, workflows, and processes to enhance analytical capabilities.
Apply distributed and parallel processing technologies, particularly Apache Spark, to process large datasets and support complex big data analytics requirements.
Perform SQL Server data imports and manage data ingestion from CSV and TXT files.
Leverage Microsoft Excel and Google Workspace tools for data analysis, reporting, and collaboration.
Utilize supporting technologies and platforms, including Pentaho for data integration and transformation, Azure Data Studio, GitHub, and Smartsheet, as required.
Document technical requirements, project activities, completed tasks, workflows, methodologies, and implementation details accurately.
Communicate technical findings, project progress, and analytical recommendations effectively to technical teams and business stakeholders.
Operate independently as a technical subject matter expert, taking ownership of assigned initiatives and delivering solutions in a dynamic, fast-paced environment.
Requirements:
Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related technical discipline.
At least 7 years of professional experience, including a minimum of 3 years of specialized experience in data engineering, data analytics, or related technical fields.
Active Secret security clearance is required.
Strong expertise in organizing, processing, and analyzing large volumes of structured and unstructured data using advanced analytical tools.
Demonstrated experience locating, accessing, integrating, merging, cleaning, and standardizing data from multiple sources, as well as developing derived metrics.
Advanced proficiency in Python, Databricks, and SQL, with working knowledge of Scala, R, and Java for data collection, engineering, and analytical applications.
Strong experience designing, scripting, debugging, implementing, and maintaining data engineering solutions.
Practical experience implementing and developing machine learning-based tools, models, or processes.
Solid understanding of distributed and parallel processing technologies, particularly Apache Spark, and their application to large-scale data processing and big data analytics.
Experience working with SQL Server and importing data from CSV and TXT files.
Strong proficiency in Microsoft Excel and Google Workspace applications.
Familiarity with Pentaho, Azure Data Studio, GitHub, and Smartsheet is an advantage.
Excellent technical documentation skills, with the ability to clearly record project requirements, methodologies, completed work, and technical processes.
Strong written and verbal communication skills, with the ability to translate complex technical concepts into clear information for different stakeholders.
Excellent analytical thinking, problem-solving, and organizational skills.
Ability to work independently, manage competing priorities, and take initiative in a fast-paced and entrepreneurial environment.
Strong attention to detail and commitment to delivering accurate, reliable, and high-quality data solutions.
Ability to collaborate effectively with multidisciplinary technical and business teams.
Work Environment and Physical Requirements:
Ability to remain seated or stationary for extended periods while working at a computer.
Comfortable operating standard office equipment, including computers and telephones.
Ability to communicate effectively through in-person meetings, telephone, video conferences, and email.
Ability to occasionally move around an office environment and lift objects weighing up to 20 pounds.
Ability to maintain the assigned work schedule and demonstrate regular, reliable attendance.
Availability for occasional domestic travel of up to 10%, as business requirements dictate.
Benefits:
Target annual salary: $150,000 USD.
Fully remote working arrangement.
Three comprehensive medical insurance plan options for full-time employees, including a Health Savings Account (HSA) option.
Two levels of dental insurance coverage.
Vision insurance.
Life and Accidental Death and Dismemberment (AD&D) insurance.
Short-term and long-term disability coverage.
Additional accident, hospital indemnity, and critical illness insurance options.
401(k) retirement savings plan with employer matching contributions.
Paid time off to support work-life balance.
Employer-sponsored professional training programs and certification opportunities.
Pet insurance benefits.
Commuter transit benefits.
Free subscription to NEOU, a virtual fitness and exercise platform.
Employee wellness and engagement initiatives through a dedicated employee care program, including:
30-, 60-, and 90-day wellness check-ins.
Personal development opportunities.
Financial management and stress management seminars.
Formal mentorship program to support professional development and career progression.
Job shadowing and cross-training opportunities.
Employee Brand Ambassador program.
Employee Assistance Program (EAP), providing access to counseling, legal guidance, financial planning, and other support resources.
Regular monthly team-building events.
Annual wellness challenges and activities promoting physical health, community engagement, and overall well-being.
Opportunities to participate in charitable initiatives and community involvement programs.