Databricks Data Engineer

About the Role We are looking for a highly skilled Senior Data Engineer with strong expertise in Apache Spark and Databricks to join our growing data engineering team. You will be responsible for designing, developing, and optimizing scalable data pipelines and applications using modern cloud data technologies. This is a hands-on role requiring deep technical knowledge, strong problem-solving skills, and a passion for building efficient, high-performance data solutions that drive business value. Responsibilities:  Design, develop, and implement scalable data pipelines and applications using Apache Spark and Databricks, adhering to industry best practices.  Perform in-depth performance tuning and optimization of Spark applications within the Databricks environment.  Troubleshoot complex issues related to data ingestion, transformation, and pipeline execution.  Collaborate with cross-functional teams including data scientists, analysts, and architects to deliver end-to-end data solutions.  Continuously evaluate and adopt new technologies and tools in the Databricks and cloud ecosystem.  Optimize Databricks cluster configurations for cost-effectiveness and performance.  Apply data engineering principles to enable high-quality data ingestion, transformation, and delivery processes.  Document technical designs, development processes, and operational procedures. Qualifications:  Bachelor’s degree in Computer Science, Engineering, or a related field.  10 years of experience in data engineering or big data development.  5 years of hands-on experience with Apache Spark and Databricks.  Deep understanding of Spark internals, Spark Streaming, and Delta Lake.  Experience in developing solutions using Azure Data Services including: Azure Databricks, Azure Data Factory, Azure DevOps, Azure Functions, Azure SQL Database, Azure Event Grid, Cosmos DB.  Familiarity with Striim or similar real-time data integration platforms is a plus.  Proficient in PySpark or Scala.  Strong experience in performance tuning, cost optimization, and cluster management in Databricks.  Solid understanding of data warehousing, ETL/ELT pipelines, and data modelling.  Experience working with cloud platforms (Azure preferred; AWS/GCP is a plus).  Familiarity with Agile/Scrum methodologies. Preferred Qualifications  Databricks Certified Professional Data Engineer certification is a strong plus.  Strong communication skills—both written and verbal—with the ability to convey technical concepts to non-technical stakeholders.

Location: hyderabad, IN

Posted Date: 6/17/2025