Databricks Developer - Java/Scala/Python

About the Role : We are seeking a highly skilled and experienced Databricks Developer to join our growing data engineering team. The ideal candidate will possess a strong understanding of Azure Databricks and its ecosystem, with a proven ability to design, develop, and optimize scalable data pipelines. You will collaborate with cross-functional teams to deliver robust and efficient data solutions that drive business insights. Responsibilities : Data Pipeline Development : - Design, develop, and implement scalable data pipelines using Azure Databricks (Spark, Delta Lake) to ingest, transform, and store structured and unstructured data from various sources. - Utilize Python, Scala, and SQL for data processing and transformation within Databricks notebooks and jobs. - Develop and maintain data pipelines for real-time and batch processing. Performance Optimization : - Analyze, optimize, and tune existing data pipelines for performance, reliability, and efficiency. - Implement best practices for data partitioning, caching, and query optimization. - Monitor and troubleshoot data pipeline performance issues. Collaboration & Communication : - Collaborate with data scientists, analysts, functional consultants, and other stakeholders to understand business requirements and translate them into technical solutions. - Communicate effectively with technical and non-technical audiences. - Work within agile development teams. Azure Service Expertise : - Provide expertise and guidance in selecting appropriate Azure services and tools (e.g., Azure Data Lake Storage, Azure Event Hubs, Azure Data Factory) to build robust data solutions. - Integrate Databricks with other Azure services for seamless data flow. Security & Compliance : - Implement security measures and standard processes to ensure data privacy and compliance with regulatory standards. - Manage access control and data encryption within Databricks. Troubleshooting & Support : - Solve issues related to data processing, performance, and system failures. - Provide support for data pipelines and related infrastructure. Continuous Learning & Improvement : - Stay updated with the latest Azure Databricks features, tools, and best practices. - Provide recommendations for improvements to data pipelines and infrastructure. - Participate in code reviews. Required Skills : Experience : 4 years of experience in data engineering with a focus on Azure Databricks. Azure Databricks : - Strong proficiency in Azure Databricks (Spark, Delta Lake). - Experience with Databricks notebooks, jobs, and workflows. - Knowledge of Databricks clusters and resource management. Programming Languages : - Proficiency in Python, Scala, and SQL. - Experience with data manipulation libraries (e.g., Pandas, Spark SQL). Azure Services : - Experience with Azure Data Lake Storage, Azure Event Hubs, Azure Data Factory, and other related Azure services. - Understanding of Azure cloud architecture. Data Warehousing & ETL : - Strong understanding of data warehousing and ETL/ELT concepts. - Experience with data modeling and schema design. Data Pipeline Development : - Proven ability to design, develop, and optimize scalable data pipelines. Problem-Solving : - Excellent problem-solving and analytical skills. Communication : - Strong verbal and written communication skills. Preferred Skills : - Experience with CI/CD pipelines. - Knowledge of data governance and data quality practices. - Experience with cloud security best practices. - Experience with other cloud platforms like AWS or GCP. - Experience with stream processing. Personal Attributes : - Highly motivated and self-directed. - Strong team player with the ability to collaborate effectively. - Detail-oriented and organized. - Ability to work under pressure and meet deadlines. - Commitment to continuous learning and professional development. (ref:hirist.tech)

Location: in, IN

Posted Date: 5/1/2025