Senior Data Engineer - ETL/Python

Responsibilities : - Lead the design and development of new ETL and data transformation jobs using Python and PySpark. - Build scalable, reliable, and efficient data pipelines to ingest, process, and transform large datasets. - Optimize existing data pipelines and workflows for performance and efficiency. - Collaborate closely with data scientists, analysts, and other stakeholders to understand data requirements and translate them into technical solutions. - Troubleshoot and resolve complex data-related issues within the ETL processes. - Implement and maintain data quality checks and monitoring systems. - Contribute to the definition and enforcement of data engineering best practices and standards. - Evaluate and adopt new data engineering technologies and tools to improve our data infrastructure. - Mentor and guide junior data engineers within the team. Required Skills: - Extensive experience (5 years) in designing, developing, and implementing ETL and data transformation jobs. - Strong proficiency in Python and PySpark for data processing and manipulation. - Deep understanding of data warehousing concepts and data modeling techniques. - Experience with various data storage solutions (i.e., SQL databases, NoSQL databases, cloud data warehouses). - Familiarity with workflow orchestration tools (i.e., Apache Airflow, Luigi). - Excellent problem-solving and analytical skills. - Strong communication and collaboration abilities. Preferred Skills : - Experience with cloud platforms (i.e., AWS, Azure, GCP) and their data engineering services. - Knowledge of data streaming technologies (i.e., Kafka, Spark Streaming). - Experience with DevOps practices and CI/CD pipelines (ref:hirist.tech)

Location: in, IN

Posted Date: 5/1/2025