Pyspark Developer - Big Data Apps

Position Title : Senior PySpark Developer Experience Range : 6 to 10 Years Location : Chennai Job Type : Full-Time Job Overview : We are looking for an experienced Senior PySpark Developer to join our data engineering team. This role involves designing and building high-performance data processing systems using Apache Spark (PySpark) and other big data technologies. The ideal candidate will have a deep understanding of data engineering principles, big data ecosystem, and cloud-based data solutions. You will play a critical role in developing scalable data pipelines, optimizing performance, and ensuring data integrity across platforms. Key Responsibilities : - Big Data Application Development : Design, develop, and maintain distributed data processing solutions using Apache Spark and Python (PySpark). - Data Pipeline Engineering : Build and manage end-to-end data pipelines for ingestion, transformation, and delivery of structured and unstructured data. - Performance Optimization : Tune and optimize Spark applications for speed, efficiency, and fault tolerance, especially with large-scale datasets. - Collaboration : Work closely with data engineers, analysts, data scientists, and business stakeholders to understand requirements and translate them into scalable solutions. - Data Quality & Governance : Implement best practices for data quality, integrity, and compliance in all stages of the data pipeline. - Automation & Orchestration : Develop automated workflows using tools like Apache Airflow, and integrate with CI/CD pipelines. - Cloud Integration : Design and deploy data solutions in cloud environments such as AWS, Azure, or GCP. - Troubleshooting & Support : Identify and resolve bottlenecks, failures, and latency issues in distributed systems. - Technical Leadership : Mentor junior developers and contribute to code reviews, architecture discussions, and best practice implementation. - Continuous Learning : Stay up to date with the latest trends in big data, cloud computing, and data engineering technologies. Required Skills & Qualifications : - 6 years of hands-on experience in data engineering with a focus on Apache Spark and Python (PySpark). - Proven ability to build and maintain scalable and high-performance big data applications. - Strong knowledge of big data frameworks like Hadoop, Hive, and Kafka. - Experience with cloud platforms such as AWS (Glue, EMR, S3), Azure (Databricks, Data Lake), or GCP. - Expertise in SQL, data warehousing, and data modeling concepts. - Proficiency with workflow orchestration tools such as Apache Airflow or similar. - Familiarity with CI/CD pipelines and tools like Jenkins, Git, etc. - Hands-on experience with containerization tools like Docker and orchestration using Kubernetes. - Understanding of distributed computing, parallel processing, and fault-tolerant system design. - Excellent problem-solving skills and the ability to work independently in a fast-paced environment. - Strong communication skills to effectively collaborate across teams. Preferred Qualifications : - Experience leading or mentoring data engineering teams. - Familiarity with Spark Structured Streaming and real-time data processing. - Knowledge of machine learning workflows and integration with data pipelines. - Certifications in Big Data (Cloudera, Hortonworks) or Cloud Platforms (AWS, Azure, GCP). - Experience in implementing data governance and compliance standards. (ref:hirist.tech)

Location: chennai, IN

Posted Date: 5/1/2025