Big Data Engineer

About the Job : Are you a seasoned data engineering professional with a passion for building robust and scalable data pipelines? Do you thrive in transforming raw data into valuable insights using cutting-edge big data technologies? If so, we have an exciting opportunity for you to join our dynamic team as a Big Data Engineer. You will play a crucial role in designing, developing, and maintaining our big data infrastructure, ensuring the quality and reliability of data that drives critical business Title : Big Data : PAN : 7 : - Design and Development of Data Pipelines : Architect, develop, and maintain scalable and efficient ETL (Extract, Transform, Load) pipelines using big data technologies such as Hadoop, Spark, and Hive. - Data Integration and Quality : Implement best practices for data integration and ensure adherence to data quality principles throughout the data lifecycle. - Testing and Validation : Analyze user stories to build comprehensive test plans, develop robust codebases, and implement business rules for rigorous data testing and validation. - Performance Optimization : Identify and implement optimizations to improve the performance and efficiency of data pipelines and big data systems. - Technology Implementation : Implement and integrate new big data technologies and tools as required. - Collaboration and Communication : Work closely with data scientists, analysts, and other stakeholders to understand their data requirements and provide effective data solutions. - Clearly communicate technical concepts to both technical and non-technical audiences. - CI/CD Implementation : Contribute to the design and implementation of continuous integration and continuous deployment (CI/CD) pipelines for data engineering processes. - Agile Participation : Actively participate in Agile/Scrum ceremonies, contributing to sprint planning, daily stand-ups, and retrospectives. - Problem Solving : Troubleshoot and resolve complex data-related issues in a timely and efficient manner. - Documentation : Create and maintain comprehensive documentation for data pipelines, processes, and systems. - Staying Current : Keep abreast of the latest advancements in big data technologies and industry best : - Bachelor's or Master's degree in Computer Science, Engineering, or a related field. - Minimum of 7 years of proven experience as a Development Data Engineer or in a similar role with a strong ETL background. - Extensive hands-on experience with big data technologies including : 1. Hadoop ecosystem (HDFS, MapReduce, YARN) 2. Spark (Core, SQL, Streaming, MLlib) 3. Hive - Solid understanding and practical experience with ETL tools and processes. - Proficiency in at least one programming language relevant to big data processing, preferably Python (PySpark). - Strong understanding of data integration patterns and best practices. - Experience in designing and implementing data quality checks and validation processes. - Knowledge of continuous integration and continuous deployment (CI/CD) pipelines. - Familiarity with Agile/Scrum development methodologies. - Excellent analytical and problem-solving skills with a keen attention to detail. - Strong written and verbal communication skills, with the ability to articulate technical concepts clearly. - Excellent collaboration and teamwork Points For : - Experience with cloud-based big data platforms (e.g., AWS EMR, Azure HDInsight, Google Cloud Dataproc). - Familiarity with other NoSQL databases (e.g., HBase, Cassandra, MongoDB). - Experience with data warehousing concepts and tools. - Knowledge of data governance and security best practices. - Contributions to open-source big data projects (ref:hirist.tech)

Location: bangalore, IN

Posted Date: 5/9/2025