Data Engineer - Spark/Hadoop

Job Summary : We are seeking a highly skilled Sr. Data Engineer / Lead Data Engineer / Associate Architect to lead the design, implementation, and optimization of scalable data architectures. The ideal candidate will have a deep understanding of data modeling, ETL processes, cloud data solutions, and big data technologies. You will work closely with cross-functional teams to build robust, high-performance data pipelines and infrastructure to enable data-driven decision-making. Experience : 5 - 12 years. Work Location : Hyderabad (Hybrid). Mandatory skills : AWS, Python, SQL, Airflow, DBT. Responsibilities : - Design and Develop scalable and resilient data architectures that support business needs, analytics, and AI/ML workloads. - Data Pipeline Development : Design and implement robust ETL/ELT processes to ensure efficient data ingestion, transformation, and storage. - Big Data & Cloud Solutions : Architect data solutions using cloud platforms like AWS, Azure, or GCP, leveraging services such as Snowflake, Redshift, BigQuery, and Databricks. - Database Optimization : Ensure performance tuning, indexing strategies, and query optimization for relational and NoSQL databases. - Data Governance & Security : Implement best practices for data quality, metadata management, compliance (GDPR, CCPA), and security. - Collaboration & Leadership : Work closely with data engineers, analysts, and business stakeholders to translate business requirements into scalable solutions. - Technology Evaluation : Stay updated with emerging trends, assess new tools and frameworks, and drive innovation in data engineering. Required Skills : - Education : Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field. - Experience : 6 12 years of experience in data engineering. - Cloud Platforms : Strong expertise in AWS data services. - Big Data Technologies : Experience with Hadoop, Spark, Kafka, and related frameworks. - Databases : Hands-on experience with SQL, NoSQL, and columnar databases such as PostgreSQL, MongoDB, Cassandra, and Snowflake. - Programming : Proficiency in Python, Scala, or Java for data processing and automation. - ETL Tools : Experience with tools like Apache Airflow, Talend, DBT, or Informatica. - Machine Learning & AI Integration (Preferred) : Understanding of how to architect data solutions for AI/ML applications (ref:hirist.tech)

Location: hyderabad, IN

Posted Date: 5/1/2025