Sr. Data Engineer

Job Description

As a skilled Senior Data Engineer with 5-8 years of experience, your role will involve designing, building, and maintaining efficient and scalable data pipelines to facilitate data integration and transformation across various sources. Your proficiency in relational databases such as MySQL and PostgreSQL, as well as NoSQL databases like MongoDB and Cassandra, will be crucial for managing and optimizing large datasets effectively.

Key Responsibilities:
- Design, build, and maintain efficient and scalable data pipelines to support data integration and transformation.
- Work with relational and NoSQL databases to manage and optimize large datasets efficiently.
- Utilize Apache Spark for distributed data processing and real-time analytics.
- Implement and manage Kafka for data streaming and real-time data integration.
- Collaborate with cross-functional teams to translate business requirements into technical solutions.
- Monitor and optimize the performance of data pipelines and architectures to ensure high availability and reliability.
- Ensure data quality, consistency, and integrity across all systems.
- Stay updated with the latest trends and best practices in data engineering and big data technologies.

Qualifications Required:
- 5-8 years of experience in data engineering, focusing on building and managing data pipelines.
- Strong proficiency in relational and NoSQL databases.
- Experience with data warehouses like Snowflake and Redshift.
- Experience in processing unstructured data stored in S3 using tools like Athena and Glue.
- Hands-on experience with Kafka for real-time data streaming.
- Understanding of ETL processes, data integration, and data pipeline optimization.
- Proficiency in programming languages like Python, Java, or Scala for data processing.
- Experience with Apache Spark for big data processing and analytics.
- Excellent problem-solving skills, attention to detail, and strong communication and collaboration skills.

If applicable, familiarity with cloud platforms like AWS, GCP, or Azure for data infrastructure would be beneficial for this role.,