Data Engineer

Job Description

About the job Data Engineer
Job Title: Data Engineer
Experience: 8 to 10 Years
?Time Zone: IST Time??
Job Type: Remote
Work Location: –
?Domain: –
??
Responsibilities:
? Data Pipeline Architecture: Design, develop, and optimize end-to-end data pipelines to extract, transform, and load (ETL) data from various sources into our data warehouse. Ensure data quality, reliability, and performance throughout the pipeline.
? Data Modeling and Schema Design: Work with data scientists, analysts, and stakeholders to understand data requirements and create scalable and efficient data models. Implement and maintain database schemas that facilitate easy data access and querying.
? Data Integration: Integrate data from diverse internal and external sources, including databases, APIs, and third-party systems. Build connectors and adaptors to ensure seamless data flow between systems.
? Performance Optimization: Continuously monitor and fine-tune the performance of data pipelines and databases… Identify bottlenecks and implement optimizations to enhance processing speed and resource utilization.
? Data Security and Governance: Implement robust security measures to safeguard sensitive data. Ensure compliance with data protection regulations and industry best practices for data governance and privacy.
? Data Transformation and Enrichment: Develop data transformation routines to enrich raw data and make it suitable for analytical processing. Apply data cleansing, aggregation, and normalization techniques as needed.
? Data Monitoring and Error Handling: Establish monitoring systems to detect data inconsistencies, anomalies, and errors. Develop automated alerts and error handling processes to ensure data integrity.
? Technology Evaluation and Implementation: Stay up-to-date with the latest data engineering technologies and best practices. Evaluate new tools and frameworks, and lead the implementation of suitable technologies to improve data processing efficiency and scalability.
? Documentation and Collaboration: Maintain comprehensive documentation for data engineering processes, data dictionaries, and workflows. Collaborate with cross-functional teams to understand their data needs and deliver effective solutions.
? Mentoring and Leadership: Mentor and provide guidance to junior data engineering team members. Act as a technical leader, driving innovation and best practices within the data engineering team.
Requirements:
? Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
? Proven track record of at least 8 years of experience in data engineering roles.
? Expertise in building and maintaining data pipelines using ETL tools like Apache Spark, Apache Airflow, or similar.
? Strong proficiency in SQL and database technologies (e.g., PostgreSQL, MySQL, NoSQL databases).
? Extensive experience with cloud-based data platforms, such as AWS, Azure, or Google Cloud Platform.
? Solid understanding of data modeling concepts and data warehousing principles.
? Proficiency in at least one programming language (e.g., Python, Java) for data manipulation and automation.
? Experience with data streaming technologies (e.g., Kafka, Kinesis) is a plus.
? Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes) is beneficial.
? Excellent problem-solving skills and the ability to work independently and as part of a team.
? Strong communication and interpersonal skills to collaborate effectively with stakeholders across the organization