Job Overview:
The Senior Data Engineer plays a pivotal role in the design, development, and optimization of robust data pipelines and infrastructure that power AI-driven and data-intensive applications.
Responsibilities:
Data Architecture & Pipeline Development
- Design, develop, and maintain scalable, efficient, and secure ETL (Extract, Transform, Load) pipelines.
- Build real-time and batch data processing frameworks for AI, analytics, and business intelligence.
- Optimize data storage, retrieval, and performance across structured and unstructured data sources.
Data Governance & Security
- Implement data governance frameworks, ensuring data integrity, consistency, and compliance with security standards.
- Monitor and enforce best practices for data privacy, encryption, and regulatory compliance (GDPR, CCPA, etc.).
Collaboration & Integration
- Work closely with Data Scientists and Machine Learning Engineers to support AI model training, inference, and deployment.
- Partner with DevOps teams to automate data workflows and ensure smooth integration with cloud platforms.
- Act as a liaison between business stakeholders and technical teams, translating business needs into data solutions.
Performance Optimization & Monitoring
- Continuously optimize data infrastructure for performance, reliability, and scalability.
- Implement monitoring solutions to detect and resolve data pipeline issues proactively.
Qualifications:
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
- 3+ years of experience in data engineering, including ETL pipeline development.
- Expertise in SQL and NoSQL databases (PostgreSQL, MongoDB, Cassandra, etc.).
- Proficiency in big data technologies such as Apache Spark, Kafka, and Hadoop.
- Hands-on experience with cloud platforms (AWS, Azure, GCP) and data warehousing (BigQuery, Snowflake, Redshift).
- Strong knowledge of data orchestration tools (Airflow, Prefect, or similar).
- Experience with data security best practices and compliance requirements.
- Excellent problem-solving skills and the ability to work in a fast-paced environment.