Renault Nissan Technical Business Centre
|Python Developer (Data Engineer)
Chennai, India
Summary
Spearheaded the development and optimization of cloud-native ETL pipelines and data workflows, leveraging Python, PySpark, and Databricks to process vast datasets for autonomous vehicle simulation platforms on GCP.
Highlights
Developed and optimized Python and PySpark data pipelines using Pandas, NumPy, and SQLAlchemy to process over 8M+ km of autonomous driving simulation data.
Orchestrated Databricks Jobs and Workflows with PySpark and Spark SQL to build ETL pipelines, processing 500,000+ simulation scenarios across 23 projects and 450+ terrain configurations.
Led the migration of production data infrastructure from Azure to GCP, re-architecting Pub/Sub messaging, Cloud Storage, and Cloud Logging pipelines across 10+ production flows with zero data loss.
Operated and monitored GCP Pub/Sub, Cloud Storage, Cloud Logging, and PostgreSQL databases, proactively resolving production data-flow issues to maintain pipeline health.
Standardized Git version control and automated deployment via GitLab CI/CD pipelines, building Docker images and integrating HashiCorp Vault for secure secret management.
Authored pytest and mock-based unit tests to validate ETL and pipeline logic, integrating GitHub Copilot into Agile workflows to accelerate learning and delivery.
Mentored and supported a 7-engineer data engineering team as Team Leader, directing JIRA task allocation and Confluence documentation, fostering continuous skill development.