DataOps Engineer - Johannesburg, South Africa - The Skills Mine

Description

Requirements

• An MS or BS degree or the equivalent practical experience
• 3+ years' experience working as a DataOps Engineer or in a similar role, with a strong focus on data engineering and automation
• 3+ years of experience using Python to develop enterprise-grade software using appropriate design patterns and principles
• Strong experience with cloud platforms such as AWS, Azure, or GCP, and expertise in utilizing cloud-based data services (e.g., AWS Glue, Azure Data Factory, Google Cloud Dataflow)
• Solid understanding of modern data architecture, including data modelling, ETL/ELT processes, and data warehousing concepts
• Experience with version control systems (e.g., Git), containerization technologies (e.g., Docker/Kubernetes), infrastructure as code (e.g., terraform), and CI/CD pipelines
• Familiarity with data governance, data security, and data privacy practices

Responsibilities

• Designed, developed, and maintained scalable data pipelines and workflows to support data acquisition, processing, and transformation activities in an agile and iterative manner
• Collaborated with data scientists, software engineers, and other stakeholders to understand data requirements and implemented efficient data solutions to foster team self-service and improve collaboration and communication between departments
• Implemented and optimized data integration processes, ensuring data consistency, accuracy, and reliability across different data sources and systems
• Automated data operations tasks, including data ingestion, data validation, and data quality monitoring, using appropriate tools and technologies
• Identified and resolved data-related issues, troubleshoot performance bottlenecks, and ensured optimal data processing and storage