Senior Data Engineer - Durban - Carlysle Human Capital

    Carlysle Human Capital
    Carlysle Human Capital Durban

    1 week ago

    Default job background
    Description

    Senior Data Engineer

    This role is for a Senior Data Engineer to help build and maintain scalable data pipelines and related systems in a research focused environment.

    You will be responsible for designing, developing, testing, and deploying technical platforms and data solutions that meet the business requirements and align with the scientific goals. You will collaborate with research scientists, internal IT and other stakeholders to ensure data quality, reliability, accessibility, security and governance.

    Responsibilities:

    • Design, develop, and maintain the end-to-end technical aspects (hardware - on prem and/or cloud) of a high performance compute and storage platform designed specifically for data pipelines in a research data intensive environment
    • Design, develop, and maintain the end-to-end data pipelines (software, DBs and processes) required to support the research scientists and data managers.
    • Support ETL processes including, data ingestion, transformation, validation, and integration processes using various tools and frameworks
    • Optimize data performance, scalability, and security
    • Provide technical guidance and support to data analysts and research scientists.
    • Design data integrations and data quality frameworks
    • Work and collaborate with the rest of the IT department to help develop the strategy for long term scientific Big Data platform architecture
    • Document and effectively communicate data engineering processes and solutions.
    • Make use of and help define the right cutting edge technology, processes and tools needed to drive technology within our science and research data management departments.

    Requirements:

    • Bachelor's degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field
    • Industry recognized IT related certification and technology qualification such as Databases and Data related certifications.
    • 7+ years experience in a Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing
    • Strong experience with technologies such as Hadoop, Kafka, Nifi or Spark or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery and Azure Synapse Analytics
    • At least 5 years advanced experience and very strong proficiency in UNIX, Linux, Windows Operating systems and preferably containerization technologies like Docker, Kubernetes etc
    • Working knowledge of various data related programming, scripting or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc.
    • Strong working experience with software compute and virtualization platforms such as VMware, Hyper-V, OpenStack, KVM etc.
    • Strong working experience with hardware compute platforms including high performance compute cluster hardware and related technologies.

    Desirable Skills:

    • Strong Experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL as well as NoSQL databases such as MongoDB, Cassandra etc.
    • Experience of Big Data technologies such as Hadoop, Spark and Hive
    • Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow
    • Experience working with containerization is advantageous
    • Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous
    • Experience working with Big Data Cloud based (AWS, Azure etc) technologies is advantageous
    • Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake advantageous
    • Strong Experience designing end-to-end data pipelines including the compute hardware infrastructure.
    • Strong knowledge of data modeling, architecture, and governance principles
    • Strong Linux Administration skills
    • Programming skills in various languages advantageous
    • Strong data security and compliancy experience
    • Excellent communication, collaboration, and problem-solving skills
    • Ability to work independently and as part of a cross-functional team
    • Interest and enthusiasm for medical scientific research and its applications
  • HR Genie

    Data Engineer

    1 hour ago

    HR Genie Durban

    Our Client a Global tech firm is seeking a Data Engineer to join their team in Durban on a contract basis. They offer stability, growth, attractive rates and a great working environment. · Database Engineer Requirements · Establishes database management systems, standards, guidel ...

  • Spar Group Limited

    Data Engineer

    3 days ago

    Spar Group Limited Durban

    Data Engineer - MS Fabric · A highly skilled Data Engineer is required to design, develop, and maintain cutting-edge data architecture and infrastructure for our business intelligence and analytics needs. · Key Responsibilities: · Design and implement data models and data structu ...

  • Ignition Group

    Data Engineer

    2 days ago

    Ignition Group Durban

    Job Summary · This role is responsible for designing, developing, and maintaining data pipelines, APIs, and data infrastructure to ensure data reliability, quality, and accessibility while optimizing performance and scalability. · Key Responsibilities · Technical Development and ...

  • Spar Group Limited

    Data Engineer

    3 days ago

    Spar Group Limited Durban

    Job Title: Data Engineer - SAP BW · The role of a Data Engineer is to design, develop and maintain cutting-edge reporting solutions within our data landscape. The position requires expertise in data modelling, HANA integration, and SAP modules. · Key Responsibilities: · SAP BW Ob ...

  • RELX

    Data Engineer III

    3 days ago

    RELX Durban

    Are you a strategic planner and problem-solver? · Do you enjoy collaborating cross-functionally to deliver on common goals? · We shape a more just world. · About Us · LexisNexis, part of RELX, is a leading global provider of legal, regulatory, and business information. We help cu ...

  • Careerbox

    Data Engineer

    3 days ago

    Careerbox Umhlanga Rocks

    Data Engineer Job Description · The data engineer plays a vital role in expanding and optimizing our data and data pipeline architecture. This includes ensuring optimal data delivery architecture is consistent throughout ongoing projects, supporting cross-functional teams with da ...

  • Careerbox

    Data Engineer Umhlanga

    2 weeks ago

    Careerbox Umhlanga Rocks

    Data Engineer Role Overview · The data engineer plays a crucial part in expanding and optimizing our data architecture, ensuring seamless data flow and collection for cross-functional teams. This professional will support software developers, database architects, data warehouse m ...

  • HR Genie

    Network Engineer

    16 hours ago

    HR Genie Durban

    Our Client, a Global Tech firm, is seeking a Network Engineer Data centre specialist to join their team in Durban on a contract basis. They offer stability, growth, attractive compensation, and a great working environment. · The Network Engineer - Data Centre Specialist is respon ...

  • NTT Data

    Network Engineer

    1 day ago

    NTT Data Durban

    Job Title: Network Engineer - Data Centre Specialist · The Network Engineer - Data Centre Specialist is a critical role responsible for designing, implementing, and maintaining the network infrastructure within data centres. This position ensures optimal performance, security, an ...

  • Pro Development Group

    Network Engineer

    6 days ago

    Pro Development Group Durban

    Job Description: · We are seeking a skilled Network Engineer – Data Centre Specialist to join our team in Durban on a contract basis. · Main Responsibilities: · Design and deploy data centre network infrastructure including routers, switches, firewalls, and load balancers. · Conf ...

  • SEIGO STAFFING

    Graduate Mechanical Engineer

    4 days ago

    SEIGO STAFFING Durban Full time

    Requirements: · Bachelor of Science in Engineering (Mechanical), Bachelor of Engineering (Mechanical) or Master of Engineering (Mechanical) · Candidate Engineer registration with ECSA · At least 0-3 years of experience in consulting engineering or contracting services, specializi ...

  • Express Employment Professionals (Durban South)

    cad engineer

    3 days ago

    Express Employment Professionals (Durban South) Durban

    Job Description · A Program Manager is required to oversee engineering changes and project deliverables in the plant team and client. · Main Responsibilities: · Track and update all engineering changes for the plant team and client on a regular basis · Monitor project deliverable ...

  • RELX

    Senior Software Engineer II

    1 week ago

    RELX Durban

    Senior Software Engineer II · We are looking for a skilled Senior Software Engineer to join our French team. · About our Company · LexisNexis Legal & Professional is a global provider of information-based analytics and decision tools for professional and business customers. · Abo ...

  • TalenCo Recruitment Specialists

    Data Scientist

    6 days ago

    TalenCo Recruitment Specialists Durban

    Job Summary · This role involves developing and deploying machine learning, AI, and algorithms to extract insights, predict outcomes, and optimize processes. The ideal candidate will have a strong background in data analysis, modeling, and strategy. · Key Responsibilities · Data ...

  • Ananzi

    Manager: Operations And Edtech Support-Durban

    3 days ago

    Ananzi Durban

    Job Title: Manager: Operations and EdTech Support · uMngeni Water Services (Pty) Ltd, a wholly owned subsidiary of uMngeni-uThukela Services (UUW), focuses primarily on marketing and expanding Section 30 Commercial Services. · We are looking for candidates who can drive transform ...

  • TalenCo Recruitment Specialists

    Data Scientist Durban

    6 days ago

    TalenCo Recruitment Specialists Durban

    Job Overview · The successful candidate will be responsible for analyzing and interpreting complex data to inform business decisions. This role requires a strong background in machine learning, AI, and data analysis. · Key Responsibilities · Data Analysis and Modeling · Develop a ...

  • SEIGO STAFFING

    Graduate Mechanical Engineer

    3 days ago

    SEIGO STAFFING Durban

    Qualifications: · Must have a Bachelor of Science in Engineering (Mechanical) or equivalent degree. Must be registered as a candidate engineer with the Engineering Council of South Africa. Minimum 0-3 years of experience in consulting engineering or contracting, specializing in b ...

  • Cletech Recruitment

    Mechanical Maintenance Engineer Durban

    6 days ago

    Cletech Recruitment Durban

    Job Title: Engineering Position · Company: Express Employment Professionals SA - PMB · Key Responsibilities · Collaborate with engineering managers to develop, design and test projects. · Analyze and research data for equipment maintenance planning. · Work with cross-functional t ...

  • Ramsay Engineering

    Business Analyst

    6 days ago

    Ramsay Engineering Durban

    Ramsay Engineering is seeking a dynamic Business Analyst to integrate operational information and enhance decision-making across the business. · Job Summary · This role focuses on aligning all departments using consistent naming conventions, maintaining data accuracy, and ensurin ...

  • Ziyana Group

    Data Centre Maintenance/Technical Manager

    1 week ago

    Ziyana Group Durban

    Data Centre Maintenance/Technical Manager · Our client in the International Service Trade and Distribution is seeking a Data Centre Maintenance/Technical Manager to join their performance-driven team. · Duties and Responsibilities: · Implement and oversee planned, preventative, c ...

Jobs
>
Durban
>
Senior data engineer