Data Engineer (Big Data)

As an experienced Data Engineer at ShyftLabs, you will be part of a dynamic team focused on building scalable software solutions for Fortune 500 companies. Your role will involve collaborating with cross-functional teams to define requirements and deliver innovative solutions. **Key Responsibilities:** - Develop and maintain data pipelines and ETL/ELT processes using Python - Design and implement scalable, high-performance applications - Develop and manage near real-time data streaming solutions using Pub, Sub, or Beam - Monitor and troubleshoot production systems to ensure reliability and performance **Qualifications Required:** - 5+ years of professional software development experience with Python - Strong understanding of software engineering best practices (testing, version control, CI/CD) - Experience building and optimizing ETL/ELT processes and data pipelines - Proficiency with SQL and database concepts - Experience with data processing frameworks - Understanding of software design patterns and architectural principles - Experience with unit testing and test automation - Experience working with any cloud provider (GCP is preferred) - Experience with CI/CD pipelines and Infrastructure as code - Experience with Containerization technologies like Docker or Kubernetes - Bachelor's degree in Computer Science, Engineering, or related field - Proven track record of delivering complex software projects - Experience with GCP services, particularly Cloud Run and Dataflow - Experience with stream processing technologies (Pub/Sub) - Familiarity with big data technologies (Airflow) - Experience with data visualization tools and libraries - Familiarity with platforms like Snowflake, Bigquery, or Databricks - GCP Data engineer certification As part of ShyftLabs, you will enjoy a competitive salary, a strong insurance package, and extensive learning and development resources to support your growth and professional development. As an experienced Data Engineer at ShyftLabs, you will be part of a dynamic team focused on building scalable software solutions for Fortune 500 companies. Your role will involve collaborating with cross-functional teams to define requirements and deliver innovative solutions. **Key Responsibilities:** - Develop and maintain data pipelines and ETL/ELT processes using Python - Design and implement scalable, high-performance applications - Develop and manage near real-time data streaming solutions using Pub, Sub, or Beam - Monitor and troubleshoot production systems to ensure reliability and performance **Qualifications Required:** - 5+ years of professional software development experience with Python - Strong understanding of software engineering best practices (testing, version control, CI/CD) - Experience building and optimizing ETL/ELT processes and data pipelines - Proficiency with SQL and database concepts - Experience with data processing frameworks - Understanding of software design patterns and architectural principles - Experience with unit testing and test automation - Experience working with any cloud provider (GCP is preferred) - Experience with CI/CD pipelines and Infrastructure as code - Experience with Containerization technologies like Docker or Kubernetes - Bachelor's degree in Computer Science, Engineering, or related field - Proven track record of delivering complex software projects - Experience with GCP services, particularly Cloud Run and Dataflow - Experience with stream processing technologies (Pub/Sub) - Familiarity with big data technologies (Airflow) - Experience with data visualization tools and libraries - Familiarity with platforms like Snowflake, Bigquery, or Databricks - GCP Data engineer certification As part of ShyftLabs, you will enjoy a competitive salary, a strong insurance package, and extensive learning and development resources to support your growth and professional development.