ETL Data Engineer

Job Description: As an experienced ETL Developer and Data Engineer, your role involves ingesting and analyzing data from multiple enterprise sources into Adobe Experience Platform. You will be responsible for the following key responsibilities: - Develop data ingestion pipelines using PySpark for both batch and streaming data processing. - Utilize multiple Data engineering services on AWS such as Glue, Athena, DynamoDb, Kinesis, Kafka, Lambda, and Redshift. - Load data from various sources like S3 bucket and on-prem data sources into Redshift. - Optimize data ingestion into Redshift and design tables in Redshift considering distribution key, compression, vacuuming, etc. - Design, develop, and optimize queries on Redshift using SQL or PySparkSQL. - Develop applications consuming services exposed as ReST APIs and write complex and performant SQL queries. Qualifications Required: - 4-6 years of professional technology experience with a focus on data engineering. - Strong experience in developing data ingestion pipelines using PySpark. - Proficiency in AWS services like Glue, Athena, DynamoDb, Kinesis, Kafka, Lambda, and Redshift. - Experience working with Redshift, including loading data, optimizing ingestion, and designing tables. - Ability to write and analyze complex SQL queries. - Experience with enterprise-grade ETL tools like Pentaho, Informatica, or Talend is a plus. - Good knowledge of Data Modeling design patterns and best practices. - Familiarity with Reporting Technologies such as Tableau or PowerBI. In this role, you will be analyzing customers' use cases and data sources, extracting, transforming, and loading data into Adobe Experience Platform. You will design and build data ingestion pipelines using PySpark, ensure performance efficiency, develop and test complex SQL queries, and support Data Architects in implementing data models. Additionally, you will contribute to the organization's innovation charter, present on advanced features, and work independently with minimum supervision. Please rate the candidate on the following areas: - Big Data - PySpark - AWS - Redshift Job Description: As an experienced ETL Developer and Data Engineer, your role involves ingesting and analyzing data from multiple enterprise sources into Adobe Experience Platform. You will be responsible for the following key responsibilities: - Develop data ingestion pipelines using PySpark for both batch and streaming data processing. - Utilize multiple Data engineering services on AWS such as Glue, Athena, DynamoDb, Kinesis, Kafka, Lambda, and Redshift. - Load data from various sources like S3 bucket and on-prem data sources into Redshift. - Optimize data ingestion into Redshift and design tables in Redshift considering distribution key, compression, vacuuming, etc. - Design, develop, and optimize queries on Redshift using SQL or PySparkSQL. - Develop applications consuming services exposed as ReST APIs and write complex and performant SQL queries. Qualifications Required: - 4-6 years of professional technology experience with a focus on data engineering. - Strong experience in developing data ingestion pipelines using PySpark. - Proficiency in AWS services like Glue, Athena, DynamoDb, Kinesis, Kafka, Lambda, and Redshift. - Experience working with Redshift, including loading data, optimizing ingestion, and designing tables. - Ability to write and analyze complex SQL queries. - Experience with enterprise-grade ETL tools like Pentaho, Informatica, or Talend is a plus. - Good knowledge of Data Modeling design patterns and best practices. - Familiarity with Reporting Technologies such as Tableau or PowerBI. In this role, you will be analyzing customers' use cases and data sources, extracting, transforming, and loading data into Adobe Experience Platform. You will design and build data ingestion pipelines using PySpark, ensure performance efficiency, develop and test complex SQL queries, and support Data Architects in implementing data models. Additionally, you will contribute to the organization's innovation charter, present on advanced features, and work independently with minimum supervision. Please rate the candidate on the following areas: - Big Data - PySpark - AWS - Redshift