Big Data Engineer

As a Big Data Engineer, your role involves designing and implementing scalable data processing pipelines using Hadoop and Spark. You will be responsible for developing and maintaining high-performance applications in Scala to process and analyze large datasets. Your key responsibilities include: - Building batch and real-time data processing solutions - Collaborating with data scientists, analysts, and business stakeholders to gather and translate data requirements - Optimizing Spark jobs and Hadoop workflows for performance, scalability, and efficiency - Troubleshooting and resolving issues related to distributed data processing systems - Ensuring data quality, validation, and integrity throughout the data lifecycle - Participating in code reviews and implementing best practices in big data development - Supporting production deployments and providing ongoing maintenance and enhancements To excel in this role, you should possess the following qualifications: - Strong understanding of distributed computing concepts - Hands-on experience in Spark (RDD, DataFrames, Datasets, Spark SQL) - Experience in writing complex transformations and performance tuning in Scala - Knowledge of cluster management and resource optimization - Familiarity with data warehousing and ETL concepts - Experience with version control systems (Git) - Understanding of CI/CD processes for data engineering pipelines (preferred) - Strong analytical and problem-solving skills - Ability to work in Agile environments and collaborate with cross-functional teams If you are passionate about working with cutting-edge technologies like SCALA, Hadoop, Spark, Pyspark, and Python, and have a Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field, this position offers a great opportunity to grow and contribute to impactful projects. As a Big Data Engineer, your role involves designing and implementing scalable data processing pipelines using Hadoop and Spark. You will be responsible for developing and maintaining high-performance applications in Scala to process and analyze large datasets. Your key responsibilities include: - Building batch and real-time data processing solutions - Collaborating with data scientists, analysts, and business stakeholders to gather and translate data requirements - Optimizing Spark jobs and Hadoop workflows for performance, scalability, and efficiency - Troubleshooting and resolving issues related to distributed data processing systems - Ensuring data quality, validation, and integrity throughout the data lifecycle - Participating in code reviews and implementing best practices in big data development - Supporting production deployments and providing ongoing maintenance and enhancements To excel in this role, you should possess the following qualifications: - Strong understanding of distributed computing concepts - Hands-on experience in Spark (RDD, DataFrames, Datasets, Spark SQL) - Experience in writing complex transformations and performance tuning in Scala - Knowledge of cluster management and resource optimization - Familiarity with data warehousing and ETL concepts - Experience with version control systems (Git) - Understanding of CI/CD processes for data engineering pipelines (preferred) - Strong analytical and problem-solving skills - Ability to work in Agile environments and collaborate with cross-functional teams If you are passionate about working with cutting-edge technologies like SCALA, Hadoop, Spark, Pyspark, and Python, and have a Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field, this position offers a great opportunity to grow and contribute to impactful projects.