Job Description:
We are seeking a talented Data Engineer to join our team at A2Cloud Solutions. As a Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines and infrastructure to support our data-driven initiatives. Your expertise in SQL, Python, PySpark, Databricks, and Azure Data Factory will be instrumental in converting raw data into high-quality datasets ready for our data scientists to use.
Responsibilities:
- Develop and maintain data pipelines using SQL, Python, PySpark, Databricks, and Azure Data Factory
- Use delta tables in Databricks to efficiently process and store data
- Collaborate with cross-functional teams to understand data requirements and implement data integration and transformation processes
- Ensure data quality and integrity by writing unit tests and performing data validation
- Promote code from development to production environments following best practices
- Document code, processes, and workflows to ensure knowledge sharing and maintainability
- Manipulate and join data from various sources, primarily SQL servers, to create comprehensive datasets
- Optimize data processing performance and scalability
Requirements:
- Strong proficiency in SQL and experience with SQL servers
- Solid programming skills in Python
- Practical experience with PySpark and Databricks
- Familiarity with Azure Data Factory or similar data integration tools
- Knowledge of delta tables in Databricks is a plus (prior experience not necessary)
- Understanding of documentation, coding standards, and software engineering best practices
- Practical experience with version control principles, preferably Git
- Ability to write unit tests and perform data validation to ensure data quality
- Excellent problem-solving and communication skills
Preferred Qualifications:
- Experience in promoting code from development to production environments
- Familiarity with converting raw data into datasets for data scientists
- Strong data manipulation and data joining skills
- Previous exposure to a variety of data sources
Benefits:
- Competitive salary
- Retirement plans
- Opportunities for professional development
- Dynamic and collaborative work environment
To apply for this position, please send your resume, cover letter, and any relevant project samples or GitHub repositories to hr@a2cloud.co.uk. Please include “Data Engineer Application” in the subject line.