A Pyspark and Databricks Developer with a good understanding of the entire ETL/Azure lifecycle with a background of data projects.
Work you'll do
As a Databricks Developer Senior Analyst on the Solutions Delivery-Canada team, you will be responsible for:
- Develop ETL and ELT pipelines using Databricks and Apache Spark to support enterprise reporting, analytics, and migration initiatives.
- Integrate data from on-premises databases, cloud storage, APIs, and third-party sources into enterprise data platforms using Azure-native services and connectors.
- Transform, validate, and optimize large datasets using PySpark, SQL, and Databricks to support analytical and operational use cases.
- Support cloud migration activities, including SAP HANA modernization, Delta Lake implementation, and data model development for analytical workloads.
- Contribute to automation, DevOps, documentation, data governance, and Agile delivery practices across the data engineering lifecycle.
The team
Solutions Delivery-Canada is an integral part of the Information Technology Services group. The principle focus of this organization is the development and maintenance of technology solutions that e-enable the delivery of Function and Marketplace Services and Management Information Systems.
Solutions Delivery Canada develops and maintains solutions built on varied technologies like Siebel, PeopleSoft Microsoft technologies and Lotus Notes. Solutions Delivery Canada has various groups which provide the best of the breed solutions to the clients by following a streamlined system development methodology. Solutions Delivery Canada comprises of groups like Usability, Application Architecture, Development and Quality Assurance and Performance.
Location: Hyderabad
Qualifications
Required:
- B.Tech; or BCA + MCA; or BSc + MSc
- 4-5 years of experience in data engineering, ETL, or cloud data platform development
- Experience developing data pipelines using Databricks and PySpark
- Experience with Azure Data Factory and Azure DevOps
- Experience writing SQL for ETL processes, data transformation, and data modeling
- Experience with cloud-based ETL services and data integration solutions
- Experience implementing data quality, data governance, or data security controls
Preferred:
- Experience delivering end-to-end Databricks implementations
- Experience integrating or migrating data from SAP HANA to cloud platforms
- Experience with Delta Lake, Databricks SQL, or Spark SQL
- Experience with DevOps, CI/CD, or infrastructure-as-code practices
- Experience with DataStage
- Knowledge of Python programming