Data Engineer (Remote)
Role: Consultant
Client: United Nations Organization
Contract type: Freelancer / Self-Employed / B2B only
Remote: Remote or on-site from Rome
Location: Rome, Italy
Duration: 3 months renewable
Visa sponsorship: Not available
To apply for this position, you can send your resume at office@idealstaffs.com referencing the job title.
Â
Technical context:
Our client has developed a data lake hosted in Microsoft Azure. This data lake is populated with data from our client's corporate systems and external sources (such as other UN agencies, IFIs, universities). The data from corporate systems is either integrated from an Oracle data warehouse or from a PostgreSQL database. Corporate data is refreshed daily. The data pipeline is optimized for speed and cost. Corporate data is stored in SQL database in Azure. Microsoft Data Factory is used for the ingestion of external data sources.
Microsoft Power BI is used for self service analytics. The data from the SQL database in the data lake is structured in tables available within PBI. Unstructured data is also stored in the data lake.
Azure Machine Learning is used for advanced analytics use cases, with some data pipelines required to supply data from feeder systems, including photos from a photo library.
Microsoft Purview is used for its data catalogue functionality.
Duties and responsibilities:
The Data Engineering Specialist provides the required expertise and will be expected to:
Deliver data engineering solutions supporting online platform analytics and data science projects.
Incorporate new data sets in the data lake using data integration technologies.
Enable the integration of external data sets based on available formats providing prompt turnaround of user requests including clear guidance on the automation of refreshes or, when not possible, details of the steps required by the user to trigger refreshes
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Azure technologies.
Model the data to produce comprehensive data models for use by non-IT data analysts
Build analytics tools that utilize the data pipeline to provide actionable insights into key business performance metrics.
Work with stakeholders including ICT to assist with data-related technical issues and support their data infrastructure needs.
Keep the data separated and secure in the data lake based on confidentiality
Provide the necessary data for monitoring of the availability of data, successfully daily operations and cost management
Document technical solutions and design
Work closely with the ICT team for all matters relating to data
Professional requirements:
Advanced university degree (Master’s degree or equivalent) in computer science, data science, analytics, statistics, information management, public administration, management or a related field. A first-level university degree in combination with qualifying experience may be accepted in lieu of the advanced university degree.
A minimum of 5 years of progressively responsible experience in applied analytics, business intelligence, information management or statistics.
Experience in mid to expert level programming capabilities in Spark, Python or Scala is required.
Experience on Azure cloud including Data Factory, Databricks, Data lake Storage is highly required.
Demonstrated experience in modern API platform design is required.
Experience in tools managing version control such as Git is highly required.
Experience with data warehousing, database programing languages like SQL, PL/SQL and noSQL databases like Cassandra or MongoDB is required.
Demonstrable knowledge of data analysis life cycle from ingest and wrangling to analysis and visualization to present findings.
Experience with development tools like IntelliJ/Eclipse/VSCode IDE, Build Tool Maven is required.
Experience working with analytical tools like Tableau, Qlik, PowerBI for semantic-layer-based data discovery is desirable.
Strong analytic skills related to working with unstructured datasets.
Excellent problem solving and troubleshooting skills.
Ability to convey complex / difficult data science topics to clients in a relatable manner.
Specific requirements:
Azure Data Functions:
Implementing data transformation logic using Azure Functions or custom code within ADF pipelines
Configuring & managing API linked services
Conducting performance tuning and optimization of ADF pipelines for improved data processing efficiency
Troubleshooting and resolving data integration issues, including data format inconsistencies, schema mismatches, and data validation errors.
To be able to develop and enhance new/existing plsql procedures
Understanding and creating the entity relationships across the data model
Developing and maintaining data pipelines for data ingestion from external sources into Azure Data Lake Storage
Technical expertise in working with REST APIs that Microsoft offers for Azure stack and to be able to use the APIs to create pipelines
Building pipelines to analyse usage statistics for Azure and Power BI and others if needed.
Providing technical guidance and support to the ML/AI/Power BI teams for data preparation and feature engineering tasks
Logic Apps:
Understanding how to set up actions, triggers, and parameters within Logic Apps
Ability to design and build workflows using the Logic Apps designer interface
Understanding how to configure triggers to initiate the Logic App workflow based on specific events, such as receiving an HTTP request, a new message in a queue, a file being added to a storage account, or a scheduled time
Azure Key Vault:
Knowledge of working with cryptographic keys in Azure Key Vault
Understanding the access control mechanisms provided by Azure Key Vault
Experience in integrating Azure Key Vault with applications and services. This includes understanding the various methods of accessing secrets, keys, and certificates from Azure Key Vault, such as using Azure SDKs, REST APIs, PowerShell, or Azure CLI
Power Apps:
Knowledge of integrating Power Apps with various data sources such as SharePoint, Microsoft Excel, SQL Server, and other cloud-based services
Power BI:
Basic knowledge on Power BI reporting and understand how data flows and datasets work with in power bi.
Azure Data bricks:
Experience in integrating Azure Databricks with various data sources and platforms. This involves reading and writing data from and to sources like Azure Storage, Azure Data Lake Storage, Azure SQL Database, Azure Synapse Analytics, or other cloud-based services.
Understanding Apache Spark and its distributed computing capabilities within Azure Databricks.
Required competencies:
Problem solver – you can propose solutions in a clear and understandable manner to non-technical users. Ability to work with other to brain storm and develop practical and innovative solution
Team Worker - you develop and maintain a positive relationship with end users; able to consult and deal effectively with all levels of the organization and establish working relations of openness and trust in a multicultural setting. Work closely with the ICT technical team in all matters related to data.
Planner and Organizer – you prioritize and organize the work for effective resolutions of functional and technical problems
Performer – you demonstrate success in the implementation and quality control of corporate projects
Accountability: Takes ownership of all responsibilities and honour commitments; delivers outputs within prescribed time, cost and quality standards; operates in compliance with organizational regulations and rules
Education:
Advanced university degree (Master’s degree or equivalent) in computer science, data science, analytics, statistics, information management, public administration, management or a related field.
A first-level university degree in combination with qualifying experience may be accepted in lieu of the advanced university degree.
Languages
Excellent written and verbal communication skills in English essential.
To apply for this position, you can send your resume at office@idealstaffs.com referencing the job title.