Falcon Consulting has been asked by Microsoft for its expertise in .NET and reporting to implement product analysis tools for the media catalogs (music, video and ebooks) on the Windows Store using the latest Azure technologies.
- Implementation of project developments and production quality control
- Development in U-SQL under Azure Data Lake Analytics of scripts for quality control of data from the ebook catalog with a library of classes containing the code behind the scripts in C# (generation of extracts, counters and data consistency checks, creation of stored procedures, archiving of data).
- Migration of projects developed in Scope and SQL Server to Azure Data Lake Analytics and development in U-SQL under Azure of data quality control scripts from the video catalog with a library of classes containing the code behind the scripts in C# (generation of extracts, counters and data consistency checks, creation of stored procedures, archiving of data).
- Administration of team subscriptions on the Azure portal, creation of application accounts, rights management, adding and configuring Azure components (Azure Data Lake Analytics, Azure Data Lake Store, Azure Blob Storage, Azure Batch, Azure SQL, Azure Machine Learning, Azure Data Factory) with PowerShell commands and scripting
- Implementation of Data Factory in Azure for the automation of the data quality control process for ebook and video catalogues.
- Development in C# 4.6 of a Custom Activity to automate the cleaning of Azure Data Lake Blinds
- Development in C# 4.6 of Custom Activities to automate the import of data from OPF files and PDF import logs.
- Creation and implementation of reports in PowerBI (visualization of extracts, formatting of counters and graphs for data consistency checking)
- Administration of the PowerBI portal, rights management, schedule management
- Participation in the training of the team in the development of U-SQL
- Use of new technologies in Azure Machine Learning and Azure Cosmos DB to adapt to different projects.
- Migration testing of projects from Azure Data Factory to Azure Data Factory v2
C# 4.6.2, U-SQL, T-SQL, Azure (Azure Data Lake Analytics, Azure Data Lake Storage, Azure Data Factory, Azure Blob Storage, Azure SQL, Azure APIs, Azure Machine Learning, Azure Cosmos DB), PowerShell, notions of Scope, notions of R
Tools used: SQL Server 2017, Visual Studio 2017, Portail Azure, PowerBI, Cosmos Scope Studio Work under: Windows 10, Azure
CHALLENGES AND SOLUTIONS
The entire project from scratch, including the implementation of the architecture, the development, the setting up of an R&D process, the writing of the functional and technical specifications, the training of Microsoft employees and the transfer of skills, was carried out by Falcon Consulting. Falcon Consulting was able to very quickly take in hand an unknown environment: Azure (Azure Data Lake Analytics, Azure Data Lake Storage, Azure Data Factory, Azure Blob Storage, Azure SQL, Azure APIs, Azure Machine Learning, Azure Cosmos DB) and expand its scope of expertise by including the Big Data part.