Datu integrācijas problēmu risinājumi datu noliktavas projektā

Abstract

Vienā no reāliem datu noliktavas projektiem bakalaura darba autoram nācās saskarties ar neunikālo personu datu integrāciju no vairākiem datu avotiem ar mērķi izveidot unikālo personu dimensiju. Darbā ir autora piedāvātais algoritms unikālo personu identificēšanai un neunikālo personu atpazīšanai un piesaistīšanai pie identificētām unikālām personām. Darbā aprakstītā metode personas datu integrācijai tika pielietota reālajā projektā un sniedz rezultātus, kuri pilnībā apmierina biznesa puses pārstāvjus. Metode ir balstīta uz bakalaura darbā teorijas daļā aprakstītām tehnikām, kuras izmanto datu integrēšanā. Īpaša uzmanība tika pievērsta tām datu integrācijas problēmām, kuras ir saistītas ar datu kvalitāti. Vairāku avotu datu apvienošana un personu atpazīšana tika realizēta, kā ETL process, kas ielāde datus datu noliktavā.

Working on one real-world data warehouse project, the author of this bachelor paper, took part in non-unique person data integration from multiple data sources with the goal of developing unique person dimension. In this paper the author proposed an algorithm for unique person identification, non-unique person recognition and a way to assign them to identified unique persons. The method for person data integration, which is described in the present paper, was used in the real project and delivered results that fully satisfied business demands. The method is based on techniques, which were used in data integration and were described in the theoretical part of this the bachelor paper. Particular attention was paid to those integration problems, which were related to data quality. Multi source data integration and person recognition were implemented as ETL process, which uploads data to data warehouse.