Parallel Entity Resolution with Apache Spark

Abstract

Entity resolution (ER) is a problem of matching records from different datasets that refer to the same entities. Accurate and fast entity resolution has broad applications in the field of data management. Although many ER approaches have been proposed, it is still a challenge to design a scalable algorithm to cope with massive datasets. In this paper, we propose a parallel entity resolution algorithm with Spark programming model, which can be executed on the computer cluster in parallel. In our approach, we adopt inverted indices and attribute-related TF-IDF weights of tokens to improve the precision and efficiency of the algorithm. Experimental results show that the proposed parallel ER algorithm achieves good accuracy and performance, and the algorithm is scalable for processing entity resolution of large datasets.