Multi-match Segments Similarity Join Algorithm Based on MapReduce

Similarity joint is applied in many fields. While the similarity joint algorithm is used to clean the massive data set, MapReduce can provide an effective framework of distributed computers. This paper mainly analyzes the similarity joint algorithm which is based on MapReduce, and proposes an improved strategy that introduces the implementation process base on the framework of MapReduce in detail after the improvement of PassJoin algorithm. The improved algorithm increases the filtering conditions to eliminate its own redundancy in the filtering phase and reduces the times of reading the original string. The time consumption is reduced considerably by experiments, which shows the effectiveness of the improved strategy.