(EI收录)Incremental Rapidly Grouping Aggregation Method for Simila

inconsistent problem, it is a common problem in multiple sources heterogeneous short text data sets; (b)

Comparison with the long text, the value of the web news headlines often contains colloquial words, it

causes the short text aggregation more difficult than the long text; (c) news headlines belong to short

usually the number of web news headlines under one topic may reach million-level. If we directly to

solve.

study. That is the object name may differ. It should be aggregated, but it not be aggregated correctly, or

different objects with a very similar name may be incorrectly aggregated together. However, in recent

method, it can rapidly aggregate similar web news headlines. The method contains three core steps: At

first, based on the BM25 algorithm to get all the most relevant news headlines. Second, using a

character-based IJW similarity algorithm and specific normalization strategy, it can rapidly obtain the

most similar news headline and efficient to finish the construction of the index. Finally, the aggregation

results will be obtained by the Group By aggregation function of the MongoDB database. The flow chart

of the RGA method is indicated in Figure 1.

our system, the similarity threshold K was set to 0.75, and it can get the best performance. Through the

update method to get the IJW algorithm. It can enhance the similarity calculation accuracy for two

strings.

algorithm to map all similar news headlines into the same MD5 value.

For the short text aggregation model, it is usually necessary to use a text similarity algorithm. The

common string-based similarity algorithms generally divide into two categories [4]: Character-based