Evaluating Opinion Summarization in Ranking

Abstract

We discuss the evaluation of rankings of documents that aim to summarize the overall opinion expressed in product reviews. Such a ranking can be used by e-commerce websites to represent a gist of the public opinion about a product. The inability of traditional IR metrics to reward such a representation is argued for. A matrix based labelling procedure serves as the framework for such evaluation. Three alternative metrics are adapted from previous similar works to evaluate opinion representativeness. Similarly, two metrics are adapted for exhaustiveness. Finally, we compare the robustness of these metrics. © 2017, Springer International Publishing AG.