Ameliorating the Herding Effect Driven by Search Engines using Diversity-Based Ranking

In competitive search settings, document publishers (authors) respond to rankings induced for queries of interest: they modify the documents to improve their future ranking. It was shown theoretically and empirically that a prevalent modification strategy of publishers is mimicking content in the documents most highly ranked in the past for the query at hand. Accordingly, publisher herding with unwarranted corpus effects (e.g., reduced topical diversity) was observed. We present the first theoretical and empirical study of competitive search settings where ranking is based not only on relevance estimation as was the case in past work, but also on search results diversification. We theoretically show that diversity-based ranking results in a min-max regret equilibrium where content mimicking, and as a result herding, are ameliorated. Analysis of ranking competitions we organized provides empirical support to our theoretical findings.