Shame to be Sham: Addressing content-based grey hat search engine optimization

We present an initial study identifying a form of content-based grey hat search engine optimization, in which a Web page contains both potentially relevant content and manipulated content: we call such pages sham documents, because they lie in the grey area between 'ham' (clearly normal) and 'spam' (clearly fake). Sham documents are often ranked ar-tifcially high in response to certain queries, but also may contain some useful information and cannot be considered as absolute spam. We report a novel annotation efort performed with the ClueWeb09 benchmark where pages were labeled as being spam, sham, or legitimate content. Sig-nifcant inter-annotator agreement rates support the claim that there are sham documents that are highly ranked by a very efective retrieval approach, yet are not spam. We also present an initial study of predictors that may indicate whether a query is the target of shamming.