Search engine behavior towards automated querying

Abstract

Automated querying systems can sometimes form a large portion of the overall traffic of a search engine, thus degrading its performance. A serious challenge to search engine designers is to ensure that while third-party automated querying agents are blocked by their defense mechanisms, human users are not put to inconvenience. We propose to study how the defense mechanisms of commercial, large-scale search engines respond to automated querying, by using a proprietary automated querying agent. These responses are quantified, and presented in the form of state models. © 2014 IEEE.