Meanings of web crawler in English

Gray turned to Kat, their scientific web crawler.

In fact, he cites some of Mozilla's web crawler stats which seem to say just the opposite.

Both are designed to let webmasters communicate their sites' structure to the Yahoo and Bing Web crawlers.

Part of the auditor's curse, he supposed, forever filing away extraneous data, like a Web crawler that never slept.

To AMP-ify its pages, a publisher adds open source code to tell Google's web crawlers that an AMP version exists.

Search was made possible by web crawlers and a similar accessibility of data must be in place for the social graph.

Web crawlers are then unleashed to scour the Internet and find documents with the same fingerprint and a colour-coded report is generated.

The company's web crawlers and indexing servers can keep pretty busy, but Gmail servers slow down at night when people are sleeping.

Web crawlers, also known as spiders, move from website to website, following links embedded in each document, and can copy everything they encounter.

To avoid web crawlers looking for keywords, snake oil companies are implying they can help combat this virus without coming right out and saying so.

Garner contacted Cuill to get his site excluded from Twiceler, and was told that there has been a spate of web crawlers masquerading as Twiceler.