- spider × 22
- robots × 68 - A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)
- web-crawlers × 160 - (No tag wiki, but already has 5 synonyms: crawler, crawlers, web-crawler, crawling, and site-crawler)
Add a comment
|
1 Answer
All three of these tags are synonyms and should be merged. web-crawlers has the most usage and would be my preferred canonical tag name. The tag wiki from robots could be used for it.