ROBOTS.TXT (Spider) Cont

Continuando "ROBOTS.TXT (Spider) "

Otro link importante

List of User-Agents (Spiders, Robots, Crawler, Browser)
+ info