# See http://www.robotstxt.org/wc/norobots.html for documentation on how to usethe robots.txt file User-Agent: * Disallow: /users/*/diary/list/date Disallow: /users/*/diary/comments Disallow: /communities/*/entries/list/date Disallow: /communities/*/entries/comments Allow: / User-agent: Yeti Disallow: /files/* Disallow: /users/* Disallow: /communities/* User-agent: Baiduspider Disallow: /files/* Disallow: /users/* Disallow: /communities/*