# Tells Scanning Robots Where They Are And Are Not Welcome # User-agent: can also specify by name; "*" is for everyone # Disallow: if this matches first part of requested path, # forget it User-agent: * Disallow: /bak/ Disallow: /bin/ Disallow: /cgi-bin/ Disallow: /gifs/ Disallow: /jpgs/ Disallow: /netscape/ Disallow: /private/ Disallow: /prj/ Disallow: /zips/