The FreeBSD Diary

The FreeBSD Diary (TM)

Providing practical examples since 1998

If you buy from Amazon USA, please support us by using this link.
[ HOME | TOPICS | INDEX | WEB RESOURCES | BOOKS | CONTRIBUTE | SEARCH | FEEDBACK | FAQ | FORUMS ]
Scripts / handy tips
 New Topic  |  Go to Top  |  Go to Topic  |  Search  |  Log In   Newer Topic  |  Older Topic 
 robots.txt syntax checker
Author: Dan Langille 
Date:   06-02-02 03:06

Your website should contain a robots.txt file.

Here's a syntax checker: http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

Reply To This Message
 
 Re: robots.txt syntax checker
Author: new 
Date:   10-02-02 22:28

hi
ummmm
what's that file for????

Reply To This Message
 
 Re: robots.txt syntax checker
Author: Dan Langille 
Date:   11-02-02 04:17

It's what a robot should read before it reads other stuff from your website. See http://www.robotstxt.org/wc/exclusion-admin.html.

Reply To This Message
 Forum List  |  Threaded View   Newer Topic  |  Older Topic 


 Forum List  |  Need a Login? Register Here 
 User Login
 User Name:
 Password:
 Remember my login:
   
 Forgot Your Password?
Enter your email address or user name below and a new password will be sent to the email address associated with your profile.
How to get the most out of the forum

phorum.org