Web Development

Ask something
Discussion Locked

Control how your site is indexed

0 helpful
1 reply

by maryweilage

Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

This week's Design and Usability Tactics e-newsletter explains how you can use a robots.txt file to control which portions of your site the search engine robots will index.

How do you control what search engine robots index? Does the method suggested in the column sound like something you'll try?

In the column, the author mentions that syntax errors aren?t always obvious to humans; however, you can use online validators to check your robots.txt file. Here are URLs to three online validators that you can use:

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

http://www.designerwiz.com/test/robots.txt_validator.htm

http://www.sxw.org.uk/computing/robots/check.html

* Please delete any extra spaces that may appear when you paste these URLs into your browser.

If you haven't subscribed to the Design and Usability Tactics TechMail, visit our subcenter to subscribe to this free e-newsletter today:
http://nl.com.com/acct_mgmt.jsp?brand=builder

1 reply

by dmerrill

I've read that proper use of robots.txt will prevent Google, Yahoo, Webcrawler, and other mainstream web spiders from indexing server areas that should be hidden from public view. FYI, I've seen firsthand evidence that Google in fact does NOT honor "Disallow" statements in robots.txt.

Because the robots.txt standard requires that you specify areas to be HIDDEN rather than areas to be indexed, it provides a roadmap for "script kiddies" who write robots and for phishers who want to scrape off a few email addresses.

I believe this standard needs to be rethought.

This discussion is closed to new replies.