Discussion on:

Use sitemap standards to help search engines

I've never heard of any standard requiring a sitemap to be an XML file.

What about CMS sitemaps, generated on the fly?

There are thousands, if not millions, of websites have flat HTML or generated site maps - does this mean that they're search engine unfriendly? No, because search engines crawl sites specifically to find what pages are really there and where they link to and from.

An XML file can easily get out of sync with sites that don't auto-generate such a file, so as a standard it's pretty weak. More to the point search engines do not look for these XML files by default, so it's not a standard at all - it might be a proposed standard but until it's widely adopted it's not a standard.

I for one won't be bothering with it for now.

It is true, the use of sitemaps helps search engines to correctly crawl and index your websites, and right now exists a sitemap standard that comes from the union of Google, Yahoo and MSN. You can visit this at http://www.sitemaps.org/

Best regards,

I loaded a sitemap for my site at www.TheScriptLibrary.com and my traffic went up by a factor of 5. However, the sitemap contained links to over 2000 pages, but doing a site:www.thescriptlibrary.com search at google only returns about 100 pages.
Any ideas?

They only index those deemed to be both relevant and providing content that does not duplicate that contained on other pages. You may have noticed that a great number of the pages actually indexed are in the Supplemental index; these are ones consider to be sufficiently similar to the primary results as to not be displayed unless specifically requested.

Bear in mind the the goal of a search engine is not to index every page of any site, but only those pages of that site whose content is both sufficiently unique and relevant to the corresponding search terms.

In short, the job of a good search engine is to point searchers to a relevant site, not necessarily to point them to each and every page within the site that mya be of interest to them.

I should have been more patient. There are now 938 pages listed. I think things improved a lot when I treated each page on its own in SEO terms and gave them all a unique title, description and keywords based on the content of each page.

can we use the site map for blog, such as if we use blogger.com??