IT Employment·20,084 discussions

Website Crawling Issue

Tags: Software

My Website is not crawling. What Can I do?

Topic

How To Solve Website Crawling Issues??

In reply to Website Crawling Issue

Hi,

to solve website crawling issues first you have to detect which type of issue facing your site, you have to go through below listed issues:
1) Check your http://www.abc.com and https;//abc.com domain included in GWT account.
2) Unblock your site by removing robot.txt from your site then your site will automatically reappear in the index.
3) Check that your site should have sitemap.xml which is simple list of directions that google should follow.
4) check if your website have crawl errors using below flow:
> Go to Google Webmaster tools
>Select your site
>Click on “Crawl”
> Click on “Crawl Errors ”
> You can see top 1000 pages having errors
5) Check if your website have lots of duplicate content. To solve this problem Pick the page you want and make rest 301.
6) Check your privacy settings should be off.
7) Make sure site should not in NOFOLLOW link in meta tag.
8) Make sure your site should have less Ajax and JavaScript usages. because these languages less indexable compared to HTML.
9) Make sure your site should have fast response time.
10) Check your Hosting down time.
11) Check you may be Deindexed it is really bad. If you got hit with a a manual penalty and from the index.

web crawling

In reply to Website Crawling Issue

If Google doesn’t index your website, then you’re pretty much invisible. You won’t show up for any search queries, and you won’t get any organic traffic whatsoever.
What is crawling and indexing?
Google discovers new web pages by crawling the web, and then they add those pages to their index. They do this using a web spider called Googlebot.
Confused? Let’s define a few key terms.
• Crawling: The process of following hyperlinks on the web to discover new content.
• Indexing: The process of storing every web page in a vast database.
• Web spider: A piece of software designed to carry out the crawling process at scale.
• Googlebot: Google’s web spider.

• When you Google something, you’re asking Google to return all relevant pages from their index. Because there are often millions of pages that fit the bill, Google’s ranking algorithm does its best to sort the pages so that you see the best and most relevant results first.
• The critical point I’m making here is that indexing and ranking are two different things.
• Indexing is showing up for the race; ranking is winning.
• You can’t win without showing up for the race in the first place.

How to get indexed by Google
Found that your website or web page isn’t indexed in Google? Try this:
1. Go to Google Search Console
2. Navigate to the URL inspection tool
3. Paste the URL you’d like Google to index into the search bar.
4. Wait for Google to check the URL
5. Click the “Request indexing” button
This process is good practice when you publish a new post or page. You’re effectively telling Google that you’ve added something new to your site and that they should take a look at it.
However, requesting indexing is unlikely to solve underlying problems preventing Google from indexing old pages. If that’s the case, follow the checklist below to diagnose and fix the problem.

[Author]

Replies