Community archive

Ask something
Discussion Locked

How do search engines work?

0 helpful
31 replies

by jardinier

Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

There are a number of things which my little brain can't compute.

Like: "When did time begin?" and "What was there before the universe came into being."

Nor can my mind compute the common-or-garden internal combustion engine. At 6,000 RPM those pistons are going up and down 100 times per second and I simply cannot get my mind to visualise this.

And so to internet search engines -- again my mind can't comprehend how, in a few seconds or less, the search engine can travel all around the world finding key words in sometimes more than one million websites/pages.

Can anyone help me to graduate from IT kindergarten to primary school?

31 replies

by deadly ernest

But my understanding is that they have programs running that scan web sites and web pages for relevant information, based on an algorhythm of some sort, and then load all this into a huge database with and even bigger index. When you run a search it checks the index and databse and gives up the info it has. This database is constantly changing as the system detects changes and adds them.

A few years back I created web sites and web pages for three businesses. When they were done I put them up on the clients' web hosting servers - for a week they sat there until the clients got back and said they were happy with them. During that time I ran a number of searches with Google, AltaVista and Yahoo and they were not found. The day I got the go ahead from them I registered all three sites with four free web search engine organisations; about 4 hours later they were findable via a search on Google, AltaVista, Yahoo and MSN.

It is because they are checking their own databases that some sites do not show on searches as the sites are not in their database, despite being actual sites in existence.

Oh all most forgot the first information that the search engines list si the info entered in each page's meta headers. That is why you can get a hit on a page and then NOT find any relevant content, the link is with something in the meta header tags which is NOT in the page itself.

I know one work colleague, from a few years back, got pissed off when she did a search for a rather specific, and unusual, sexual fetish - he got four hits back to the same web site. However, the site was nothing to do with sex or what she was after. But it was loaded with all sorts of geek useful shareware and freeware tools for hacking and security testing. When I examined the source code for her I found the meta header tags took up more space than the home page - the designer had included just about every possible variant of any sexual reference you could have in English, French and German - it was a bit of an education for me finding out what some of those fetishes were about. Boy that person wanted to make sure a lot of people linked to the site. And very little actual dealt with what was on the site.

by lukcad

it is nice question. I think that you have a little time to read that article from world community of wikipedia: http://en.wikipedia.org/wiki/Search_engine
I suppose it is enougth understable. In the middle of article you will find the answer that will enougth for your smart pupils, like this: A search engine operates, in the following order

Web crawling
Indexing
Searching
Web search engines work by storing information about a large number of web pages, which they retrieve from the WWW itself. These pages are retrieved by a web crawler (sometimes also known as a spider) ? an automated web browser which follows every link it sees, exclusions can be made by the use of robots.txt. The contents of each page are then analyzed to determine how it should be indexed (for example, words are extracted from the titles, headings, or special fields called meta tags). Data about web pages is stored in an index database for use in later queries. Some search engines, such as Google, store all or part of the source page (referred to as a cache) as well as information about the web pages, whereas some store every word of every page it finds, such as AltaVista. This cached page always holds the actual search text since it is the one that was actually indexed, so it can be very useful when the content of the current page has been updated and the search terms are no longer in it. This problem might be considered to be a mild form of linkrot, and Google's handling of it increases usability by satisfying user expectations that the search terms will be on the returned web page. This satisfies the principle of least astonishment since the user normally expects the search terms to be on the returned pages. Increased search relevance makes these cached pages very useful, even beyond the fact that they may contain data that may no longer be available elsewhere.

When a user comes to the search engine and makes a query, typically by giving key words, the engine looks up the index and provides a listing of best-matching web pages according to its criteria, usually with a short summary containing the document's title and sometimes parts of the text. Most search engines support the use of the boolean terms AND, OR and NOT to further specify the search query. An advanced feature is proximity search, which allows you to define the distance between keywords.

The usefulness of a search engine depends on the relevance of the result set it gives back. While there may be millions of Web pages that include a particular word or phrase, some pages may be more relevant, popular, or authoritative than others. Most search engines employ methods to rank the results to provide the "best" results first. How a search engine decides which pages are the best matches, and what order the results should be shown in, varies widely from one engine to another. The methods also change over time as Internet usage changes and new techniques evolve.

Most web search engines are commercial ventures supported by advertising revenue and, as a result, some employ the controversial practice of allowing advertisers to pay money to have their listings ranked higher in search results.

The vast majority of search engines are run by private companies using proprietary algorithms and closed databases, the most popular currently being Google, MSN Search, and Yahoo! Search. However, Open source search engine technology does exist, such as ht://Dig, Nutch, Senas, Egothor, OpenFTS, DataparkSearch and many others

by rob mekel

1st When did time begin: when the clock started ticking.

2nd What was there before the universe came into being: the diverse.

3rd think about the frequentie of light or a hummingbird with a wing-beat of 50 (average)beats a second. The rotation of the plate is only the issue if the frequentie of the reader isn't fast enough. The movent to center or outbound of the plate is relatively slow (+/- 20 m/s) to the rotation of the plate (+/- 97 m/s).

4th Ask yourself the question why do we place tags on threads. Why do we cache info. Why do providers update their cached info on the search engines?

Do some of these answers (Q&A??) enlighten you?

Rob

by oneamazingwriter

You didn't specify what clock ticked, and for a moment I paniced thinking if mine stopped everything is over, but then I reversed that thought and considered if I get a supply of batteries and make certain it's well maintained, time will go on. :D

(There's no need to thank me. My motives are not totally altruistic.) :)

by rob mekel

What about switching the old for the new batteries :0 ... does your clock stop ticking for just a tiny little part of a second ?:| ;) :D

[i]Pweeew, I'm glad it ain't my clock.[/i]

Rob

by oneamazingwriter

I'll see what I can do to get it switched to AC. A guy tokd me once that he's like to "fix my clock". Maybe he can do it for me! He seemed grouchy, but perhaps when I explain that this is a mission to save the world, he will cheer up.

by rob mekel

"Saving the world" that should bring him around. Otherwise put in your womanly charms. That for sure would do the trick. :0 :D :)

I'm glad to be a man. As my clock started ticking it doesn't stop untill the last heartbeat. :D ;)
[i]that is I hope it doesn't, as it gives a lot of joy [/i]

Rob

by dr dij

'the last question' by arthur c clarke
answers how universe began, and what was there before it.

by charliespencer

That's an Isaac Asimov short. Arthur gave us the other end of the equation in "The Nine Billion Names of God".

by sleepin'dawg

I've been a Hawking afficianado for quite some time but for some reason had never thought of accessing information about him and his work over the net. Thanks for the short cut.

[b]Dawg[/b] ]:)

by jardinier

Thanks for the link. Actually I have owned a copy of Stephen Hawking's "A Brief History Of Time" for some years, but have never put aside the "time" to read it. :)

by jardinier

I use [b]answers.com[/b] when searching for material for my various websites, because it usually brings up a wikipedia entry which includes photographs, and if there is no wikipedia entry, it switches to google.

I switched from Google to [b]ask.com[/b]when I was researching for an article on accessing child pornography on the net (pursuant to a discussion I had started at TR).

Google had censored out all the websites relating to the subject, other than a few which spoke out against child pornography.

[b]ask.com[/b] gave a warning that I would be accessing adult material, giving me the option to proceed if I wished.

by jmgarvin

1) The search engine sends out things called crawlers. They look for meta tags, content, and key context on a page. This will help to classify the page and help in the search terms for it.

2) The engine then stores the crawler information so that is can deal with it...

3) The information is parsed, classified, and stored in the proper "bucket." In this step an intellegent piece of software is run. It is able to decide on its own what is what. It depends on the search engine, but typically we are talking about some kind of neural network or SVM. This gets pretty technical, so you might want to read on the term: Data Mining

4) The information can now be searched with the proper search terms

Does that help?

by sleepin'dawg

I would tell you how to except it is sort of like giving driving directions to someone. You may pass a location every day but it has become so routine that you've forgotten all the details of how you get there.

I would suggest checking Wikipedia and/or Google and Yahoo for more detailed information. Now there's an irony for you searching for instuctions on how to search for instructions on how to search.........oh, I'm developing a brain cramp. :^0

[b]Dawg[/b] ]:)

by oz_media

Well there are groups of engines, that differ a great deal from each other.

I have been offering Search Engine Optimization for quite a fews years now. It's AMAZING how many well designed pages simply are not optimized for engines.

Yes, all engines work with an indexing system. SOME use a spider that crawls your page, they seek a balance between keywords, meta tags, ALT tags and body copy.

They all ruyn with different ranking algorhythms, some don't like mroe than three 'similar' keywords in the meta tags, some require a minimum number of maximum number of META TAG keywords before they disqualify your page entirely. This puts tag spammers on the bottom of the pile.

Most offer positioning (indexing) based on a balance between keywords, meta tags, ALT tags and body copy. It is a fine balance that changes with each engines algorhythms. This way a site with 'SEX SEX SEX SEX' etc. in the tags will actually be disqualified from ranking well. In he days of yore, such sites would automatically be th enumber one returned hit, but the engines got smart.

SEO, is so funny sometimes, I haveworked on sites that cost a company 10's of thousands to have designed, then it is completely buried in search engines, rendering it useless. With proper optimization, it is easy to get the site to rank in the top 10 returned hits for most keywords and search engines.

I cant believe people pay for Google placement, it is a piece of cake to get in the top 10, unpaid hits on Google. Just time and patience.

ANYWAY, how does this answer your question?
This is all part of the indexing system the engines use to recall pages when keywords are searched. How they retrieve pages is pretty irrelevant, as long as you have a well indexed page, you will have success.

So far I have worked on (provided SEO for) about 23 sites, EVERY SINGLE ONE OF THEM, has pages that rank in the top 10 for major engines under the owners chosen keyords.
I get a flat fee for the optimization and several thousand fo reach page that makes top 10, more for hitting number 1.

These pages automatically drop in rank over time, therefore an ongoing submission and tweaking service provides some really nice dosh on a monthly basis. get 5 or 7 sites going and you're good for at least 5K month, cash in hand.

by jardinier

For 12 months I used various of the website checking services offered by SEVENtwentyfour.com, because they offered me a big discount. However the fee for optimization was way too high for my purposes -- non-profit websites.

More recently I downloaded some free software from a site whose name I forget, but which included instructions for optimization. I never found the time to study it.

Well I am happy enough to know that my various websites are frequently accessed by many people, some of whom send an appreciative email, and others ask dumb questions, like the lass (I have referred to this in another discussion) who found the page "Magic" on one of my websites and asked if I could teach her black magic.

Well I warned her about going down that road, but she is still emailing me asking for the best place to find solutions to her personal problems.

I have had some remarkable experiences with people finding my websites. On one occasion I received a phone call from a Sydney lass asking if I sold laptop computers. She had used the key words: [b]macintosh - laptop - for - sale[/b] and my website that deals with IT came up number one on the search.

On another occasion I was checking the spelling of a word from Hindu philosophy, and the second site listed by the search engine was my own website, and the particular word was in the BODY of an article.

by oz_media

Body copy will generally rank much higher than other tags, however if competing for a top position against another similar site, it takes a finely tuned balance between body, ALT (image tags), meta's etc. Each search engine ranks differently and the algorhythms change constantly. It's a full time job sometimes just keeping up to date with the engines changing all the time.

by jardinier

The key element I was missing of course is that the search engines already have an indexed database to access.

by deepsand

Meta-search engines send simultaneous inquiries re. your search to multiple search engines, compile the results, and produce a list of lists of all the results.

@ other commonly used meta-search engines are MetaCrawler ( one of the 1st, if not the 1st, meta-search engine) and DogPile.

by oz_media

Most major engines require manual submission, a form to be filled our describing your site, it's purpose etc. Yahoo will have live staff members crawl and verify that your site is what you say it is and then rank it based on relevance.

Most require submission of your site's URL, they will then place it enqueue to be crawled by their bot and ranked for relevance.

This is why most peopl ecan't understand why they aren't found when they build a site, most of the time, it's never been properly submitted to begin with.

by jardinier

My hosting company -- mrpc.net -- provides site-building software: Site Studio 1.7 so that I can build my websites with minimal knowledge.

However I have taken great care to include as many keywords as possible -- up to one dozen -- and also a comprehensive description.

But as noted earlier, and confirmed by yourself, words within blocks of text are a high priority for search engines.

by sleepin'dawg

my searches and that is precisely what I did. I can't recall the engine I used but I posted Search, Searches, Searching and found all sorts of information on how to formulate a search, the Boolean method being among the best of them. Once you select a method and use it over any period of time, it becomes second nature to adopt that as your normal method of conducting a search and you tend to forget there are other methods as well. Don't ask me what they are; I've forgotten but if I have any problems with my current method, I know where to go to find alternatives; not that I can recall having had to do so.

[b]Dawg[/b] ]:)

by oz_media

You may also get ditched by many search engines that don't see ALT tags for images, and for using too many keywords, especially ones that aren't prominent in your pages. Each engine requires a different balance, I have yet to see site design software that offers such info.

This discussion is closed to new replies.