Google Spiders

[Martin]

Darren, when I go into the 'Viewing Who's Online' menu (by clicking the icon next to the list of people online at that moment), I almost always see several socalled 'Google Spiders' browsing the boards.

What is that?

Cheers, Martin

 

[phil_t]

I get that too!! Ive got a feeling that its people who do a Google search and then enter the Book Forum through a Google Results Page, but im not positive.

Phil

 

[Martin]

Yeah, could be it. Pity I didn't actually ask you, isnt it

No, seriously, I think you're right.

Cheers, Martin

 

[phil_t]

Fine, in that case i shall never be helpful again!!

Shouldnt be too much of an effort!!

Phil

 

[Martin]

No, I'm kidding, I think you're right.

Cheers, Martin

 

[RaVeN]

Slap him again Phil. Slap him again!


RaVeN

 

[phil_t]

Ooh, ooh, looky, an Altavista Spider!!

Phil

 

[murphyz]

Indeed, they are 'spiders' or 'robots' that are automated to scour the internet and log site content. Google have a system whereby sites that have the most links to them from popular sites move up the scale so that they appear nearer the top of the google list.

For example, thebookforum.com is currently #2 in a google search for 'book forum'. If we get enough visitors and, if other 'popular' sites link here, then the number one spot will be ours.
Please note, the linkage really does only work with legit sites. You can't create 40 pages with a link to here and expect anything to happen.

It's also possible to prevent the spiders from visiting the site by created a robots.txt that contains permissions for which directories the bots are allowed to view. They check for this file first and are set to follow directions, so if it tells them to mind thei own business, they have to turn around and go elsewhere. This is handy for things such as riddle sites that don't want future riddle pages to appear in search engines.

Mxx

 

[Martin]

Okay, thank you.

Cheers, Martin