Welcome to the MacNN Forums.

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

You are here: MacNN Forums > Software - Troubleshooting and Discussion > Developer Center > Getting spiders to index dynamic sites...

Getting spiders to index dynamic sites...
Thread Tools
SirCastor
Professional Poster
Join Date: Jan 2001
Location: Salt Lake City, UT USA
Status: Offline
Reply With Quote
Feb 1, 2006, 02:35 PM
 
Anyone have any thoughts about this? I don't know specifically how spiders go about finding sites, do they just call up the page with an http request? I need to get some exposure for a website that I've designed.

My Sites aren't ridiculously dynamic. They plug in the head of a webpage and the tail end. It gives me a little flexibility in overal site design while keeping the pages managible. Any suggestions? Recommendations? (I'm not using any database stuff here, so that's not an issue.)
2008 iMac 3.06 Ghz, 2GB Memory, GeForce 8800, 500GB HD, SuperDrive
8gb iPhone on Tmobile
     
moodymonster
Mac Elite
Join Date: Sep 2003
Location: London
Status: Offline
Reply With Quote
Feb 1, 2006, 06:40 PM
 
just make sure your site's internal links are good ie links to the main pages are there on every page and so on. And use tags like h1, h2 etc so spider's can know how to breakdown the info. Repeat keywords if you can.

Search for SEO (Search Engine Optimisation) in google for more in depth stuff.

Get the site linked to by putting it's address around, in your sig here, web directories, anywhere you can link to the site from.

The more links to your site spiders find, the higher they will rank it.

You can find out how sites rank by looking at http://www.alexa.com/ (Click on traffic rankings at the top).
     
mr_friendly
Fresh-Faced Recruit
Join Date: Mar 2006
Status: Offline
Reply With Quote
Mar 3, 2006, 10:39 AM
 
not sure if this is part of your question, but if you want the spiders to "find" each of your dynamically generated pages, you could put on your SiteMap page a listing of the dynamic pages. Let's say you have a dynamically generated page to display a database's article item. Insert onto the SiteMap page a For-Loop that would grab a resultset of all the articles in the db, then write hyperlinks to the dynamic page for each item.

Yes, Moody is correct that hte more links you have the faster the spiders will index your site. I also agree with Moody's suggestion of researching SEO. You may want to check out these sites for more info: http://seochat.com/ and http://cre8asiteforums.com/ and http://www.seomoz.org/beginners.php

Good luck.
     
CaptainHaddock
Grizzled Veteran
Join Date: Apr 2004
Location: Nagoya, Japan • 日本 名古屋市
Status: Offline
Reply With Quote
Mar 4, 2006, 12:48 AM
 
One issue is how your page URLs are formed. If they have query strings, e.g. " yoursite.com/page.php?dynamicstuff", Google and other engines might not index those pages, considering them not to be permanent content.

If your pages have normal URLs and they're all linked, you should have no problem.
     
Love Calm Quiet
Mac Elite
Join Date: Mar 2001
Location: CO
Status: Offline
Reply With Quote
Mar 4, 2006, 05:05 AM
 
It's strange... Google does seem to de-emphasize links with "php?" in them. Yet some get through.
For example:
Doing a google on ... CaptainHaddock and MacNN ... yields:
http://forums.macnn.com/showthread.php?t=278598
(though not 759 links)
And googling for CaptainHaddock yields several at macosxhints.

I've seen references in blog software to some process that produces "static links" or some such out of the dynamic blog content. Apparently there's some way to have the best of both.
Anybody know exactly what they're doing?
TOMBSTONE: "He's trashed his last preferences"
     
Chris O'Brien
Grizzled Veteran
Join Date: Nov 2003
Location: Hebburn, UK
Status: Offline
Reply With Quote
Mar 4, 2006, 06:40 AM
 
They're probably using mod_rewrite or scripting it in themselves, or maybe even just using a framework like rails.
Just who are Britain? What do they? Who is them? And why?

Formerly Black Book
     
CaptainHaddock
Grizzled Veteran
Join Date: Apr 2004
Location: Nagoya, Japan • 日本 名古屋市
Status: Offline
Reply With Quote
Mar 4, 2006, 11:52 AM
 
Yeah, almost every blogging engine uses mod_rewrite to make every link look like a real file, e.g."myblog.com/articles/someone-is-stalking-me-using-google", and then Apache converts that on-the-fly into a query string for the site engine. I gather it's pretty easy to have ".php?foo" look like "/foo" to search engines that way.
     
Love Calm Quiet
Mac Elite
Join Date: Mar 2001
Location: CO
Status: Offline
Reply With Quote
Mar 4, 2006, 11:58 PM
 
Originally Posted by Chris O'Brien
They're probably using mod_rewrite or scripting it in themselves, or maybe even just using a framework like rails.
Thank you, Chris. That shows me right where I need to start expanding my PHP sophistication.
TOMBSTONE: "He's trashed his last preferences"
     
   
 
Forum Links
Forum Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Top
Privacy Policy
All times are GMT -4. The time now is 08:50 AM.
All contents of these forums © 1995-2017 MacNN. All rights reserved.
Branding + Design: www.gesamtbild.com
vBulletin v.3.8.8 © 2000-2017, Jelsoft Enterprises Ltd.,