[surferboy]

I see there are a number of free and paid scripts out there (including a service offered by google) that will put a search bar into your website. Some of them use spider-like indexing to build the search database. I am building an academic, private website with password protected directories. In light of this, would one of these types of search technologies work? Does anyone have any advice on building a search engine for this type of purpose?

Thanks.

[jay3ld]

it will not be able to enter password protected directorys. but instead of taking time to make every directory password protected put a file called robots.txt in your web root and just tell it what directory its not allowed in:

Code:
User-agent: *
Disallow: /admin/
Disallow admin.php
Disallow /help/admin.php

doing this will stop any bots from touching your site.

Code:
User-agent: *
Disallow: /

[besson3c]

Originally Posted by surferboy

I see there are a number of free and paid scripts out there (including a service offered by google) that will put a search bar into your website. Some of them use spider-like indexing to build the search database. I am building an academic, private website with password protected directories. In light of this, would one of these types of search technologies work? Does anyone have any advice on building a search engine for this type of purpose?

Thanks.

If this is an academic site, Google offers a free University search program where you can have Google index your site, and provide a Google search field on your site.

Is your site database driven or just a bunch of static pages? It is much harder to do searches well on pages that consist of static text. Any script you find would simply scrape text from the page as best as it can. If your site is database driven, you could add keywords and/or page summaries for the search engine to search, or simply allow the search engine to go to town on the page content itself, if saved into a field.

[surferboy]

Good insight. It's mostly static information that I will update periodically. I was thinking of something like this.

[besson3c]

Originally Posted by surferboy

Good insight. It's mostly static information that I will update periodically. I was thinking of something like this.

I'm sure that this script is designed to read the contents of each file and parse out the relevant information. You may need SSH access to set this up as a cron job to automatically update its index every night (or however often), unless this search works in real-time (and doens't become slow once the site starts to become pretty big). The beauty of a database system is that searches are always up-to-the-minute and instantaneous.

[surferboy]

I like what you're sayin'. Since I'm not too terribly deep into writing the pages, can you point me to a basic tutorial on building a database site? I have dreamweaver, bbedit and some other tools at my disposal.

I'll read a little first- but if you know of a good primer on how to do this- and how to incorportate a search engine- I'd be grateful....

[surferboy]

OK, I've done some reading on content management systems, etc. Wow! It's a whole different world- and seems complex to. I know there are a lot of systems like typos2, xoops, and Mambo out there that help you with this sort of stuff, but I don't have a clue how to implement these systems. I already have a page design, etc. Is it really worth it to separate content from design? How hard is it for me to take my design and create a database driven site?

Wow- this is a left turn for me!

[jay3ld]

id go with phpnukes cms really good and you can do just about anything with it. including making your own design.

http://nukecops.com

there is even a googletap for nuke that will do allot of mod rewrite in your .htaccess for you so google can index it better

[besson3c]

Content management systems like Mambo are very general purpose, flexible, adaptable. There are a lot of features you may never use, they aren't streamlined to the particular needs of a site. If you want a CMS as simple and customized for your site as possible, it would probably be best to build one of your own.

Database driven sites consist of 3 "layers": the back-end, faceless database (e.g. MySQL or PostgreSQL), the part that is rendered in a browsers and your visitors interact with, and a layer driven by some sort of middleware language (e.g. PHP) which serves as a bridge between these two - something that can take some database values, and output them to a page. Another popular component within this mix is phpMyAdmin, which provides a web-based GUI to MySQL, so that you can manipulate the data in your database without having to use the MySQL command line, or produce some PHP scripts to help manage your content.

You can build these components for your Mac to play around with using the instructions I've created here.

These aforementioned multi-purpose CMS tools generally require either MySQL or PostgreSQL installed, in addition to a working web server with PHP support. I've always created my own CMS tools for my customers, but if these tools will only be used by yourself, something like Mambo may not be a bad idea. However, phpMyAdmin may provide all that you need too...

If you are comfortable with Unix, you can probably setup a server running Apache, PHP, MySQL, phpMyAdmin, and Mambo pretty easily. This would provide the tools you need for a database driven site.

PHP is very easy to learn, because you can use it alongside of your HTML.. for example:

<p>
Hello there. <b>this is typical HTML</b>.<br>
Blah.
</p>

<?php
print "<i>This is a sentence generated by \'PHP\"</i>";
?>

<p>
More HTML
</p>


If you want maximum control over your site in this environment, learning some PHP would not be a bad idea, although not strictly required if you plan on using these multi-purpose tools.

Let me know if this helps!