This program is an extension of EIT's wwwwais, which in turn was very loosely based on the Perl waisq interface program that comes with NCSA's httpd.
http://www.rru.com/~meo/useful/swish.html
.
http://www.rru.com/~meo/useful/swish-1.1.3.tar.gz
.
You may want to put SWISH things somewhere such as
/usr/local/httpd/conf/swish
. You may also want
to create a directory to hold SWISH databases, such as
/usr/local/httpd/swish/sources
, or you can
simply keep them in the indexed directories.
Note that versions of freeWAIS other than 2.02, including the commercial WAIS software, may or may not work for what you want to do. Note that all WAIS-related software has strange quirks and bugs - you may have to experiment a good deal...
If you're only using freeWAIS to do Web searching,
you may want to put it all somewhere web-related such as
/usr/local/httpd/wais
.
You'll also want to create a directory to hold the WAIS
database for your site, somewhere like
/usr/local/httpd/wais/sources
.
After you've compiled (and installed) freeWAIS and/or SWISH,
make sure the waisindex, waisq,
waissearch, and/or swish programs
are somewhere in your executable path (somewhere such as
/usr/local/bin
).
Now you'll want to put a script such as the following C-shell script somewhere where you can run it. This script will index your Web site into a searchable database that WAIS clients can read.
#!/bin/csh set rootdir = /usr/local/www # This is the root directory of the Web tree you want to index. set index = /usr/local/httpd/wais/sources/index # This is the name your WAIS indexes will be built under. # Index files will be called index.* in the /usr/local/httpd/wais/sources # directory, in this example. set indexprog = /usr/local/httpd/wais/waisindex # The full pathname to your waisindex program. set nonomatch cd $rootdir set num = 0 foreach pathname (`du $rootdir | cut -f2 | tail -r`) echo "The current pathname is: $pathname" if ($num == 0) then set exportflag = "-export" else set exportflag = "-a" endif $indexprog -l 0 -nopairs -nocat -d $index $exportflag $pathname/*.html $indexprog -l 0 -nopairs -nocat -d $index -a $pathname/*.txt $indexprog -l 0 -nopairs -nocat -d $index -a $pathname/*.c $indexprog -nocontents -l 0 -nopairs -nocat -d $index -a $pathname/*.ps $indexprog -nocontents -l 0 -nopairs -nocat -d $index -a $pathname/*.gif $indexprog -nocontents -l 0 -nopairs -nocat -d $index -a $pathname/*.au $indexprog -nocontents -l 0 -nopairs -nocat -d $index -a $pathname/*.hqx $indexprog -nocontents -l 0 -nopairs -nocat -d $index -a $pathname/*.xbm $indexprog -nocontents -l 0 -nopairs -nocat -d $index -a $pathname/*.mpg $indexprog -nocontents -l 0 -nopairs -nocat -d $index -a $pathname/*.pict $indexprog -nocontents -l 0 -nopairs -nocat -d $index -a $pathname/*.tiff @ num++ end echo "$num directories were indexed."
Make sure you've configured everything correctly, then run this script. The more files your Web site has, the longer indexing will take.
Taking as an example the above configuration in the script,
you'd have the directory /usr/local/httpd/wais/sources
and a number of files with the prefix index
in the
directory. The name of the database you've just created is
index.src
.
If you're using SWISH, you'll want to create a configuration file (one for each index, perhaps) . The configuration file below will do pretty much the same thing as the above script for waisindex:
# SWISH configuration file IndexDir /usr/local/www # This is the root directory of the Web tree you want to index. IndexFile /usr/local/httpd/swish/sources/index.swish # This is the name your SWISH index will be built as. IndexOnly .html .txt .c .ps .gif .au .hqx .xbm .mpg .pict .tiff # Only files with these suffixes will be indexed. IndexVerbose yes # Put this to show indexing information as swish is working. NoContents .ps .gif .au .hqx .xbm .mpg .pict .tiff # Files with these suffixes won't have their contents indexed, # only their file names.
Assuming you named the config file swish.conf
and placed it in /usr/local/httpd/swish/
, you
now type:
swish -c /usr/local/httpd/swish/swish.confto run swish and index your site.
Taking as an example the above configuration in the
script, you'd have the directory
/usr/local/httpd/swish/sources
and one
file called index.swish
in the directory.
The name of the database you've just created is
index.swish
.
http://www.rru.com/~meo/useful/w4ais-3.1.tar.gz
.
Please read the license before building the source
or using the program.
Should you have licensing or business-related questions
only, please contact SUzi Styrofoam at
suzi@rru.com. Depending
on several issues, you may also need to contact EIT, but
if so, we will tell you, or it will be apparent from the
licensing documentation.
The orginal wwwwais required you to specify a configuration
file path in the source code. You might want to make the path
to this file something like
/usr/local/httpd/conf/w4ais/site.conf
.
For w4ais, this simply becomes the default path; others
may be specified by the WWWW_ variables. These are
typically set by the indexing program.
Now compile w4ais - it seems to compile fine
with gcc
.
Put the program in your Web server's
/cgi-bin
directory. It's a
CGI
(Common Gateway Interface) program, so it
should be in this directory. Make sure it's
executable by everyone.
# w4ais configuration file # 30/May/1998 # Miles O'Neal, meo@rru.com # Documentation at http://www.netads.com/Doc/w4ais.html # #################################################################### # # NEW STUFF [meo@rru.com 12/Jan/98] Body /www/pages/rru/srchbody.inc # file containing BODY tag (for colors, etc) [if unset, default provided] PromptString "Search for:" # prompt to place before keywords text field # Only one of "Instructions" or "Help" should be provided. #Instructions /www/pages/rru/srchinst.inc # file containing instructions text (after keyword text field) # [defaults are provided] Help /~meo/srchinst.html # URL containing instructions text (after keyword text field) Trailer /www/pages/rru/srchtrail.inc # file containing trailer text [if unset, no trailer text] # #################################################################### PageTitle /www/pages/rru/srchhead.html #PageTitle "Searches-B-We Search Results" # If this is a string, it will be a title only. # If it specifies an HTML file, this file will be prepended to w4ais results. SelfURL "http://www.rru.com/cgi-bin/w4ais" # The self-referencing URL for w4ais. MaxHits 50 # The maximum number of results to return. SortType score # How results are sorted. This can be "score", "bytes", "lines", # "title", or "type". Format table # How results are formatted. This can be "brief", "standard", or # "table". wwwwais only supported the "standard" format. AddrMask "all" # Only addresses specified here will be allowed to use the gateway. # For the above mask option, these rules apply: # 1) You can use asterisks in specifying the string, at either # ends of the string: # "192.100.*", "*100*", "*2.100.2" # 2) You can make lists of masks: # "*192.58.2,*.2", "*.100,*171.128*", ".58.2,*100" # 3) A mask without asterisks will match EXACTLY: # "192.100.58.2" # 4) Define as "all" to allow all sites. #WaisqBin /usr/local/bin/waisq #WaissearchBin /usr/local/bin/waissearch SwishBin /www/httpd/cgi-bin/swish # The full path to your waisq, waissearch or swish program. SourceRules replace "/www/pages/rru/" "http://www.rru.com/" SourceRules replace "/u/" "~" SourceRules replace "public_html/" "" #SourceRules prepend "http://www.rru.com/cgi-bin/munge?" #SourceRules append "?$KEYWORDS#first_hit" #SwishSource /www/pages/rru/index.swish "Roadkills-R-Us" #WaisSource /usr/local/httpd/wais/index/index.src "Search EIT's Web (bolded results)" #WaisSource quake.think.com 210 directory-of-servers "WAIS directory of servers" # For waisq sources: # WaisSource full_path_to_source/source.src "description" # For waissearch sources: # WaisSource host.name port source "description" UseIcons yes # yes = use icons, no = don't IconUrl /icons # Where all your icons are kept. TypeDef .html "HTML file" $ICONURL/text.xbm text/html TypeDef .htm "HTML file" $ICONURL/text.xbm text/html : TypeDef .src "WAIS index" $ICONURL/index.xbm text/plain TypeDef .?? "unknown" $ICONURL/unknown.xbm text/plain # Information for figuring out file types based on suffix. # Suffix matching is case insensitive. # TypeDef .suffix "description" file://url.to.icon.for.this.type/ MIME-type # You can use $ICONURL in the icon URL to substitute the root icon directory.Variables such as SelfURL, SortType, and MaxHits should be fairly explanatory; see below for a list of options. You can use the AddrMask option to allow only certain sites to search your site, and WaisqBin, WaissearchBin, and SwishBin are pointers to your
waisq
, waissearch
, and
swish
programs, respectively.
waisq
,
specify the full pathname to the source you want to search (ending
in .src
) and a short description of the database.
For WAIS servers, specify the host name, port, source name
(without a .src
) and a short description. For SWISH
indexes, specify the full path to the source (ending in
.swish
) and a short description.
If you've specified more than one WAIS and/or SWISH source in your configuration file, a pop-up menu will appear on the w4ais page with the descriptions as menu items (in the order that you specified the sources). Choose one and start searching! If you have only one source specified, no menu will appear. Just enter your search text and hit return to search that source.
There are three operations you can specify: replace, append, and prepend. They will parse the pathname in the order they appear in the configuration file. More than one command and its arguments can appear on the same line, but it's easier to read when commands are broken up over a few lines. You can't put a command and its argument(s) on different lines, however.
Commands apply to the WAIS or SWISH source specified just before the commands. Here's the syntax:
replace "the string you want replaced" "what to change it to" This replaces all occurrences of the old string with the new one. prepend "a string to add before the result" append "a string to add after the result"In any command argument,
$KEYWORDS
will be
replaced with the keywords you used to search, so you
can pass them to filtering programs that can use them.
One good program is
print_hit_bold.pl,
a Perl program which bolds the found text in search results.
Study the above sample configuration file and try things out. You'll find you can do a lot of nifty things with WAIS sites and filters.
On a TypeDef line, you need to specify the suffix for the particular type(with a period), a short description to include in results (this shouldn't be any more than two or three words), the URL to the icon representing the file type (unused if you're not displaying icons), and the MIME type corresponding to the particular type.
The information specified with the suffix
.??
will be associated with all other files
that w4ais can't figure out.
<FORM METHOD="get" ACTION="/cgi-bin/w4ais"> Search for: <INPUT TYPE="text" NAME="keywords" SIZE=40> <INPUT TYPE="submit" VALUE=" Search "> </FORM>Enter keywords for your search. You can try both single and multi-word searches, separating words with ``and'', ``or'' and/or ``not''. You may use parentheses for grouping. A trailing asterisk (``*'') matches all words beginning with whatever precedes the asterisk.
NOTE: The above all work for swish; I'd hope they also work for waisq or waissearch, but don't know.
The waissearch program does things remotely by contacting WAIS servers on different machines, each of which has their own databases. In order for waissearch to do its thing, you need to tell it a machine name and a port to connect to, and that machine needs to have a WAIS server of their own running on that port.
By telling w4ais to use the waissearch program and specifying a host name and port in the URL (see below), you can search WAIS databases on other machines. If you wish to run your own WAIS server on your machine, make sure you have the waisserver program (from the freeWAIS package) and run it like this (all on one line with no returns):
./waisserver -p 2010 -d /usr/local/httpd/wais/sources -e /usr/local/httpd/logs/wais.log &This will run a WAIS server on your machine on port 2010. This server should log its results to a file named
wais.log
and will search through any databases
located in the /usr/local/httpd/wais/sources
directory.
Say you have a database named "index.src" in that directory and you've started your WAIS server with the line above. To search "index.src" using your server, you could call w4ais with a URL like this (all on one line, of course):
http://foo.bar.com/cgi-bin/w4ais?searchprog=waissearch& host=foo.bar.com&port=2010&source=index&keywords=heart+of+goldFor both SWISH and W4AIS, I recommend one configuration file each for every separate index. For instance, you might have an overall public site index, an internal index, and smaller indexes for individual users. The configuration files may be kept wherever you wish, but the obvious places are in the roots of the various page trees, or in central locations in the server config directories. I usually prefer the latter.
conffile example: /cgi-bin/w4ais?conffile=/www/httpd/conf/wwwwais/meo.conf description: Specifies the config file. One will be compiled in as a default, but you can have multiple files so that different directories, users, projects, etc. can be configured with different appearances, defaults, etc. selection example: /cgi-bin/w4ais?selection=none example: /cgi-bin/w4ais?selection="My+WAIS+server" example: /cgi-bin/w4ais?selection="Files+about+me" description: Specifies the index source to use (as set up in the configuration file). The argument is the source description. If selection is not defined as "none", any corresponding source information in the configuration file will override arguments and environment variables. searchprog example: /cgi-bin/w4ais?searchprog=waissearch description: Specifies the program to do the searching. This can be "waisq", "waissearch", or "swish". source example: /cgi-bin/w4ais?source=index.src description: Specifies the index database to search. sourcedir example: /cgi-bin/w4ais?sourcedir=/usr/local/sources description: Specifies the directory the index database resides in. (You can have multiple databases in the same directory) maxhits example: /cgi-bin/w4ais?maxhits=40 description: Determines the maximum number of URLs to return after a search. keywords example: /cgi-bin/w4ais?keywords=these+are+keywords description: You can specify the search keywords by using the keywords label. isindex example: /cgi-bin/w4ais?isindex=these+are+keywords description: This works the same as keywords. format example: /cgi-bin/w4ais?format=brief description: How results are formatted. This can be "brief", "standard", or "table". wwwwais only supported the "standard" format. sorttype example: /cgi-bin/w4ais?sorttype=bytes description: This determines how w4ais sorts its output. Valid arguments for sort are "score", "lines", "bytes", "title", and "type". version example: /cgi-bin/w4ais?version=true description: This gives the version information for w4ais and the search program it runs. host example: /cgi-bin/w4ais?host=eit.com description: This gives the host machine to search. This is only valid when using waissearch to do the searching. port example: /cgi-bin/w4ais?port=2010 description: This gives the port of the wais server that will be doing the searching. This is only valid when using waissearch with w4ais. useicons example: /cgi-bin/w4ais?useicons=yes description: This tells w4ais to use icons for different files in the search results. This can be "yes" or "no". iconurl example: /cgi-bin/w4ais?iconurl=http://www.eit.com/icons/ description: This tells w4ais the master URL at which to find the icons it may need. <keywords only> example: /cgi-bin/w4ais?these+are+keywords description: Keywords can be specified by themselves from "isindex" forms. This only works if no other options are used. <no arguments> example: /cgi-bin/w4ais description: Calling the program with no arguments brings up a blank field in which users can enter search keywords.Examples of specifying multiple options in the URL:
/cgi-bin/w4ais?isindex=these+are+keywords&maxhits=80 /cgi-bin/w4ais?source=index.src&keywords=test+searchThere are other ways w4ais can get variable information - you can specify these variables in forms using either the GET or POST methods, and PATH_INFO is supported as well, so you can make something like:
<FORM METHOD="get" ACTION="/cgi-bin/w4ais/host=quake.think.com&port=210&searchprog=waissearch"> Search for: <INPUT TYPE="text" NAME="keywords" SIZE=40> <INPUT TYPE="submit" VALUE=" Search "> </FORM>The above form sets w4ais up to search the WAIS server at
quake.think.com
, port 210. Note that you can
use the POST method as well in this example, and the result
will be exactly the same.
Environment variables are also supported - just put "WWWW_"
before variables and make everything uppercase. For instance,
instead of putting searchprog=waisq in a URL,
you can type setenv WWWW_SEARCHPROG waisq
and run
w4ais from a shell script.
If you come across any bugs or problems while searching a WAIS server, please send me the host, port, and source information so I can try things out and track down the problem.
If you know of other file indexers that can be interfaced with W4AIS, I highly recommend that you try modifying W4AIS to do so - after all, the point is to provide a single, easy to use method of searching many different types of indexes.