Recommended Gateway Sites for the Deep Web

And Specialized and Limited-Area Search Engines

 Michael Hunter    12/9/03
Web version:  http://people.hws.edu/hunter/deepwebgate03.htm

 

This portion of the Internet consists of information that requires interaction to display such as dynamically-created pages, real-time information and databases.  Currently estimated to be over 100 times larger than the surface web,  the Deep Web houses billions of documents in databases and other sources, over 95% of which are available to the public.  As crawler-based search engines cannot access these documents, specialized sources such as these currently provide our only access.

 

General Gateways   |    Humanities   |   Social Sciences  

Science and Technology   |   Health Sciences  
Business and Government  |   Reference, Popular Culture   |  Other


General Gateways:

 

Invisible Web Directory (highly recommended)  http://www.invisible-web.net/

            An excellent gateway to some of the best research-oriented invisible web resources available.  Arranged by category and sub-category. Moderate in size with sites well selected and organized.  Maintained by Chris Sherman and Gary Price.

 

Resource Discovery Network    http://www.rdn.ac.uk
                A well-annotated listing of Deep Web resources. Compiled and maintained by over 70 educational and research organizations, largely from the UK. Hubs include
*  ALTIS - Hospitality, Leisure, Sport and Tourism
*  Artifact - Arts and Creative Industries
*  BIOME - Health and Life Sciences
*  EEVL - Engineering, Mathematics and Computing
*  GEsource - Geography and Environment
*  Humbul - Humanities
*  PSIgate - Physical Sciences
*  SOSIG - Social Sciences, Business and Law

 

Infomine  (highly recommended)           http://infomine.ucr.edu

            U. of California at Irvine’s directory of excellent academic research sites now includes search capability for all databases found on the sites.

 

WWW Virtual Library   (highly recommended)   http://vlib.org/

          Includes traditional disciplines, area studies, minority studies and other academic and research areas.  Excellent annotations with dates of coverage.

 

Dmoz.org   (recommended)                  http://dmoz.org/

                The librarian-created Open Directory offers this comprehensive listing of reviewed, quality websites and databases by subject area. 

 

Profusion   (recommended)       http://profusion.com/

            A portalized search site that searches “invisible web” as well as conventional crawler-based search engines related to your search term(s).  Search box at top accesses several large general engines.  Deep Web content must be accessed through subject areas below.

 

Completeplanet   (highly recommended)         http://completeplanet.com/

            A human-compiled directory of deep web resources, with broad subject coverage but fewer sites per category.  Ratings available for each site listed.

  

Flipper      http://flipper.com/

Strong in news and e-commerce.  Actually performs the searches in the databases, returning results in drop-down menus.  Crawler results are paid placements from searchboss.com via spaex.com.

               

IncyWincy          http://www.incywincy.com/

                Covers over 100,000 embedded databases, identifying those relevant to the search and

searches them. 

 ______________________________________________________________________________

Humanities:

 

WWW Virtual Library   (highly recommended)   http://vlib.org/

          Includes traditional disciplines, area studies, minority studies and other academic and research areas.  Excellent annotations with dates of coverage. 

 

Infomine      http://infomine.ucr.edu/

            The University of California’s premier resource offers a listing of over 140 specialized and general search engines and services.  Choose “Internet Enabling Tools” at the homepage and type search engines and a subject in the search box.

 

Fossick    (highly recommended)    http://fossick.com/

                One of the richest gateways to specialized search engines.  Main categories open into more specialized topics, with a wide array of sites.  Located in Australia, offers an international orientation.  Descriptions given if you “hover”.

 

Humbul     (highly recommended)     http://www.humbul.ac.uk/

            Covers languages and literatures, philosophy, history, religion, archaeology, American studies and other related areas.  Maintained by the University of Oxford.

 

Voice of the Shuttle     (highly recommended)   http://vos.ucsb.edu/

            Top choice for Humanities resources.  Maintained by Alan Liu of UC Santa Barbara.  Includes Anthropology, Area Studies and Minority Studies as well as other Humanities disciplines.

 

Bloomsbury Research Centre    http://www.bloomsburymagazine.com/

            Provided by Bloomsbury Publishers, this metasite includes dictionaries, quotations and literary reference works as well as links to genre-specific gateways, awards and festivals and book trade information.

 

Shakesearch     http://www.rhymezone.com/shakespeare/

            Full-text search of all works by Shakespeare with links to related sources

 

Bases de données bibliographiques sur la Toile

          (recommended)         http://www-bu.univ-paris8.fr/Ref/DocBdd.html

            Includes free bibliographic databases arranged by academic discipline (check the bottom of the page).  Also includes gateway sites to library catalogs worldwide.

 

History Matters  (highly recommended)   http://historymatters.gmu.edu/

            Features  media-rich resources for teaching U.S. history at the secondary and college levels.  The associated site www.history is a collection of quality sites with extensive descriptions.

__________________________________________________________________________________________________ 
Social Sciences:

Rutgers University Research Guides:  Social Science and Law  (highly recommended) 

            http://www.libraries.rutgers.edu/rul/rr_gateway/research_guides/research_guides.shtml#social_sciences

            Excellent source, covering Environmental Studies, Women’s Studies, Latin American Studies, Library & Information Science and Alcohol Studies as well as the traditional social science disciplines.

 

WWW Virtual Library   (highly recommended)   http://vlib.org/

          Includes traditional disciplines, area studies, minority studies and other academic and research areas.  Excellent annotations with dates of coverage. 

 

Infomine      http://infomine.ucr.edu/

            The University of California’s premier resource offers a listing of over 140 specialized and general search engines and services.  Choose “Internet Enabling Tools” at the homepage and type search engines and a subject in the search box.

 

Fossick    (highly recommended)    http://fossick.com/

                One of the richest gateways to specialized search engines.  Main categories open into more specialized topics, with a wide array of sites.  Located in Australia, offers an international orientation.  Descriptions given if you “hover”.

 

Statistical Resources on the Web        http://www.lib.umich.edu/govdocs/stats.html

            One of the best, national and international in scope, with links to U. of Michigan’s excellent Foreign Governments Document Collection.

 

Social Science Information Gateway (UK & Europe)       http://sosig.ac.uk/

            Offers articles, reports, databases and organizations in major social science fields, business and women’s studies.  Most resources are UK and European-based.  The Grapevine feature accesses academic departments, latest conferences and resumes.  Part of the RDN.

 

SocioSite      http://www.pscw.uva.nl/sociosite

          Sociology gateway maintained by the University of Amsterdam.  Extensive and well-structured subject access to a wide range of sociological topics.  Includes more specialized gateway sites, association home pages, articles and conference papers.  Predominantly English language sources.

 

Psychcrawler   (highly recommended)   http://www.psychcrawler.com/

            A product of the American Psychological Association created to provide quick access to quality Internet content in the field of psychology. 

_______________________________________________________________________________

Business and Government:

 Rutgers University Research Guides:  Business  (highly recommended) 

            http://www.libraries.rutgers.edu/rul/rr_gateway/research_guides/busi/business.shtml

          Excellent source for Business News, Market Research, Company Profiles, Job Searching and Financing Your Business.  National and international in scope.

 

First Gov      (highly recommended)       http://firstgov.gov/

            New government information gateway.  Comprehensive and well maintained.

 

Direct Search   (recommended)          http://www.freepint.com/gary/direct.htm

Compiled by Gary Price of George Washington University, this gateway concentrates on research resources.  Large site and somewhat difficult to use, but strong in business, economics, news and speech transcripts

               

GPO Access  (recommended)             http://www.access.gpo.gov/

                Best interface for deep web content on US Government websites with full-text databases

 

Beaucoup  (highly recommended)      http://www.beaucoup.com/

            Collection of over 2,500 searchable databases and search engines

 

Business.com      http://www.business.com/

            Information pages on 64,000+ public, private and international companies as well as Human Resources, Marketing and Consumer Services.

 

EGovLinks       http://egovlinks.com/

            Maintained by an independent agency, this site offers sites not available at First Gov.  Challenging to navigate.

 

Search.com      http://www.search.com/

Gateway to over 800 engines.  Subject access is through Directory Categories, then subtopics then enter search term(s)

________________________________________________________________________________

Science and Technology:

 SciCentral      http://www.sciquest.com

            NOTE:  Choose SciCentral within the GoTo Box.  Gateway site to directories, databases and metasites.  Organized by broad subject area (Biological, Health, Engineering, Physical, Chemical, etc.) and resource type.  Strong in breaking science news with links to current research highlights

 

Sci-TechResources      http://www.scitechresources.gov/

            Searchable database with descriptions of sites that are key entry points to U.S. government science and technology resources

 

Scientific Search Engines      http://www.scientific-search-engines.com/

            Actually a gateway to quality search services and sites in all academic disciplines in English and German.  Maintained by LLEK in Germany.

 

Fossick    (highly recommended)        http://fossick.com/

                One of the richest gateways to specialized search engines.  Main categories open into more specialized topics, with a wide array of sites.  Located in Australia, offers an international orientation.  Descriptions given if you “hover

 

Scirus  (highly recommended)        http://www.scirus.com/

            Created by Elsevier Science, this gateway covers scientific, technical and medical sources with over 60 million pages, listing Web resources as well as the fee-based Elsevier Science Collection.  Excludes sites with no scientific content.  Includes pdf and  postscript files and indexes full text of all documents.  Fine Advanced Search features.  Classified into broad subject areas that can be searched individually.  Customization offered

 

Martindale’s Reference Desk  (highly recommended)   http://www.martindalecenter.com

                Maintained by James Martindale of UC-Irvine, a gateway to high-quality sources in a wide range of reference topics, especially strong in sciences and health sciences.

 

WWW Virtual Library  (highly recommended)    http://vlib.org/Science.html

            Comprehensive metasites for biosciences, health, earth sciences, physics and chemistry.  Maintained by universities and research institutions.  (For agriculture, computer science and eingeneering see http://vlib.org/.

 

Biology Links      (highly recommended)     http://mcb.harvard.edu/BioLinks.html

                Includes data banks, model organisms, biological software directories and other related gateway sites.  Maintained by the Dept. of Microbiology at Harvard.

 

Internet Directory for Botany     (highly recommended)      http://www.botany.net/IDB

            Rich collection of databases, links and specialized data resources.  Maintained by an international team of scholars. 

 

Virtual Search Engines (highly recommended)     http://www.virtualfreesites.com/search.html

            Classified listing of over 10,000 search services and subject-specific metasites.  Especially strong in computers and information technology.  Unusually excellent and helpful site descriptions.  Geared to the professional searcher.

 

Internet.com      http://internet.com/

            One of the most complete gateway sites of this type, covering all operating systems, markup languages, wireless, downloads and more;  comprehensive and up-to-date.

 

Calculators On-line Center   (highly recommended)   http://www.martindalecenter.com/calculators.html

                The most comprehensive collection of this type on the Internet. 

 

Infomine      http://infomine.ucr.edu/

            The University of California’s premier resource offers a listing of over 140 specialized and general search engines and services.  Choose “Internet Enabling Tools” at the homepage and type search engines and a subject (eg. Science) in the search box.

 

Search4Science    http://search4science.com/

            Although this service forwards all searches to Google (with identical results) it also offers useful lists of links to other gateway sites, by broad subject area.

 

Search.com      http://search.cnet.com/

Gateway to over 800 engines.  Subject access is through Directory Categories, then subtopics then enter search term(s)

 

Statpages         http://members.aol.com/johnp71/javastat.html

                Web pages that perform calculations, probabilities, plotting and other interactive programming environments. 

_______________________________________________________________________________

Health Sciences:

 Healthfinder      (highly recommended)     http://www.healthfinder.gov/

            The National Institutes of Health maintain access to over 100 health-related limited-area engines and database search pages from government and other sites as well.  Consumer-oriented information covering prevention and self-care, infants, teens, senions, fraud and complaints and much more  Enter search engines in the search box at the URL given above.

 

Martindale’s Health Sciences Guide   (highly recommended)

  http://www.martindalecenter.com/HSGuide.html

                A true Virtual Medical Center covering general and specialized medical fields as well as allied areas such as Dental, Medical Law, Nutrition, Pharmacy and Veterinary.  Includes online courses, tutorials, databases and video files.

 

RxList                 http://rxlist.com/

            In-depth drug information covering over 4,000 U.S products.  Each entry contains description, clinical pharmacology, indications, usage, contraindications, warnings, precautions, drug interactions,adverse reactions, drug abuse and dependence, dosage and administration and animal pharmacology.

 

Acurian   http://www.acurian.com/patient/

            Provides comprehensible searchable listings of clinical trials actively enrolling participants, detailed drug information and news and commentaries from leading medical and research programs.

 

WWW Virtual Library  (highly recommended)    http://vlib.org/Medicine.html

          Strong in traditional medical areas, allied health fields and alternative medicine.  (For medical research see Biosciences section of the Virtual Library

________________________________________________________________________________

Reference, Popular Culture, Lifestyle:

 

Martindale’s Reference Desk  (highly recommended)   http://www.martindalecenter.com

                Maintained by James Martindale of UC-Irvine, a gateway to high-quality sources in a wide range of reference topics.   Excellent collection of free online foreign language dictionaries.

  

WWW Virtual Library   (highly recommended)   http://vlib.org/

          Includes encyclopedias, dictionaries, biographical sources, geographical sources, statistics, almanacs and yearbooks.  Excellent annotations with dates of coverage.

 

Allsearchengines       http://allsearchengines.com/

            Especially strong in listings of search engines and metasites for Images & Sounds, Games, Software & Computers, Entertainment, Books and “Site Awards and Best-of-the-Web”.  Little or no description of sites given.

 

Foundation Center Online      http://fdncenter.org/

                Includes an excellent “Online Library” of resources on foundations and non-profits and grant-seeking as well as a directory of grantmakers from all types of business and private sources.

 

Internet Public Library  http://www.ipl.org/

            One of the best for public library needs, including exhibits, news, teens, youth collections and tutorials on the research process.  Now includes guides formerly known as Argus Clearinghouse.

 

First-Search    http://first-search.com/

            Fine collection of engines and metasites, especially strong in Reference and Lifestyle areas.

 

Search.com       http://www.search.com/

                Strong in popular culture, tech, news and reference  Offers meta-interface to foreign language search engines in over 40 countries. 

________________________________________________________________________________

Other:

 

Blogdex    http://blogdex.net

                A search engine maintained at MIT for weblogs,  postings by individuals  often centered around a particular topic or issue.  Quality can vary from the polemical to highly technical.  A growing genre and worth knowing about.   Blogspot.com is a good place to start your own blog.

           

Dmoz.org   (recommended)               http://dmoz.org/

                The librarian-created Open Directory offers this comprehensive listing of blogs by subject area.  Path:  Computer/Internet/On the Web/Weblogs

               

Search Systems        http://www.searchsystems.net

                Provides access to over 2,000 public records and legal databases worldwide.

 

Virtual Gumshoe         http://virtualgumshoe.com/

            Gateway to locator databases, from Adoption Resources to Voters Records.  Associations and individuals covered extensively.

 

NewPages                        http://www.newpages.com/

            Portal to information on independent bookstores, independent publishers, literary periodicals, alternative newsweeklies and more. 

 

SearchPDF        http://searchpdf.adobe.com/

            A search site offered by Adobe exclusively for files in pdf format.  Use in conjunction with Google for locating pdf files. 

 

Disinformation      http://www.disinfo.com/

            Collects and indexes the most “hidden” and subversive material on the ‘Net.  Over 30 million documents.

_____________________________________________________________________

Country or Region-Specific:

 

Beaucoup  (highly recommended)      http://www.beaucoup.com/

            Collection of over 2,500 searchable databases and search engines

 

Search Engine Colossus       (highly recommended)     http://www.searchenginecolossus.com/

            Over 920 search engines and subject directories, organized by country with brief descriptions of scope, language and origin of each.

 

Finderseeker     http://www.finderseeker.com

            The only gateway that allows cross-searching for specialized engines by topic AND country, eg. education and Italy.

 

Country Search Engines    http://www.philb.com/countryse.htm

            Includes subject metasites focused on individual countries as well as search engines and subject directories.  Up-to-date, but without annotations.

 

Main Portals       http://www.mainportals.com/#countries

            A listing of major, portalized search services in 84 countries

 

Nationsonline      http://www.nationsonline.org/oneworld/

            Smaller listing of search services in 45 countries, with each designated as a search engine or subject directory (index).

 

Official City Sites      http://officialcitysites.org/    (highly recommended)

            Covers cities in the U.S, Canada, UK, France, Germany, Netherlands, Australia, New Zealand and Japan.

 

Country-specific Information Sources

 Nationsonline      http://www.nationsonline.org/oneworld/

          A wealth of up-to-date information on individual countries, cultural, political, social and econcomic.

 

ELDIS        http://www.eldis.org

            Information sources on development and the environment, country-specific and international in scope

 

World Skip           http://www.worldskip.com/

            News, information and “International Marketplace”; strong in e-commerce.