Changes in Harvest 1.9 <author>Kang-Jin Lee <tt/lee@arco.de/ <date>2003-10-8 <abstract> After presenting long term goals of Harvest development in the last SINN 02 conference, this article describes recent progresses already achieved. This includes the description of changes in the gatherer where the embedded database was replaced by a more advanced embedded database and changes in the broker for the transition from current SOIF to XML. It also gives an overview of short term goals in Harvest development. <toc> <sect>Gatherer <sect1>Switched from GDBM to QDBM The most visible change is the switch of the embedded database from GDBM to <htmlurl url="http://qdbm.sourceforge.net/" name="QDBM">. QDBM's advantages are: <itemize> <item>Improved performance <item>Support for databases larger than 2GB on 32BIT systems <item>Large object support </itemize> The gatherer is much faster now on systems which gather over fast links like local network. In this scenario, the bottleneck still seems to be the database access, especially when gathering large amount (some GBs) of data on current hardware (ix86, 650MHz, IDE harddisks). While QDBM increased the gathering speed quite a lot in this scenario, the gatherer still suffers from the phenomenon of wild harddisk activity without actually doing any collection. Even though it happens much later than with GDBM, this still needs some investigation. It seems to be a filesystem buffer issue of the underlying OS, so perhaps this is the reason why there are/were some databases making raw access to the disk bypassing the filesystem layer of the OS. The improved speed won't be noticeable in situations where the bandwidth is the main limiting factor, of course and there is still some work to be done. In an additional attempt to speed up the gathering, the default configuration was changed to not to sort keywords, which saves two forks (sort and uniq) for each document. This will make summaries larger but I have to check if I can get rid of the "keywords" attribute alltogether or have to write a function to create word list. However, this will depends on how we will map SOIF to XML. I will also get rid of URI attribute from summaries, which was just introduced to improve the search results when using Glimpse. Current integration QDBM to Harvest is ugly but makes it easy to tweak the parameters. This will be cleaned up after fine tuning of QDBM parameters. <sect>Broker <sect1>Localized User Interface Thanks to many individuals on the net as well as some colleagues, we have a large number of localized user interface. The current list includes English, French, German, Italian, Russian, Spanish and Swedish and people continue to donate localized user interface. <sect1>Bibliometric Ranking Prototype A bibliometric ranking prototype is available at <htmlurl url="http://harvest.sourceforge.net/wip/rank-20021230.tar.gz" name="http://harvest.sourceforge.net/wip/rank-20021230.tar.gz">. It creates a database which stores the URL and number of URLs with a link pointing to it as tuples in a database. This could be used to evaluate the importance and relevance of a specific URL. After some testing there were following issuses that need to be addressed: <itemize> <item>There must be a rather complete pool of linked documents. If the pool is too small or incomplete, the number of links pointing to a document would usually be one, which isn't very helpfull to evaluate its importance. <item>Some documents like the footnotes pages in latex2html generated html documents have large number of pages pointing to them. This leads to bogus ranking results. </itemize> <sect1>Support for Zebra as Fulltext Indexer <htmlurl url="http://www.indexdata.dk/zebra" name="Zebra"> was merged into Harvest source tree. The usual steps to build, install and configure Harvest will result in a usable Zebra based serach system along with the old Glimpse based search system. For users, this shoud make it easy to migrate to Zebra. For developers, it is easy to compare the results between Glimpse and Zebra. <sect1>Data Modelling based on GILS based on Work of Peter Valkenburg / TERENA The current data model used in Harvest is based on Peter Valkenburg's work done for the TERENA project and is based on SOIF to GILS mapping from the Nordic Web Index (NWI). <sect1>Switch from SOIF to XML After some testing, I found flaws in the regular expression based SOIF input filter of Zebra. I decided to switch to switch to XML which is supported by Zebra, rather than trying to work around this problem. Since the data modeling isn't finished yet, I wrote a Perl script <tt/soif2gils.pl/ to to convert SOIF to XML, because it is much faster to test some different mappings than doing the mapping in C. <tt/soif2gils.pl/ converts SOIF files located in <em/object/ directory into GILS files into <em/objects-xml/ directory. It is intended to be used for testing how to map SOIF to XML and will become obsolete once the broker can handle XML objects. <sect>ToDo <sect1>Finalize Data Modelling The modeling of data isn't finished, but it seems that GILS offers a complete framework to hold the summarized objects. I don't see any reason to create a new framework, but I appreciate your wishes or suggestions because in this development stage, it would be easy to make changes in data model. <sect1>Switch from SOIF to XML in Broker The transition of the broker's data format from SOIF to XML will be done in single steps, so any existing components which uses SOIF will remain usable until the transition of all components are finished. To do this, following steps are necessary: <itemize> <item>Import XML from other Brokers/Gatherers <item>Import SOIF and transform to XML <item>Store XML objects als large objects in QDBM </itemize> <sect1>Switch from SOIF to XML in Gatherer Keeping backward compatibility for the gatherer during transition is also necssary and will require following steps: <itemize> <item>Transform SOIF summaries from summarizers to XML <item>Modify summarizers to produce summaries in XML <item>Modify gatherer to export XML <item>Store XML objects als large objects in QDBM </itemize> I plan to experiment with large objects of QDBM, which seems to be very promising, especially for the broker. It will make bookkeeping of the objects easier and reduce the memory requirement, because the broker doesn't have to use a registry which is kept in memory. When having the objects as large objects in the gatherer, it would also be possible to share the objects between gatherer and broker, when both are running on same machine. <sect1>Maintainance Releases of stable Version Bugfixes introduced in the development tree will be ported to the stable version of Harvest. It will also feature a Russian translation of Harvest User's Manual when Andrej finishes the translation. <sect>Sample Files <sect1>SOIF This is a sample SOIF file: <code> @FILE { http://harvest.sourceforge.net/ update-time{10}: 1065602907 full-text{718}: Harvest: A Distributed Search System Harvest: A Distributed Search System Home Sites using Harvest Download Contributed Code Todo List Links Contributors User's Manual FAQ Installation ChangeLog NEWS Harvest Harvest Homepage Miscellaneous Documents and Presentations Directory Index of Work in Progress Version of Harvest SourceForge: Project Info - Harvest Stable Version of Harvest Homepage of stable Version Historic Versions of Harvest Harvest User's Manual 1.4.pl2 (January 31, 1996) Harvest User's Manual 1.4.pl2 as PostScript Developers Kang-Jin Lee Javier Masa Marin Harald Weinreich Home Sites using Harvest Download Contributed Code Todo List Links Contributors User's Manual FAQ Installation ChangeLog NEWS headings{111}: Harvest: A Distributed Search System Harvest Stable Version of Harvest Historic Versions of Harvest Developers title{37}: Harvest: A Distributed Search System url-references{836}: harvest/doc/index.html harvest/doc/sites.html harvest/doc/download.html harvest/contrib/index.html harvest/doc/todo.html harvest/doc/links.html harvest/doc/CONTRIBUTORS harvest/doc/html/manual.html harvest/doc/html/FAQ.html harvest/INSTALL.harvest harvest/ChangeLog harvest/NEWS harvest/doc/index.html misc/ wip/ http://sourceforge.net/projects/harvest/ harvest-1.8/doc/index.html harvest-1.4.pl2-docs/ harvest-1.4.pl2-docs/user-manual.ps.gz developers/lee/ developers/masa/ http://www.weinreichs.de/ harvest/doc/index.html harvest/doc/sites.html harvest/doc/download.html harvest/contrib/index.html harvest/doc/todo.html harvest/doc/links.html harvest/doc/CONTRIBUTORS harvest/doc/html/manual.html harvest/doc/html/FAQ.html harvest/INSTALL.harvest harvest/ChangeLog harvest/NEWS http://harvest.sourceforge.net/ http://sourceforge.net/ keywords{595}: Home Sites using Harvest Download Contributed Code Todo List Links Contributors User's Manual FAQ Installation ChangeLog NEWS Harvest Homepage Miscellaneous Documents and Presentations Directory Index of Work in Progress Version of Harvest SourceForge: Project Info - Harvest Homepage of stable Version Harvest User's Manual 1.4.pl2 (January 31, 1996) Harvest User's Manual 1.4.pl2 as PostScript Kang-Jin Lee Javier Masa Marin Harald Weinreich Home Sites using Harvest Download Contributed Code Todo List Links Contributors User's Manual FAQ Installation ChangeLog NEWS md5{32}: 2ba0877c91bbc00e6db037d5604ea860 uri{31}: http://harvest.sourceforge.net/ file-size{4}: 3142 type{4}: HTML gatherer-version{6}: 1.9.10 gatherer-host{10}: dyn214.tab gatherer-name{37}: Contents of the dyn214.tab WWW server refresh-rate{6}: 604800 time-to-live{7}: 2419200 last-modification-time{10}: 1039163872 description{37}: Harvest: A Distributed Search System } </code> <sect1>XML This is the XML file converted from the sample SOIF file by <tt/soif2gils.pl/. <code> <gils> <availability> <linkage> http://harvest.sourceforge.net/ </linkage> </availability> <dateOfLastModification> 1039163872 </dateOfLastModification> <abstract> Harvest: A Distributed Search System </abstract> <author> </author> <localSubjectIndex> <localSubjectTerm> Home Sites using Harvest Download Contributed Code Todo List Links Contributors User's Manual FAQ Installation ChangeLog NEWS Harvest Homepage Miscellaneous Documents and Presentations Directory Index of Work in Progress Version of Harvest SourceForge: Project Info - Harvest Homepage of stable Version Harvest User's Manual 1.4.pl2 (January 31, 1996) Harvest User's Manual 1.4.pl2 as PostScript Kang-Jin Lee Javier Masa Marin Harald Weinreich Home Sites using Harvest Download Contributed Code Todo List Links Contributors User's Manual FAQ Installation ChangeLog NEWS </localSubjectTerm> </localSubjectIndex> <supplementalInformation> <bytes> 3142 </bytes> </supplementalInformation> <crossReference> <linkage> harvest/doc/index.html harvest/doc/sites.html harvest/doc/download.html harvest/contrib/index.html harvest/doc/todo.html harvest/doc/links.html harvest/doc/CONTRIBUTORS harvest/doc/html/manual.html harvest/doc/html/FAQ.html harvest/INSTALL.harvest harvest/ChangeLog harvest/NEWS harvest/doc/index.html misc/ wip/ http://sourceforge.net/projects/harvest/ harvest-1.8/doc/index.html harvest-1.4.pl2-docs/ harvest-1.4.pl2-docs/user-manual.ps.gz developers/lee/ developers/masa/ http://www.weinreichs.de/ harvest/doc/index.html harvest/doc/sites.html harvest/doc/download.html harvest/contrib/index.html harvest/doc/todo.html harvest/doc/links.html harvest/doc/CONTRIBUTORS harvest/doc/html/manual.html harvest/doc/html/FAQ.html harvest/INSTALL.harvest harvest/ChangeLog harvest/NEWS http://harvest.sourceforge.net/ http://sourceforge.net/ </linkage> </crossReference> <title> Harvest: A Distributed Search System </title> <Body-of-text> Harvest: A Distributed Search System Harvest: A Distributed Search System Home Sites using Harvest Download Contributed Code Todo List Links Contributors User's Manual FAQ Installation ChangeLog NEWS Harvest Harvest Homepage Miscellaneous Documents and Presentations Directory Index of Work in Progress Version of Harvest SourceForge: Project Info - Harvest Stable Version of Harvest Homepage of stable Version Historic Versions of Harvest Harvest User's Manual 1.4.pl2 (January 31, 1996) Harvest User's Manual 1.4.pl2 as PostScript Developers Kang-Jin Lee Javier Masa Marin Harald Weinreich Home Sites using Harvest Download Contributed Code Todo List Links Contributors User's Manual FAQ Installation ChangeLog NEWS </Body-of-text> </gils> </code> <sect1>Zebra Input Filter used to parse SOIF This is the <em/soif.flt/ from Zebra. This is unusable, but shows the mapping from SOIF to GILS. <code> # Crude input-filter for SOIF records -- one record per file. # Author: Peter Valkenburg / TERENA (valkenburg@terena.nl) # Version 0.2 (09/09/1998). # This sort of follows the Nordic Web Index convention of GILS attribute use. # Modified by Kang-Jin Lee (lee@arco.de) # 07/10/1999 # We'll use GILS structured records. BEGIN { begin record gils } # URL will be GILS' availability/linkage /^@[A-Za-z](-|[.A-Za-z_])* { / BODY /$/ { begin element availability data -element linkage $1 end element } # Type will be GILS' availability/linkageType /^[tT]ype{[0-9]+}:\t/ BODY /$/ { begin element availability data -element linkageType $1 end element } # Last modification time will be Bib-1 Use Attribute 1012 /^[lL]ast-[mM]odification-[tT]ime{[0-9]+}:\t/ BODY /$/ { data -element dateOfLastModification $1 } # The MD5 checksum is used as a unique identifier under Bib-1 Use Attribute 1007 /^[mM][dD]5{[0-9]+}:\t/ BODY /$/ { data -element controlIdentifier $1 } # Description will be Bib-1 Use Attribute 62 /^[dD]escription{[0-9]+}:\t/ BODY /^((-|[._A-Za-z0-9])+{[0-9]+}:\t.*|})$/ { data -element abstract $1 unread 2 } # Author will be Bib-1 Use Attribute 1003 (if gils.abs maps originator to it!!) /^[aA]uthor{[0-9]+}:\t/ BODY /^((-|[._A-Za-z0-9])+{[0-9]+}:\t.*|})$/ { data -element author $1 unread 2 } # Keywords will be GILS' localSubjectIndex/localSubjectTerm /^[kK]eywords{[0-9]+}:\t/ BODY /^((-|[._A-Za-z0-9])+{[0-9]+}:\t.*|})$/ { begin element localSubjectIndex data -element localSubjectTerm $1 unread 2 end element } # File-size will be GILS' supplementalInformation/bytes /^[fF]ile-[sS]ize{[0-9]+}:\t/ BODY /$/ { begin element supplementalInformation data -element bytes $1 unread 2 end element } # Update-Time will be GILS' supplementalInformation/lastChecked /^[uU]pdate-[tT]ime{[0-9]+}:\t/ BODY /$/ { begin element supplementalInformation data -element lastChecked $1 unread 2 end element } # url-references will be GILS' crossReference/linkage /^[uU]rl-[rR]eferences{[0-9]+}:\t/ BODY /^((-|[._A-Za-z0-9])+{[0-9]+}:\t.*|})$/ { begin element crossReference data -element linkage $1 unread 2 end element } # Title will be Bib-1 Use Attribute 4 /^[tT]itle{[0-9]+}:\t/ BODY /^((-|[._A-Za-z0-9])+{[0-9]+}:\t.*|})$/ { data -element Title $1 unread 2 } # Body and Partial-Text will be Bib-1 Use Attribute 1010 # Is Body really commonly used in SOIF? Anyway, Full-Text is used by Harvest. #/^[bB]ody{[0-9]+}:\t/ BODY /^((-|[._A-Za-z0-9])+{[0-9]+}:\t.*|})$/ { # data -element sampleText $1 # unread 2 # } /^[fF]ull-[tT]ext{[0-9]+}:\t/ BODY /^((-|[._A-Za-z0-9])+{[0-9]+}:\t.*|})$/ { data -element sampleText $1 unread 2 } /^[pP]artial-[tT]ext{[0-9]+}:\t/ BODY /^((-|[._A-Za-z0-9])+{[0-9]+}:\t.*|})$/ { data -element sampleText $1 unread 2 } /^(-|[a-zA-Z0-9])+{[0-9]+}:\t/ BODY /^((-|[_A-Za-z0-9])+{[0-9]+}:\t.*|})$/ { unread 2 } END { end record } </code> <sect1>BIB-1 Attribute Set This is BIB-1 attribute set. It is a subset of GILS. <code> # $Id: bib1.att,v 1.1 2002/10/22 12:51:09 adam Exp $ # Bib-1 Attribute Set name bib1 reference Bib-1 att 1 Personal-name att 2 Corporate-name att 3 Conference-name att 4 Title att 5 Title-series att 6 Title-uniform att 7 ISBN att 8 ISSN att 9 LC-card-number att 10 BNB-card-number att 11 BGF-number att 12 Local-number att 13 Dewey-classification att 14 UDC-classification att 15 Bliss-classification att 16 LC-call-number att 17 NLM-call-number att 18 NAL-call-number att 19 MOS-call-number att 20 Local-classification att 21 Subject-heading att 22 Subject-Rameau att 23 BDI-index-subject att 24 INSPEC-subject att 25 MESH-subject att 26 PA-subject att 27 LC-subject-heading att 28 RVM-subject-heading att 29 Local-subject-index att 30 Date att 31 Date-of-publication att 32 Date-of-acquisition att 33 Title-key att 34 Title-collective att 35 Title-parallel att 36 Title-cover att 37 Title-added-title-page att 38 Title-caption att 39 Title-running att 40 Title-spine att 41 Title-other-variant att 42 Title-former att 43 Title-abbreviated att 44 Title-expanded att 45 Subject-precis att 46 Subject-rswk att 47 Subject-subdivision att 48 Number-natl-biblio att 49 Number-legal-deposit att 50 Number-govt-pub att 51 Number-music-publisher att 52 Number-db att 53 Number-local-call att 54 Code-language att 55 Code-geographic att 56 Code-institution att 57 Name-and-title att 58 Name-geographic att 59 Place-publication att 60 CODEN att 61 Microform-generation att 62 Abstract att 63 Note att 1000 Author-title att 1001 Record-type att 1002 Name att 1003 Author att 1004 Author-name-personal att 1005 Author-name-corporate att 1006 Author-name-conference att 1007 Identifier-standard att 1008 Subject-LC-childrens att 1009 Subject-name-personal att 1010 Body-of-text att 1011 Date/time-added-to-db att 1012 Date/time-last-modified att 1013 Authority/format-id att 1014 Concept-text att 1015 Concept-reference att 1016 Any 1016,4,1005,62 att 1017 Server-choice att 1018 Publisher att 1019 Record-source att 1020 Editor att 1021 Bib-level att 1022 Geographic-class att 1023 Indexed-by att 1024 Map-scale att 1025 Music-key att 1026 Related-periodical att 1027 Report-number att 1028 Stock-number att 1030 Thematic-number att 1031 Material-type att 1032 Doc-id att 1033 Host-item att 1034 Content-type att 1035 Anywhere att 1036 Author-Title-Subject </code> <sect1>GILS Attribute Set This is the GILS attribute set, which will be used in Harvest to store the summarized objects. <code> # $Id: gils.att,v 1.1 2002/10/22 12:51:09 adam Exp $ name gils reference GILS-attset include bib1.att att 2000 Distributor att 2001 Distributor-Name att 2002 Index-Terms # Subject-Terms-Contr. att 2003 Purpose att 2004 General-Access-Constraints att 2005 Use-Constraints att 2006 Distributor-Organization att 2007 Distributor-Street-Address att 2008 Distributor-City att 2009 Distributor-State-or-Province att 2010 Distributor-Zip-or-Postal-Code att 2011 Distributor-Country att 2012 Distributor-Network-Address att 2013 Distributor-Hours-of-Service att 2014 Distributor-Telephone att 2015 Distributor-Fax att 2016 Resource-Description att 2017 Order-Information att 2018 Technical-Prerequisites att 2019 Available-Time-Structured att 2020 Available-Time-Textual att 2021 Linkage att 2022 Linkage-Type att 2023 Contact-Name att 2024 Contact-Organization att 2025 Contact-Street-Address att 2026 Contact-City att 2027 Contact-State-or-Province att 2028 Contact-Zip-or-Postal-Code att 2029 Contact-Country att 2030 Contact-Network-Address att 2031 Contact-Hours-of-Service att 2032 Contact-Telephone att 2033 Contact-Fax att 2034 Agency-Program att 2035 Sources-of-Data att 2036 Subject-Thesaurus att 2037 Methodology att 2038 West-Bounding-Coordinate att 2039 East-Bounding-Coordinate att 2040 North-Bounding-Coordinate att 2041 South-Bounding-Coordinate att 2042 Place-Keyword att 2043 Place-Keyword-Thesaurus att 2044 Time-Period-Structured att 2045 Time-Period-Textual att 2046 Cross-Reference-Title att 2047 Cross-Reference-Linkage att 2049 Original-Control-Identifier att 2050 Supplemental-Information att 2051 Record-Review-Date att 2052 Originator-Dissemination-Control att 2053 Security-Classification-Control att 2054 Cost att 2055 Cost-Information att 2056 Schedule-Number att 2057 Controlled-Subject-Index att 2058 Uncontrolled-Term att 2059 Spatial-Domain att 2060 Bounding-Coordinates att 2061 Place att 2062 Time-Period att 2063 Availability att 2064 Order-Process att 2065 Available-Time-Period att 2066 Access-Constraints att 2067 Point-of-Contact att 2068 Cross-Reference att 2069 Available-Linkage att 2070 Cross-Reference-Relationship att 2071 Language-of-Record att 2072 Beginning-Date att 2073 Ending-Date att 2074 Controlled-Term </code> <sect1>GILS File This is a sample GILS file from Zebra. <code> <gils> <Title> UTAH EARTHQUAKE EPICENTERS <Acronym> UUCCSEIS </Acronym> </Title> <Originator> UTAH GEOLOGICAL AND MINERAL SURVEY </Originator> <Local-Subject-Index> APPALACHIAN VALLEY; EARTHQUAKE; EPICENTER; SEISMOLOGY; UTAH </Local-Subject-Index> <Abstract> Five files of epicenter data arranged by date comprise this data set. These files are searchable by magnitude and longitude/latitude. Hardcopy of listing and plot of requested area available. Epicenter location and date, magnitude, and focal depth available. <Format> DIGITAL DATA SETS </Format> <Data-Category> TERRESTRIAL </Data-Category> <Comments> Data are supplied by the University of Utah Seismograph Station. The Utah Geologcial and Mineral Survey (UGMS) is merely a clearinghouse of the data. </Comments> </Abstract> <Spatial-Domain> <Geographic-Coverage> US STATE </Geographic-Coverage> <Coverage-Description> UTAH </Coverage-Description> <Bounding-Coordinates> <West-Bounding-Coordinate> -114 </West-Bounding-Coordinate> <East-Bounding-Coordinate> -109 </East-Bounding-Coordinate> <North-Bounding-Coordinate> 42 </North-Bounding-Coordinate> <South-Bounding-Coordinate> 37 </South-Bounding-Coordinate> </Bounding-Coordinates> </Spatial-Domain> <Time-Period> <Time-Period-Textual> -PRESENT </Time-Period-Textual> </Time-Period> <Availability> <Distributor> <Organization> UTAH GEOLOGICAL AND MINERAL SURVEY </Organization> <Street-Address> 606 BLACK HAWK WAY </Street-Address> <City> SALT LAKE CITY </City> <State> UT </State> <Zip-Code> 84108 </Zip-Code> <Country> USA </Country> <Telephone> (801) 581-6831 </Telephone> </Distributor> <Resource-Description> UTAH EARTHQUAKE EPICENTERS </Resource-Description> <Technical-Prerequisites> <Data-Set-Type> AUTOMATED </Data-Set-Type> <Access-Method> BATCH </Access-Method> <Number-of-Records> 8,700 </Number-of-Records> <Computer-Type> PC NETWORK </Computer-Type> <Computer-Location> SALT LAKE CITY, UT </Computer-Location> </Technical-Prerequisites> </Availability> <Access-Constraints> <Documentation> NONE </Documentation> </Access-Constraints> <Use-Constraints> <Status> OPERATIONAL </Status> </Use-Constraints> <Point-of-Contact> <Name> BILL CASE </Name> <Organization> UTAH GEOLOGICAL AND MINERAL SURVEY </Organization> <Street-Address> 606 BLACK HAWK WAY </Street-Address> <City> SALT LAKE CITY </City> <State> UT </State> <Zip-Code> 84108 </Zip-Code> <Country> USA </Country> <Telephone> (801) 581-6831 </Telephone> </Point-of-Contact> <Control-Identifier> ESDD0006 </Control-Identifier> <Record-Source> UTAH GEOLOGICAL AND MINERAL SURVEY </Record-Source> <Date-of-Last-Modification> 198903 </Date-of-Last-Modification> </gils> </code> </article>