Natural Language Processing Tools | Parsing | Information Retrieval

Natural Language Processin

Freeware Tools

ALE (Attribute Logic Engine) o Description: ALE is a environment that integrates phrase structure parsing and constraint logic programming with typed feature structures. It can handle several formalisms including HPSG, PATR-II, DCG grammars, and Prolog, Prolog-II, and LOGIN programs. Sample grammars are provided with the distribution. o Platforms: Platforms with SICStus Prolog, or Quintus Prolog. o Source: The latest version is available from the CMU Artificial Intelligence Repository. o Reference: Additional information is available from the CMU Artificial Intelligence Repository. o Contact: CGPARSER o Description: CGParser is a linear parser of Conceptual Graphs. It was written using the YACC compiler generator utility. The distribution includes examples of various levels of complexity for testing purposes. o Platforms: UNIX. o Source: The latest version is available from the Consortium for Lexical Research. o Reference: Additional information is available from the Consortium for Lexical Research. o Contact: CHARON o Description: CHARON is an environment for the development and testing of LFG grammars. It integrates parsers, semantic components and the generator, and provides a user-interface for the compilation and the testing of LFG grammars. o Platforms: UNIX. o Source: The latest version is available from

o 

Reference: Additional information is available from

Conc Description: Conc is used for producing concordances of texts. It also produces a frequency index for each word in the text. It displays the original text, the concordance, and the index each in synchronized windows. o Platforms: Mac. o Source: The latest version is available from the Consortium for Lexical Research. o Reference: Additional information is available from the Consortium for Lexical Research. o Contact: ELIZA o Description: This is the classic NLP program by Weizenbaum. It allows for a simple first assignment in NLP. Students are asked to develop a new knowledge base for some domain other that the classic psychoanalyst-patient one. o Platforms: PC, Mac, VAX, UNIX and others. o Source: The latest version is available from the CMU Artificial Intelligence Repository. o Reference: N/A. o Contact: N/A. ENGLEX o Description: Englex is a lexicon for morphological analysis of English text. It is intended for use with PC-KIMMO (or programs that use the PC-KIMMO parser, such as KTEXT). Combined with software, it facilitates production of sets of records of the morphological constituents in English texts. o Platforms: PC, Mac, and UNIX. o Source: The latest version is available from the Consortium for Lexical Research. o Reference: Additional information is available from the Consortium for Lexical Research. o Contact: FLEX (Fast Lexical Analyzer Generator) o Description: FLEX is a generator of lexical pattern recognizers. It is an extension to the UNIX LEX lexical analyzer utility. o Platforms: UNIX. o Source: The latest version is available from the
o o Source: The latest version is available from the Consortium for Lexical Research.    Reference: Additional information is available from the Consortium for Lexical Research. o Platforms: PC. o Reference: Additional information is available from the Consortium for Lexical Research. and Sun. o Platforms: UNIX. o Contact: agfl@cs. Grammar Workbench o Description: The Grammar Workbench is an environment for the development and analysis of grammars. LINK o Description: LINK is a parser for Link Grammar. o Contact: evan@txsil. It is intended for both phonology students and researchers in that it facilitates understanding of phonological rule fundamentals and helps manage large complex bodies of phonological It also includes a Link Grammar for English.tcu. o Contact: vern@ee. It is an auxiliary program for PC-KIMMO.kun.kun. It also incorporates input and output filters/conditions. o .nl. a context-free formalism for the description of natural language. KGEN o Description: KGEN is a program for building morphological parsers for NLP o Reference: Additional information is available from the Consortium for Lexical Research.cs. It is geared towards the AGFL (Affix Grammars over a Finite Lattice) formalism. o Source: The latest version is available from hades. FONOL o Description: Fonol is a programming language for experimenting with Transformation-Grammar-style phonological rules. o Platforms: PC (and platforms with Turbo Pascal).gov. o Platforms: PC.kun. o Contact: brandon@gamma.lonestar. o Reference: Additional information is available from hades.lbl. o Source: The latest version is available from the Consortium for Lexical Research.

html o Reference: http://www. allowing the inclusion of interactive diagrams within Web o   Lotec Description: The Lotec Speech Recognition Package is a simple set of libraries and tools for building single-speaker. and a set of Pearl scripts for automating the use of the above tools. o Contact: tools@cse. a Neural Network training package. o Contact: a set of sound-file format conversion utilities.ltg. o Source: N/ OGI Speech Tools o Description: The OGI Speech Tools are a set of speech data manipulation tools including an X Windows display tool (Lyre) for displaying data in a time synchronous fashion.cmu.ltg. we envisage widespread application within other areas involving the presentation and interpretation of highly structured It is available free of charge for non-commercial purposes. Contact: sleator@cs. o Platforms: Java o Source: http://www. o Platforms: UNIX. o Source: The latest version is available from the Originally designed for use with linguistic diagrams. low-quality continuous speech recognition a set of C library routines (LIBNSPEECH) for speech data manipulation. o Reference: Additional information is available from The Natural Language Software Registry. LT Thistle o Description: LT Thistle is a parameterizable display engine and editor for diagrams. Reference: Additional information is available from the Consortium for Lexical Research. o Reference: N/A.html o Contact: Jo Calder o o  Source: The latest version is available from the Consortium for Lexical Research. o Platforms: Sun.

It employs a bottom-up and breadth-first parsing algorithm. o Source: The latest version is available from the Consortium for Lexical Research.lonestar. and a surface-level form. Mac. o Platforms: PC. o Platforms: Platforms with SICStus Prolog. o Platforms: UNIX. o Reference: Additional information is available from the Consortium for Lexical Research. published by the Summer Institute of Linguistics (1990). o Source: The latest version is available from the Consortium for Lexical Research. o Contact: evan@txsil.e. o Contact: N/A. Antworth. SYNTACTICA o Description: SYNTACTICA is a system for grammar development with a simple graphical user interface. Commercial Tools  Alvey Natural Language Tools (ANLT) o Description: The Alvey Natural Language Tools is a set of tools for use in natural language processing .sunysb. o Contact: rlarson@semlab1. or introductory linguistics classes with a syntax component. a lexical-level   PC-KIMMO o Description: PC-KIMMO is a popular program among computational linguists. and PCKIMMO: A Two-Level Processor for Morphological Analysis by Evan L. and NLP system developers. It generates and/or recognizes words using a two-level model of word structure. SAX (Sequential Analyzer for syntaX and semantics) o Description: SAX is a syntactic analyzer for Definite Clause Grammar. o Source: The latest version is available from the Consortium for Lexical Research. It is intended for use in introductory syntax o Reference: Additional information is available from the Consortium for Lexical Research.. o Reference: Additional information is available from the Consortium for Lexical Research. Distribution includes a Japanese grammar and some sample Japanese data. i. descriptive linguists.

cam. University of Western o  . It is independent of formalism. parameter extraction/tracking and tools to automate measurement and support data logging. Contact: N/A.2. a grammar and a lexicon. ALEP Description: The Advanced Language Engineering Platform (ALEP) is a versatile and flexible general purpose NLP platform. an extensive on-line documentation. These include a morphological analyzer. a spectral a formant tracker.19. Functions include speech capture. Simpkins. ALEP low-cost facility using mass-produced and widely-available hardware. and an experiment generator/controller. editing. Platforms: UNIX. syntactic and semantic analysis of a considerable subset of English. Jamieson. They can be used independently or with a grammar development environment to form a complete system for the morphological. o Platforms: Platforms supporting Prolog by BIM 4. L1840 LUXEMBOURG.Canadian Speech Research Environment o Description: CSRE is designed to support speech research by providing a powerful. incorporates a number of standards such as SGML. and various tools for text handling. GNU Emacs 19. o Source: Available on tape through contact below.0.0. linguistic processing. spectral analysis procedures. o Platforms: PC. a time-domain analyzer. a pitch tracker. o Contact: Mr. OSF/MOTIF 1. 11b Bvd Joseph II. o Source: N/A. N.5. and MOTIF and comes with a graphical user interface. a speech synthesizer. K. 3D displays. o Reference: Additional information is available from The Natural Language Software Registry. LUXEMBOURG. o Reference: Additional information is available from The Natural Language Software Registry. ClauseDB 2. o Contact: Donald G. Reference: Additional information is available from ftp.o o o o  research. parsers. ISO character sets. Cray Systems. an acoustic signal synthesizer. . CSRE -. and debugging. and replay. Source: N/ CSRE components include a speech editor.

D. filter design. Hearing Health Care Research Unit. or DEC 3100/5000 and ALPHA computer running UNIX. It employs a parser. o Contact: Cilla DeVries. and filtering. Canada. and a debugger. Marketing Department. file manipulation. NL Builder (TM) o Description: NL Builder (TM) may be used to develop NLP applications or experiment with various linguistic components. Sun. It consists of a tokenizer. NeXT. plotting. USA 20003. and UNIX. S. Ave. natural language generator. data I/O and conversion.E. o Platforms: PC. o Source: N/A. Their functionality includes spectrum analysis. o Reference: Additional information is available from . a dictionary. o Source: N/A. Entropic Research Laboratory. o Source: N/A. speech processing. SGI. lexical acquisition tools. a morphological analyzer. Intelligent Connector (ICon). and others. London..Entropic Signal Processing System o Description: ESPS is a set of signal and speech processing utilities. Apollo. a parser. time series manipulation. Director of Sales and Marketing. Its extension mechanism. and a deductive system that interprets English questions in the context of the specific applications. U. VAX. 600 Penn. Suite 202. a semantic network KRL. Alameda..A. Natural Language Inc. a semantic interpreter. o Reference: Additional information is available from The Natural Language Software Registry. Ontario N6G1H1. Mac.. o Platforms: MS-Windows. o Reference: Additional information is available from The Natural Language Software Registry.C. Washington.S. Natural Language (TM) o Description: Natural Language (TM) is an extensible natural language interface to relational SQL databases. HP 9000/700. semantic interface. 1125 Atlantic Avenue. may be used to customize Natural Language to specific applications. Communicative Disorders. pattern classification. ESPS . "C" hooks. o Platforms: SUN. VMS. o Contact: Ken Nelson.   Ontario. CA 94501.

 The Natural Language Software Registry.mchenry@textanalysis. o Source: N/A. and the Conceptual Grammar (TM) hierarchical knowledge base management system (KBMS). o Contact: Edwin R. Features a rapid prototyping GUI. 877-235-6259 USA toll free. analyzers run on Windows PC and Linux. Synchronetics.. U. Synchronetics. o Reference: http://www. and additional paradigms. summarization. Baltimore. o Contact: Maureen McHenry.. Inc. Applications include information extraction..S. It constructs multipass text analyzers that combine pattern-based. categorization.A.textanalysis. Front St. 301 N. Inc. ODBC database connectivity. the NLP++ (R) programming language (interpreted/compiled). MD 21202. o Platforms: VisualText runs on Windows PC. Addison. Comes with the TAIParse general analyzer and others. . Inc. grammarbased. Text Analysis International. Open architecture integrated with C++. natural language generation (NLG). maureen. VisualText (R) o Description: VisualText is an integrated development environment (IDE) for


Parsing Resources Taggers online.a search engine for programming resources.Statistical Part-of-Speech Tagging . University of Zurich) The LINGUIST List: Software The Natural Language Software Registry Language Software Helpdesk o Frequently Asked Questions PennTools . Natural Language Processing Research Group: RESOURCES ICOT Free Software Netlib Repository (mirror in Japan) General Information     Sourcebank .Software Tools for NLP Software Archive        CMU Artificial Intelligence Repository Resources Available Through CRL SIL Computing Resources Linguistics Tools at the University of Vaasa in Finland Leeds University.Software Some publically available NLP packages SAL (Scientific Applications on Linux) Artificial Intelligence             Public Domain Generic Tools: An Overview . Morphological Analyzer      A Perl/Tk text tagger Conexor Cogilex R&D inc . Resources related to content analysis and text analysis .Makers of expert tools for natural language processing CLAWS part-of-speech tagger TnT . email message containing addresses Parsers and Taggers Information (by Steven Paul Abney) Relator Language Processing Resources Corpus Search Tools Neural Networks & Statistics: Software Tagger.Computational Linguistics Resources At Penn.a paper written by Tomaz Erjavec A collection of online interactive CL tools (Computational Linguistics Group.

CG Parser .Natural deduction categorial grammar and lambda-calculus parser.a system for automatic language analysis Attribute-Logic Engine (ALE) System and Grammars . language teachers. Eric Brill's Part of Speech Tagger Software Plaza: Brill's Tagger Morphy . at Kyoto University.                     POS tagger for Spanish Tagging and Parsing tools AUTASYS .An integrated tool for German morphology and statistical part-of-speech tagging.Lexical platform for the Spanish language. o Mike Scott's Home Page o Oxford University Press A Lexical Analyzer for HTML and Basic SGML ARIES Natural Language Tools . Japan. It is intended for linguists.Frank Smadja's Collocation Extractor. and anyone who needs to examine language. Stemmer      Porter stemmer Porter stemmer Dutch Porter stemmer IRIS stemmer Iterated Lovins stemmer Collocation  Xtract .A freeware logic programming and grammar parsing integrated suite of programs for looking at how words behave in texts.A Fully Automatic English Wordclass Analysis System TOSCA/LOB tagger Relaxation Labelling Based Multi-Tagger The QTAG Part of Speech Tagger QTAG: A portable Parts of Speech Tagger The Alvey Natural Language Tools The XTAG Project TreeTagger . WordSmith Tools .Wordsmith Tools is the Swiss Army knife of lexical analysis .An adaptation of Brill's tagger to Windows 95/98.a language independent part-of-speech tagger Xerox Part-of-Speech Tagger The Edinburgh/Cambridge Morphological Analyser System Winbrill . Parser     Malaga .Japanese morphological analyzer (JUMAN) and parser (KNP) developed by Nagao Lab. Head-Corner Parser (by Gertjan van Noord) . Korean Morphological Analyzer Natural Language Tools .

The Apple Pie Parser is a bottom-up probabilistic chart parser which finds the parse tree with the best score by best-first search algorithm. along with the Alembic system to enable the automatic acquisition of domain-specific tagging heuristics.An Environment for Managing Corpus and Multilingual Web Server The IMS Corpus Toolbox Webpage X Kobe Phoenix Laboratory .An RST (Rhetorical Structure Theory) Markup Tool. MonoConc (concordance program) MonoConc for Windows (concordance program) Text Analysis Computing Tools (TACT) The Lingua Project: The World of MultiLingual Parallel Concordancing (http://prune. Concordance . The Lingua Project: The World of MultiLingual Parallel Concordancing (http://www.Left-head-corner Island Parser Textual Corpora and Tools for their Exploration Language Modeling   Maximum Entropy Modeling Maximum Entropy Modeling Toolkit . concordances.Corpus Wizard program.0 and Windows 95/98 which makes wordlists.Comprehensive Unification Formalism Apple Pie Parser . Multext: Multilingual Text Tools and Corpora XCorpus . Second Edition Cass Partial Parser CHILL: An empirical parser acquisition system using inductive logic programming ISSCO Tools .This page describes tools and formats for creating and managing linguistic annotations.loria. The System Quirk . RST Annotation Tool Qwick .         A basic parser written to illustrate the bottom up parsing algorithms in Natural Language Understanding.a suite of tools for the analysis of a corpus.loria.Workbench for Terminology. . Georgetown University Natural Language Processing Parser Modularity Demo page PC-PATR: A syntactic parser IMS Stuttgart: The CUF Web Page .Sentences alignment tool in multilingual corpora.A program for Windows NT 4. Link Grammar Parser Corpus Tools                     WebCorp Concordances: Producing and Using them XCES: Corpus Encoding Standard for XML RST Tool . and Web Concordances from your electronic texts. Lexicography and Text Analysis.corpus browser Linguistic Annotation . Alembic Workbench .

Monoids. and other formal language theory objects.a symbolic computation environment for finite-state machines. Speech      HTK: Hidden Markov Model Toolkit CSLU Toolkit The Epos Speech Synthesis System ISIP public domain speech to text system o The ISIP Automatic Speech Recognition Toolkit CSLU Toolkit (Center for Spoken Language Understanding. Oregon Graduate Institute of Science and Technology) . Dunning's program Gertjan van Noord's program Doug Beeferman's program FSA Tools       Finite State Utilities Automata Learning from Theory to Practice o Downloadable Software Index to finite-state machine software. and projects FSA utilities o FSA Utilities: A Toolbox to Manipulate Finite-state Automata Grail .A program for the computation of Automata.       CMU-Cambridge Statistical Language Modeling Toolkit CMU Statistical Language Modeling Toolkit by Roni Rosenfeld o Program o Document Trigger Toolkit Simple Good-Turing Smoothing Smoothing tools software by Joshua Goodman and Stanley Chen Language modeling tools Statistical Decision Trees HMM      A HMM mini-toolkit (by Anand Venkataraman) HMM Software see also: Exercise: Using a Hidden Markov Model Discrete HMM Toolkit Hidden Markov Model (HMM) Toolbox Meta-MEME: Motif-based Hidden Markov Models of Biological Sequences Language Identification    Ted E. products. AMoRE . and Regular Expressions. regular expressions.

A statistics package for analysis of associations between discrete variables. IR-STAT-PAK .A program to compute descriptive and analytic statistics for the TREC IR trials.SMART Modifications Bow: A Toolkit for Statistical Language Modeling.A visual interface to textual information. Classification and Clustering ifile . Text Retrieval.a Search Engine For Text MG .software for indexing and searching text documents.A general mail filtering system.     Computer generation of accent marks Spoken Natural Language Processing Group Software CMU Error Analysis Toolkit Audio Tools VOICEBOX: Speech Processing Toolbox for MATLAB Mathematical Software  NIST Guide to Available Mathematical Software Statistics   Bayesian inference Using Gibbs Sampling CoCo . Machine Learning     Machine Learning Toolbox (MLT) The Machine Learning Programs Repository The RIPPER rule learner mFOIL . Support Vector Machine    SVMLight SVM package by William Noble Grundy Kernel Machines Web Site Information Retrieval & Filtering            Rubryx: Text Classification Program seft .An ILP systems designed to handle noisy examples. SMART Software and test collections (Cornell University) o see also SMART links Doug Oard's Research Software Page . Labeled data sets for information extraction .Managing Gigabytes Isearch . Yavi .

HTTP copying and mirroring tool.for quantitative analyses of texts. HTTrack . recall.The Multext character translation program Evalb .A tool for fuzzy cluster analysis LNKnet Pattern Classification Software Principal Direction Divisive Partitioning k-means clustering WWW    w3mir .String/Pattern Matching   Online Approximate String Matching Strmat package (exact string matching and suffix trees) Sentence Boundary Detector   SATZ: An Adaptive Sentence Boundary Detector Adwait Ratnaparkhi's MXTERMINATOR Clustering/Classification     FCLUSTER . non crossing and tagging accuracy for given data. TiMBL: Tilburg Memory Based Learner MtRecode . The OC1 decision tree software system IND Version 2. GATE . SNoW learning program The µ-TBL Homepage . It reports precision. transcripts.a graphical user interface for annotating the discourse structure of spoken dialogue.The Web mirror utility.Simples Text Analyse Tool WebCONC German Morphology Browser (online service) 'mat2D' Matrix/Vector Library in C Content Analysis Resources . and images.General Architecture for Text Engeneering. HTML Conversion. Shareware and Freeware Other Tools                   TextSTAT . and text.0 .Computer Assisted Qualitative Data Analysis Software Suffix sort Nb .Logic Programming Tools for Transformation-Based Learning ROOT: An Object-Oriented Data Analysis Framework CAQDAS Networking Project .A bracket scoring program.creation and manipulation of decision trees from data Paai's text utilities . monologue.

Language Technology Group.a pattern discovery tool XGobi . like Where is London? Software Announcements Tools for drawing and graphically editing trees Paul Nation's vocabulary programs syllable prediction code (a simple lisp function) Pratt . 150 miscellaneous text processing tools. NODElib . and retrieves it in response to human requests. XTAG morpholyzer post-processors for English Stemming.DN2 is an intelligent self-relating free format database system which accepts data in human text format. TOOLDIAG: Pattern recognition toolbox The DN2 Home Page .              Shoebox 3.Neural Optimization Development Engine library . Human Communication Research Centre. 75 text statistics and bitext geometry tools. University of Edinburgh Introducing environmentalism and post-fordism into NLP (NeuroTran) Tools for Estonian Language Dan Melamed's Page . Good-Turing Smoothing Software.A system for multivariate data visualization.0 for Windows and Macintosh . Teaching materials for statistical NLP by Chris Brew.A database program oriented to the needs of a field linguist's dictionary.Simulated Annealing Program.

0beta trec_eval.3.2006/18/09 Please. Zettair is a (small) set of software written in language C for text indexing and retrieval. The straightforward programming style makes easy to add other features (in indexing for instance). 3 IR tools tested or in use within the RIM team Zettair team site tool site Previously known as lucy.Some information retrieval tools Michel Beigbeder -.7.0.8. It is easy to add its own weighting scheme too. Links:   Notes on Trec Eval.gz trec_eval.v3beta trec2_eval trec_eval_hp trec1_eval The software for doing IR system evaluation. Small guide on how to use trec_eval with mg. .1.gz trec_eval.tar. indexing and retrieval system for text.8.7.gz trec_eval. Evaluation trec_eval trec_eval trec_eval.tar. mg tool site book site MG is an open-source compressing. let me know if you find any error in the following information. Comment: The index format is very easy to understand.tar.

. it is very difficult to create new code to directly access to the index (again this is due to the complex compression mechanisms in use). Comment: Not easy to install.64x from UTAH.images.2x mg-1.3c mg-1. Links:   Version 1.3x mg-1. The configuration process is error prone. and textual images. Development discontinued since 1992.1x mg-1. they seem derived from the above version. Internal principles by Jian-Yun Nie (in french) Internal implementation (data structures. It is possible to experiment with different weighting schemes.) by Jian-Yun Nie (in french) . both in the indexing and the querying phases. but anyway it is a very good book on both compression and information retrieval.3. It is written in language C.3g in use at the New Zealand Digital Library. here are some links on how to use it. Versions mg-1.. . It is written in language C. Extensions that fit well in the vector model are not too difficult but it is quite impossible to add other ones. Development discontinued since August 1999 Comment: The book does not help too much to understand the software. Some (badly) documented features actually don't work. The software is more difficult to extend than Zettair because there is heavy use of (complex) macros to tackle with the compression features.3. But we succeeded in some extensions by inserting our own code in some key points. smart smart tool site Smart implements the basic vector model of information retrieval. The configuration mechanism is difficult to understand. However.3.      Tutorial for beginners by Hans Paijmans A smart tutorial by Tassos Tombros (Glasgow) Another documentation set for smart by Christian Meunier and Ghita Bouayad. Links: Because this software is difficult to use and its internal documentation is not good.

23. maintain and display Yahoo! like directories. DataparkSearch Engine tool site (C) (Most recently modified file: 2005-12-01 in V4.13 2001-07-16) webbase is a crawler for the Internet. intranet or local system. (Last release 0. (Last version 5. (Last version 1. It has two main functions : crawl the WEB to get documents and build a full text database with this documents.17.0 2001-07-23) unac is a C library and command that removes accents from a string. (Last version 1.35) DataparkSearch Engine is a full-featured open sources web-based search engine released under the GNU General Public License and designed to organize search within a website. full-featured text search engine library written entirely in Java. Lemur tool site (C++) The Lemur Toolkit for Language Modeling and Information Retrieval. Senga tool site (Mainly C++) Senga is a development group focused on information retrieval software.4. (Last version 2.7 softwares not tested Cheshire tool site (Mainly C) (Most recently modified file: 2005-01-13 in V2.      Catalog is a perl program that allows to create.0 2002-09-02) uri is a library that analyses URIs and transform them. group of websites.0 2001-09-10) .7.41) A Next-Generation Online Catalog and Full-Text Information Retrieval System.3) Lucene is a high-performance. Lucene tool site (Java) (Most recently modified file: 2004-11-29 in V1.03 2001-07-11) The purpose of GNU mifluz is to provide a C++ library to build and query a full text inverted index.

released under the GPL. providing indexing and retrieval functionalities. as well as front-ends for document classification (rainbow). Finnish. Wumpus tool site (C++) (Most recently modified file: 2005-11-30 in V2005-11-30) Wumpus is an information retrieval system. It's written in C++.0. The current distribution includes the library. More generally. . it is a modular platform for the rapid development of large-scale Information Retrieval applications. Russian. German. Its main purpose is to study issues that arise in the context of indexing dynamic text collections in multi-user environments.Development discontinued since 2002. intranet and desktop search engines. French.0. language modeling and information retrieval programs. Spanish. and Swedish). English.2) Xapian is an Open Source Probabilistic Information Retrieval library. Structured boolean search operators. Phrase and proximity searching. Relevance feedback. document retrieval (arrow) and document clustering (crossbow). 1 library not tested bow Bow library site Bow (or libbow) is a library of C code useful for writing statistical text analysis.9. Xapian tool site (C++) (Most recently modified file: 2005-07-15 in V 0.2) Terrier is a software for the rapid development of Web. Dutch. Stemming (Danish.2) (Most recently modified file: 2005-03-17 in V1. Features: Ranked probablistic search. Norwegian. Portuguese. Terrier tool site (Java) (Last version 1. Italian.

Performing test/train splits. The library and its front-ends were designed and written by Andrew McCallum. The name of the library rhymes with `low'. Do smoothing across N-gram models. Including N-grams among the tokens. according to several different methods. and several other methods. Claim to be finished. and automatic classification tests. sparse fashion. Classification and Clustering Bow (or libbow) is a library of C code useful for writing statistical text analysis. receiving and answering queries over a socket. Mapping strings to integers and back again. finding text files. language modeling and information retrieval programs. TFIDF. . Scoring queries for retrieval or classification. Pruning vocabulary by word counts or by information gain. The library does not:    Have English parsing or part-of-speech tagging facilities. Operating in server mode. About the library The library provides facilities for:                Recursively descending directories. Building a sparse matrix of document/token counts. as well as front-ends for document classification (rainbow). very efficiently. with some contributions from several graduate and undergraduate students. Text Retrieval. Setting word vector weights according to Naive Bayes.BOW Bow: A Toolkit for Statistical Language Modeling. Smoothing word probabilities according to Laplace (Dirichlet uniform). M-estimates. Building and manipulating word vectors. and Good-Turning. Reading the document/token matrix from disk in an efficient. not `cow'. document retrieval (arrow) and document clustering (crossbow). Finding `document' boundaries when there are multiple documents per file. The current distribution includes the library. Writing all data structures to disk in a compact format. Tokenizing a text file. WittenBell.

Most appreciated are bug reports accompanied by fixes." http://www.cs. Patches to the code are most". SUNOS. year = 1996} Obtaining the Source Source code for the library can be downloaded from this directory. "Bow: A toolkit for statistical language modeling. Over a year ago. Solaris. text retrieval. It is released under the Library GNU Public License (LGPL). . author = "Andrew Kachites McCallum". Irix and HPUX. Thus.cmu. it also provides TFIDF/Rocchio. there are currently three executable programs based on the library. classification and clustering". Andrew Kachites. The code conforms to the GNU coding standards. Feel free to send me mail asking for help. Different versions are indicated by eight digit sequences that indicate year.cs. Unfortunately I do not have time to help rainbow's many users with all their compilation and usage problems. text retrieval. however please acknowledge its use with a citation: McCallum. title = "Bow: A toolkit for statistical language modeling. but probably could with small note = "http://www. it doesn't do this any more. but please do not necessarily expect me to have time to help. it compiled on WindowsNT (with a GNU build environment). Citation You are welcome to use the code under the terms of the licence for research or commercial purposes. the most recent version is the one with the largest version number. Claim to be bug-free. 1996. including Linux. Bow Library Front-Ends Provided in the library source distribution.  Have good documentation. Probabilistic Indexing and K-nearest neighbor. Here is a BiBTeX entry: @unpublished{McCallumLibbow. While mostly designed for classification by naive Bayes.  Rainbow is an executable program that does document classification. It is developed on a Linux system. month and day.cmu. It is known to compile on most UNIX systems. classification and clustering.

  Arrow is an executable program that does document retrieval. . It currently only performs simple TFIDF-based retrieval. Crossbow is a an executable program that does document clustering (and also classification).

Sign up to vote on this title
UsefulNot useful