Description

Most research in speeding up text mining involves algorithmic improvements to induction algorithms, and
yet for many large scale applications, such as classifying or indexing large document repositories, the time
spent extracting word features from texts can itself greatly exceed the initial training time. This paper
describes a fast method for text feature extraction that folds together Unicode conversion, forced
lowercasing, word boundary detection, and string hash computation. We show empirically that our integer
hash features result in classifiers with equivalent statistical performance to those built using string word
features, but require far less computation and less memory.