Uses of Interface
opennlp.tools.tokenize.Tokenizer

Packages that use Tokenizer

Package	Description
opennlp.tools.tokenize
opennlp.tools.tokenize.lattice

Uses of Tokenizer in opennlp.tools.tokenize

Classes in opennlp.tools.tokenize that implement Tokenizer

Modifier and Type	Class	Description
class	WhitespaceTokenizer	A basic Tokenizer implementation which performs tokenization using white spaces.
class	WordpieceTokenizer	A Tokenizer implementation which performs tokenization using word pieces.

Uses of Tokenizer in opennlp.tools.tokenize.lattice

Classes in opennlp.tools.tokenize.lattice that implement Tokenizer

Modifier and Type	Class	Description
final class	LatticeTokenizer	Dictionary-driven segmentation for languages written without spaces: a Viterbi search over the word lattice of a MecabDictionary, minimizing the sum of word costs and connection costs.
final class	UnigramSegmenter	Frequency-driven segmentation for Chinese and similar scripts: a Viterbi search that maximizes the summed log-probability of the words in a user-supplied frequency lexicon, with unlisted characters falling back to single-character words.