Class UnigramSegmenter

java.lang.Object
opennlp.tools.tokenize.lattice.UnigramSegmenter
All Implemented Interfaces:
Tokenizer
public final class UnigramSegmenter extends Object implements Tokenizer
Frequency-driven segmentation for Chinese and similar scripts: a Viterbi search that maximizes the summed log-probability of the words in a user-supplied frequency lexicon, with unlisted characters falling back to single-character words. This is the unigram model behind common Chinese segmenters. It omits the connection costs used by LatticeTokenizer, making it suitable for lexicons containing words and counts.

The lexicon format is one entry per line: the word, its count, and optionally a tag, separated by whitespace. The lexicon file is user-supplied; no lexicon data is bundled. Every reported span is in original text coordinates.

Instances are immutable and safe to share between threads.

Since:
3.0.0

  • Method Details

    • load

      public static UnigramSegmenter load(Path lexicon) throws IOException
      Loads a frequency lexicon encoded in UTF-8.
      Parameters:
      lexicon - The lexicon file. Must not be null.
      Returns:
      The segmenter. Not null.
      Throws:
      IOException - Thrown if reading fails or the lexicon is empty or malformed.
      IllegalArgumentException - Thrown if lexicon is null.

    • load

      public static UnigramSegmenter load(Path lexicon, Charset charset) throws IOException
      Loads a frequency lexicon.
      Parameters:
      lexicon - The lexicon file: one word, its count, and an optional tag per line. Must not be null.
      charset - The lexicon encoding. Must not be null.
      Returns:
      The segmenter. Not null.
      Throws:
      IOException - Thrown if reading fails or the lexicon is empty or malformed.
      IllegalArgumentException - Thrown if a parameter is null.

    • load

      public static UnigramSegmenter load(InputStream lexiconStream, Charset charset) throws IOException
      Loads a frequency lexicon from a stream.
      Parameters:
      lexiconStream - The lexicon content. Must not be null. Not closed.
      charset - The lexicon encoding. Must not be null.
      Returns:
      The segmenter. Not null.
      Throws:
      IOException - Thrown if reading fails or the lexicon is empty or malformed.
      IllegalArgumentException - Thrown if a parameter is null.

    • tokenize

      public String[] tokenize(String text)
      Splits a string into its atomic parts.

      Reports the segmented surfaces, whitespace omitted.

      Specified by:
      tokenize in interface Tokenizer
      Parameters:
      text - The string to be tokenized.
      Returns:
      The String[] with the individual tokens as the array elements.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • tokenizePos

      public Span[] tokenizePos(String text)
      Finds the boundaries of atomic parts in a string.

      Reports the segmented spans in original text coordinates, whitespace omitted.

      Specified by:
      tokenizePos in interface Tokenizer
      Parameters:
      text - The string to be tokenized.
      Returns:
      The spans (offsets into s) for each token as the individuals array elements.
      Throws:
      IllegalArgumentException - Thrown if text is null.