WhitespaceTokenizer (Apache OpenNLP Tools 2.1.1 API)

java.lang.Object
- opennlp.tools.tokenize.WhitespaceTokenizer

All Implemented Interfaces:

Tokenizer
```
public class WhitespaceTokenizer
extends Object
```
A basic Tokenizer implementation which performs tokenization using white spaces.
To obtain an instance of this tokenizer use the static final INSTANCE field.

Field Summary

Fields
Modifier and Type Field Description

static WhitespaceTokenizer INSTANCE
Use this static reference to retrieve an instance of the WhitespaceTokenizer.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`void`	`setKeepNewLines(boolean keepNewLines)`	Switches whether to keep new lines or not.
`String[]`	`tokenize(String s)`	Splits a string into its atomic parts.
`Span[]`	`tokenizePos(String d)`	Finds the boundaries of atomic parts in a string.

Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - INSTANCE
```
public static final WhitespaceTokenizer INSTANCE
```
    Use this static reference to retrieve an instance of the WhitespaceTokenizer.
- Method Detail
  - tokenizePos
```
public Span[] tokenizePos(String d)
```
    Description copied from interface: Tokenizer
    
    Finds the boundaries of atomic parts in a string.
    
    Parameters:
    
    d - The string to be tokenized.
    
    Returns:
    
    The spans (offsets into {@code s}) for each token as the individuals array elements.
  - tokenize
```
public String[] tokenize(String s)
```
    Description copied from interface: Tokenizer
    
    Splits a string into its atomic parts.
    
    Specified by:
    
    tokenize in interface Tokenizer
    
    Parameters:
    
    s - The string to be tokenized.
    
    Returns:
    
    The String[] with the individual tokens as the array elements.
  - setKeepNewLines
```
public void setKeepNewLines(boolean keepNewLines)
```
    Switches whether to keep new lines or not.
    
    Parameters:
    
    keepNewLines - True if new lines are kept, false otherwise.