Welcome to ICU-Tokenizer’s documentation!

ICU-tokenizer is a python package used to perform universal language normalization and tokenization using the International Components for Unicode.

Library Reference

• Commandline Tools
  • Sentence Splitting
  • Normalize
  • Tokenize
• Python API Reference
  • Sentence Splitter
  • Normalizer
  • Tokenizer