# Chinese Text Normalization for Speech Processing

## Problem

Search for "Text Normalization"(TN) on Google and Github, you can hardly find open-source projects that are "read-to-use" for text normalization tasks. Instead, you find a bunch of NLP toolkits or frameworks that *supports* TN functionality.  There is quite some work between "support text normalization" and "do text normalization".

## Reason

* TN is language-dependent, more or less.

    Some of TN processing methods are shared across languages, but a good TN module always involves language-specific knowledge and treatments, more or less.

* TN is task-specific.

    Even for the same language, different applications require quite different TN.

* TN is "dirty"

    Constructing and maintaining a set of TN rewrite-rules is painful, whatever toolkits and frameworks you choose.  Subtle and intrinsic complexities hide inside TN task itself, not in tools or frameworks.

* mature TN module is an asset

    Since constructing and maintaining TN is hard, it is actually an asset for commercial companies, hence it is unlikely to find a product-level TN in open-source community (correct me if you find any)

* TN is a less important topic for either academic or commercials.

## Goal

This project sets up a ready-to-use TN module for **Chinese**. Since my background is **speech processing**, this project should be able to handle most common TN tasks, in **Chinese ASR** text processing pipelines.

## Normalizers

1. supported NSW (Non-Standard-Word) Normalization

    |NSW type|raw|normalized|
    |-|-|-|
    |cardinal|这块黄金重达324.75克|这块黄金重达三百二十四点七五克|
    |date|她出生于86年8月18日，她弟弟出生于1995年3月1日|她出生于八六年八月十八日 她弟弟出生于一九九五年三月一日|
    |digit|电影中梁朝伟扮演的陈永仁的编号27149|电影中梁朝伟扮演的陈永仁的编号二七一四九|
    |fraction|现场有7/12的观众投出了赞成票|现场有十二分之七的观众投出了赞成票|
    |money|随便来几个价格12块5，34.5元，20.1万|随便来几个价格十二块五 三十四点五元 二十点一万|
    |percentage|明天有62％的概率降雨|明天有百分之六十二的概率降雨|
    |telephone|这是固话0421-33441122<br>这是手机+86 18544139121|这是固话零四二一三三四四一一二二<br>这是手机八六一八五四四一三九一二一|

    acknowledgement: the NSW normalization codes are based on [Zhiyang Zhou's work here](https://github.com/Joee1995/chn_text_norm.git)

1. punctuation removal

    For Chinese, it removes punctuation list collected in [Zhon](https://github.com/tsroten/zhon) project, containing
    * non-stop puncs
        ```
        '＂＃＄％＆＇（）＊＋，－／：；＜＝＞＠［＼］＾＿｀｛｜｝～｟｠｢｣､、〃》「」『』【】〔〕〖〗〘〙〚〛〜〝〞〟〰〾〿–—‘’‛“”„‟…‧﹏'
        ```
    * stop puncs
        ```
        '！？｡。'
        ```

    For English, it removes Python's `string.punctuation`

1. multilingual English word upper/lower case conversion
    since ASR/TTS lexicons usually unify English entries to uppercase or lowercase, the TN module should adapt with lexicon accordingly.

## Supported text format

1. plain text, preferably one sentence per line(most common case in ASR processing).
    ```
    今天早饭吃了没
    没吃回家吃去吧
    ...
    ```
    plain text is default format.

2. Kaldi's transcription format
    ```
    KALDI_KEY_UTT001    今天早饭吃了没
    KALDI_KEY_UTT002    没吃回家吃去吧
    ...
    ```
    TN will skip first column key section, normalize latter transcription text

    pass `--has_key` option to switch to kaldi format.

_note: All input text should be UTF-8 encoded._

## Run examples

* TN (python)

make sure you have **python3**, python2.X won't work correctly.

`sh run.sh` in `TN` dir, and compare raw text and normalized text.

* ITN (thrax)

make sure you  have **thrax** installed, and your PATH should be able to find thrax binaries.

`sh run.sh` in `ITN` dir. check Makefile for grammar dependency.

## possible future work

Since TN is a typical "done is better than perfect" module in context of ASR, and the current state is sufficient for my purpose, I probably won't update this repo frequently.

there are indeed something that needs to be improved:

* For TN, NSW normalizers in TN dir are based on regular expression, I've found some unintended matches, those pattern regexps need to be refined for more precise TN coverage.

* For ITN, extend those thrax rewriting grammars to cover more scenarios.

* Further more, nowadays commercial systems start to introduce RNN-like models into TN, and a mix of (rule-based & model-based) system is state-of-the-art.  More readings about this, look for Richard Sproat and KyleGorman's work at Google.

END