Cherry-pick to r1.4 branch (#3798)
* [TTS]add Diffsinger with opencpop dataset (#3005)
* Update requirements.txt
* fix vits reduce_sum's input/output dtype, test=tts (#3028)
* [TTS] add opencpop PWGAN example (#3031)
* add opencpop voc, test=tts
* soft link
* Update textnorm_test_cases.txt
* [TTS] add opencpop HIFIGAN example (#3038)
* add opencpop voc, test=tts
* soft link
* add opencpop hifigan, test=tts
* update
* fix dtype diff of last expand_v2 op of VITS (#3041)
* [ASR]add squeezeformer model (#2755)
* add squeezeformer model
* change CodeStyle, test=asr
* change CodeStyle, test=asr
* fix subsample rate error, test=asr
* merge classes as required, test=asr
* change CodeStyle, test=asr
* fix missing code, test=asr
* split code to new file, test=asr
* remove rel_shift, test=asr
* Update README.md
* Update README_cn.md
* Update README.md
* Update README_cn.md
* Update README.md
* fix input dtype of elementwise_mul op from bool to int64 (#3054)
* [TTS] add svs frontend (#3062)
* [TTS]clean starganv2 vc model code and add docstring (#2987)
* clean code
* add docstring
* [Doc] change define asr server config to chunk asr config, test=doc (#3067)
* Update README.md
* Update README_cn.md
* get music score, test=doc (#3070)
* [TTS]fix elementwise_floordiv's fill_constant (#3075)
* fix elementwise_floordiv's fill_constant
* add float converter for min_value in attention
* fix paddle2onnx's install version, install the newest paddle2onnx in run.sh (#3084)
* [TTS] update svs_music_score.md (#3085)
* rm unused dep, test=tts (#3097)
* Update bug-report-tts.md (#3120)
* [TTS]Fix VITS lite infer (#3098)
* [TTS]add starganv2 vc trainer (#3143)
* add starganv2 vc trainer
* fix StarGANv2VCUpdater and losses
* fix StarGANv2VCEvaluator
* add some typehint
* [TTS]【Hackathon + No.190】 + 模型复现:iSTFTNet (#3006)
* iSTFTNet implementation based on hifigan, not affect the function and execution of HIFIGAN
* modify the comment in iSTFT.yaml
* add the comments in hifigan
* iSTFTNet implementation based on hifigan, not affect the function and execution of HIFIGAN
* modify the comment in iSTFT.yaml
* add the comments in hifigan
* add iSTFTNet.md
* modify the format of iSTFTNet.md
* modify iSTFT.yaml and hifigan.py
* Format code using pre-commit
* modify hifigan.py,delete the unused self.istft_layer_id , move the self.output_conv behind else, change conv_post to output_conv
* update iSTFTNet_csmsc_ckpt.zip download link
* modify iSTFTNet.md
* modify hifigan.py and iSTFT.yaml
* modify iSTFTNet.md
* add function for generating srt file (#3123)
* add function for generating srt file
在原来websocket_client.py的基础上,增加了由wav或mp3格式的音频文件生成对应srt格式字幕文件的功能
* add function for generating srt file
在原来websocket_client.py的基础上,增加了由wav或mp3格式的音频文件生成对应srt格式字幕文件的功能
* keep origin websocket_client.py
恢复原本的websocket_client.py文件
* add generating subtitle function into README
* add generate subtitle funciton into README
* add subtitle generation function
* add subtitle generation function
* fix example/aishell local/train.sh if condition bug, test=asr (#3146)
* fix some preprocess bugs (#3155)
* add amp for U2 conformer.
* fix scaler save
* fix scaler save and load.
* mv scaler.unscale_ blow grad_clip.
* [TTS]add StarGANv2VC preprocess (#3163)
* [TTS] [黑客松]Add JETS (#3109)
* Update quick_start.md (#3175)
* [BUG] Fix progress bar unit. (#3177)
* Update quick_start_cn.md (#3176)
* [TTS]StarGANv2 VC fix some trainer bugs, add add reset_parameters (#3182)
* VITS learning rate revised, test=tts
* VITS learning rate revised, test=tts
* [s2t] mv dataset into paddlespeech.dataset (#3183)
* mv dataset into paddlespeech.dataset
* add aidatatang
* fix import
* Fix some typos. (#3178)
* [s2t] move s2t data preprocess into paddlespeech.dataset (#3189)
* move s2t data preprocess into paddlespeech.dataset
* avg model, compute wer, format rsl into paddlespeech.dataset
* fix format rsl
* fix avg ckpts
* Update pretrained model in README (#3193)
* [TTS]Fix losses of StarGAN v2 VC (#3184)
* VITS learning rate revised, test=tts
* VITS learning rate revised, test=tts
* add new aishell model for better CER.
* add readme
* [s2t] fix cli args to config (#3194)
* fix cli args to config
* fix train cli
* Update README.md
* [ASR] Support Hubert, fintuned on the librispeech dataset (#3088)
* librispeech hubert, test=asr
* librispeech hubert, test=asr
* hubert decode
* review
* copyright, notes, example related
* hubert cli
* pre-commit format
* fix conflicts
* fix conflicts
* doc related
* doc and train config
* librispeech.py
* support hubert cli
* [ASR] fix asr 0-d tensor. (#3214)
* Update README.md
* Update README.md
* fix: 🐛 修复服务端 python ASREngine 无法使用conformer_talcs模型 (#3230)
* fix: 🐛 fix python ASREngine not pass codeswitch
* docs: 📝 Update Docs
* 修改模型判断方式
* Adding WavLM implementation
* fix model m5s
* Code clean up according to comments in https://github.com/PaddlePaddle/PaddleSpeech/pull/3242
* fix error in tts/st
* Changed the path for the uploaded weight
* Update phonecode.py
# 固话的正则 错误修改
参考https://github.com/speechio/chinese_text_normalization/blob/master/python/cn_tn.py
固化的正则为:
pattern = re.compile(r"\D((0(10|2[1-3]|[3-9]\d{2})-?)?[1-9]\d{6,7})\D")
* Adapted wavlmASR model to pretrained weights and CLI
* Changed the MD5 of the pretrained tar file due to bug fixes
* Deleted examples/librispeech/asr5/format_rsl.py
* Update released_model.md
* Code clean up for CIs
* Fixed the transpose usages ignored before
* Update setup.py
* refactor mfa scripts
* Final cleaning; Modified SSL/infer.py and README for wavlm inclusion in model options
* updating readme and readme_cn
* remove tsinghua pypi
* Update setup.py (#3294)
* Update setup.py
* refactor rhy
* fix ckpt
* add dtype param for arange API. (#3302)
* add scripts for tts code switch
* add t2s assets
* more comment on tts frontend
* fix librosa==0.8.1 numpy==1.23.5 for paddleaudio align with this version
* move ssl into t2s.frontend; fix spk_id for 0-D tensor;
* add ssml unit test
* add en_frontend file
* add mix frontend test
* fix long text oom using ssml; filter comma; update polyphonic
* remove print
* hotfix english G2P
* en frontend unit text
* fix profiler (#3323)
* old grad clip has 0d tensor problem, fix it (#3334)
* update to py3.8
* remove fluid.
* add roformer
* fix bugs
* add roformer result
* support position interpolation for langer attention context windown length.
* RoPE with position interpolation
* rope for streaming decoding
* update result
* fix rotary embeding
* Update README.md
* fix weight decay
* fix develop view confict with model's
* Add XPU support for SpeedySpeech (#3502)
* Add XPU support for SpeedySpeech
* fix typos
* update description of nxpu
* Add XPU support for FastSpeech2 (#3514)
* Add XPU support for FastSpeech2
* optimize
* Update ge2e_clone.py (#3517)
修复在windows上的多空格错误
* Fix Readme. (#3527)
* Update README.md
* Update README_cn.md
* Update README_cn.md
* Update README.md
* FIX: Added missing imports
* FIX: Fixed the implementation of a special method
* 【benchmark】add max_mem_reserved for benchmark (#3604)
* fix profiler
* add max_mem_reserved for benchmark
* fix develop bug function:view to reshape (#3633)
* 【benchmark】fix gpu_mem unit (#3634)
* fix profiler
* add max_mem_reserved for benchmark
* fix benchmark
* 增加文件编码读取 (#3606)
Fixed #3605
* bugfix: audio_len should be 1D, no 0D, which will raise list index out (#3490)
of range error in the following decode process
Co-authored-by: Luzhenhui <luzhenhui@mqsz.com>
* Update README.md (#3532)
Fixed a typo
* fixed version for paddlepaddle. (#3701)
* fixed version for paddlepaddle.
* fix code style
* 【Fix Speech Issue No.5】issue 3444 transformation import error (#3779)
* fix paddlespeech.s2t.transform.transformation import error
* fix paddlespeech.s2t.transform import error
* 【Fix Speech Issue No.8】issue 3652 merge_yi function has a bug (#3786)
* 【Fix Speech Issue No.8】issue 3652 merge_yi function has a bug
* 【Fix Speech Issue No.8】issue 3652 merge_yi function has a bug
* 【test】add cli test readme (#3784)
* add cli test readme
* fix code style
* 【test】fix test cli bug (#3793)
* add cli test readme
* fix code style
* fix bug
* Update setup.py (#3795)
* adapt view behavior change, fix KeyError. (#3794)
* adapt view behavior change, fix KeyError.
* fix readme demo run error.
* fixed opencc version
---------
Co-authored-by: liangym <34430015+lym0302@users.noreply.github.com>
Co-authored-by: TianYuan <white-sky@qq.com>
Co-authored-by: 夜雨飘零 <yeyupiaoling@foxmail.com>
Co-authored-by: zxcd <228587199@qq.com>
Co-authored-by: longRookie <68834517+longRookie@users.noreply.github.com>
Co-authored-by: twoDogy <128727742+twoDogy@users.noreply.github.com>
Co-authored-by: lemondy <lemondy9@gmail.com>
Co-authored-by: ljhzxc <33015549+ljhzxc@users.noreply.github.com>
Co-authored-by: PiaoYang <495384481@qq.com>
Co-authored-by: WongLaw <mailoflawrence@gmail.com>
Co-authored-by: Hui Zhang <zhtclz@foxmail.com>
Co-authored-by: Shuangchi He <34329208+Yulv-git@users.noreply.github.com>
Co-authored-by: TianHao Zhang <32243340+Zth9730@users.noreply.github.com>
Co-authored-by: guanyc <guanyc@gmail.com>
Co-authored-by: jiamingkong <kinetical@live.com>
Co-authored-by: zoooo0820 <zoooo0820@qq.com>
Co-authored-by: shuishu <990941859@qq.com>
Co-authored-by: LixinGuo <18510030324@126.com>
Co-authored-by: gmm <38800877+mmglove@users.noreply.github.com>
Co-authored-by: Wang Huan <wanghuan29@baidu.com>
Co-authored-by: Kai Song <50285351+USTCKAY@users.noreply.github.com>
Co-authored-by: skyboooox <zcj924@gmail.com>
Co-authored-by: fazledyn-or <ataf@openrefactory.com>
Co-authored-by: luyao-cv <1367355728@qq.com>
Co-authored-by: Color_yr <402067010@qq.com>
Co-authored-by: JeffLu <luzhenhui@gmail.com>
Co-authored-by: Luzhenhui <luzhenhui@mqsz.com>
Co-authored-by: satani99 <42287151+satani99@users.noreply.github.com>
Co-authored-by: mjxs <52824616+kk-2000@users.noreply.github.com>
Co-authored-by: Mattheliu <leonliuzx@outlook.com>
4 months ago
|
|
|
|
# Copyright 2021 Mobvoi Inc. All Rights Reserved.
|
|
|
|
|
# flake8: noqa
|
|
|
|
|
import codecs
|
|
|
|
|
import re
|
|
|
|
|
import sys
|
|
|
|
|
import unicodedata
|
|
|
|
|
|
|
|
|
|
remove_tag = True
|
|
|
|
|
spacelist = [' ', '\t', '\r', '\n']
|
|
|
|
|
puncts = [
|
|
|
|
|
'!', ',', '?', '、', '。', '!', ',', ';', '?', ':', '「', '」', '︰', '『', '』',
|
|
|
|
|
'《', '》'
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def characterize(string):
|
|
|
|
|
res = []
|
|
|
|
|
i = 0
|
|
|
|
|
while i < len(string):
|
|
|
|
|
char = string[i]
|
|
|
|
|
if char in puncts:
|
|
|
|
|
i += 1
|
|
|
|
|
continue
|
|
|
|
|
cat1 = unicodedata.category(char)
|
|
|
|
|
#https://unicodebook.readthedocs.io/unicode.html#unicode-categories
|
|
|
|
|
if cat1 == 'Zs' or cat1 == 'Cn' or char in spacelist: # space or not assigned
|
|
|
|
|
i += 1
|
|
|
|
|
continue
|
|
|
|
|
if cat1 == 'Lo': # letter-other
|
|
|
|
|
res.append(char)
|
|
|
|
|
i += 1
|
|
|
|
|
else:
|
|
|
|
|
# some input looks like: <unk><noise>, we want to separate it to two words.
|
|
|
|
|
sep = ' '
|
|
|
|
|
if char == '<': sep = '>'
|
|
|
|
|
j = i + 1
|
|
|
|
|
while j < len(string):
|
|
|
|
|
c = string[j]
|
|
|
|
|
if ord(c) >= 128 or (c in spacelist) or (c == sep):
|
|
|
|
|
break
|
|
|
|
|
j += 1
|
|
|
|
|
if j < len(string) and string[j] == '>':
|
|
|
|
|
j += 1
|
|
|
|
|
res.append(string[i:j])
|
|
|
|
|
i = j
|
|
|
|
|
return res
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def stripoff_tags(x):
|
|
|
|
|
if not x: return ''
|
|
|
|
|
chars = []
|
|
|
|
|
i = 0
|
|
|
|
|
T = len(x)
|
|
|
|
|
while i < T:
|
|
|
|
|
if x[i] == '<':
|
|
|
|
|
while i < T and x[i] != '>':
|
|
|
|
|
i += 1
|
|
|
|
|
i += 1
|
|
|
|
|
else:
|
|
|
|
|
chars.append(x[i])
|
|
|
|
|
i += 1
|
|
|
|
|
return ''.join(chars)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def normalize(sentence, ignore_words, cs, split=None):
|
|
|
|
|
""" sentence, ignore_words are both in unicode
|
|
|
|
|
"""
|
|
|
|
|
new_sentence = []
|
|
|
|
|
for token in sentence:
|
|
|
|
|
x = token
|
|
|
|
|
if not cs:
|
|
|
|
|
x = x.upper()
|
|
|
|
|
if x in ignore_words:
|
|
|
|
|
continue
|
|
|
|
|
if remove_tag:
|
|
|
|
|
x = stripoff_tags(x)
|
|
|
|
|
if not x:
|
|
|
|
|
continue
|
|
|
|
|
if split and x in split:
|
|
|
|
|
new_sentence += split[x]
|
|
|
|
|
else:
|
|
|
|
|
new_sentence.append(x)
|
|
|
|
|
return new_sentence
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
class Calculator:
|
|
|
|
|
def __init__(self):
|
|
|
|
|
self.data = {}
|
|
|
|
|
self.space = []
|
|
|
|
|
self.cost = {}
|
|
|
|
|
self.cost['cor'] = 0
|
|
|
|
|
self.cost['sub'] = 1
|
|
|
|
|
self.cost['del'] = 1
|
|
|
|
|
self.cost['ins'] = 1
|
|
|
|
|
|
|
|
|
|
def calculate(self, lab, rec):
|
|
|
|
|
# Initialization
|
|
|
|
|
lab.insert(0, '')
|
|
|
|
|
rec.insert(0, '')
|
|
|
|
|
while len(self.space) < len(lab):
|
|
|
|
|
self.space.append([])
|
|
|
|
|
for row in self.space:
|
|
|
|
|
for element in row:
|
|
|
|
|
element['dist'] = 0
|
|
|
|
|
element['error'] = 'non'
|
|
|
|
|
while len(row) < len(rec):
|
|
|
|
|
row.append({'dist': 0, 'error': 'non'})
|
|
|
|
|
for i in range(len(lab)):
|
|
|
|
|
self.space[i][0]['dist'] = i
|
|
|
|
|
self.space[i][0]['error'] = 'del'
|
|
|
|
|
for j in range(len(rec)):
|
|
|
|
|
self.space[0][j]['dist'] = j
|
|
|
|
|
self.space[0][j]['error'] = 'ins'
|
|
|
|
|
self.space[0][0]['error'] = 'non'
|
|
|
|
|
for token in lab:
|
|
|
|
|
if token not in self.data and len(token) > 0:
|
|
|
|
|
self.data[token] = {
|
|
|
|
|
'all': 0,
|
|
|
|
|
'cor': 0,
|
|
|
|
|
'sub': 0,
|
|
|
|
|
'ins': 0,
|
|
|
|
|
'del': 0
|
|
|
|
|
}
|
|
|
|
|
for token in rec:
|
|
|
|
|
if token not in self.data and len(token) > 0:
|
|
|
|
|
self.data[token] = {
|
|
|
|
|
'all': 0,
|
|
|
|
|
'cor': 0,
|
|
|
|
|
'sub': 0,
|
|
|
|
|
'ins': 0,
|
|
|
|
|
'del': 0
|
|
|
|
|
}
|
|
|
|
|
# Computing edit distance
|
|
|
|
|
for i, lab_token in enumerate(lab):
|
|
|
|
|
for j, rec_token in enumerate(rec):
|
|
|
|
|
if i == 0 or j == 0:
|
|
|
|
|
continue
|
|
|
|
|
min_dist = sys.maxsize
|
|
|
|
|
min_error = 'none'
|
|
|
|
|
dist = self.space[i - 1][j]['dist'] + self.cost['del']
|
|
|
|
|
error = 'del'
|
|
|
|
|
if dist < min_dist:
|
|
|
|
|
min_dist = dist
|
|
|
|
|
min_error = error
|
|
|
|
|
dist = self.space[i][j - 1]['dist'] + self.cost['ins']
|
|
|
|
|
error = 'ins'
|
|
|
|
|
if dist < min_dist:
|
|
|
|
|
min_dist = dist
|
|
|
|
|
min_error = error
|
|
|
|
|
if lab_token == rec_token:
|
|
|
|
|
dist = self.space[i - 1][j - 1]['dist'] + self.cost['cor']
|
|
|
|
|
error = 'cor'
|
|
|
|
|
else:
|
|
|
|
|
dist = self.space[i - 1][j - 1]['dist'] + self.cost['sub']
|
|
|
|
|
error = 'sub'
|
|
|
|
|
if dist < min_dist:
|
|
|
|
|
min_dist = dist
|
|
|
|
|
min_error = error
|
|
|
|
|
self.space[i][j]['dist'] = min_dist
|
|
|
|
|
self.space[i][j]['error'] = min_error
|
|
|
|
|
# Tracing back
|
|
|
|
|
result = {
|
|
|
|
|
'lab': [],
|
|
|
|
|
'rec': [],
|
|
|
|
|
'all': 0,
|
|
|
|
|
'cor': 0,
|
|
|
|
|
'sub': 0,
|
|
|
|
|
'ins': 0,
|
|
|
|
|
'del': 0
|
|
|
|
|
}
|
|
|
|
|
i = len(lab) - 1
|
|
|
|
|
j = len(rec) - 1
|
|
|
|
|
while True:
|
|
|
|
|
if self.space[i][j]['error'] == 'cor': # correct
|
|
|
|
|
if len(lab[i]) > 0:
|
|
|
|
|
self.data[lab[i]]['all'] = self.data[lab[i]]['all'] + 1
|
|
|
|
|
self.data[lab[i]]['cor'] = self.data[lab[i]]['cor'] + 1
|
|
|
|
|
result['all'] = result['all'] + 1
|
|
|
|
|
result['cor'] = result['cor'] + 1
|
|
|
|
|
result['lab'].insert(0, lab[i])
|
|
|
|
|
result['rec'].insert(0, rec[j])
|
|
|
|
|
i = i - 1
|
|
|
|
|
j = j - 1
|
|
|
|
|
elif self.space[i][j]['error'] == 'sub': # substitution
|
|
|
|
|
if len(lab[i]) > 0:
|
|
|
|
|
self.data[lab[i]]['all'] = self.data[lab[i]]['all'] + 1
|
|
|
|
|
self.data[lab[i]]['sub'] = self.data[lab[i]]['sub'] + 1
|
|
|
|
|
result['all'] = result['all'] + 1
|
|
|
|
|
result['sub'] = result['sub'] + 1
|
|
|
|
|
result['lab'].insert(0, lab[i])
|
|
|
|
|
result['rec'].insert(0, rec[j])
|
|
|
|
|
i = i - 1
|
|
|
|
|
j = j - 1
|
|
|
|
|
elif self.space[i][j]['error'] == 'del': # deletion
|
|
|
|
|
if len(lab[i]) > 0:
|
|
|
|
|
self.data[lab[i]]['all'] = self.data[lab[i]]['all'] + 1
|
|
|
|
|
self.data[lab[i]]['del'] = self.data[lab[i]]['del'] + 1
|
|
|
|
|
result['all'] = result['all'] + 1
|
|
|
|
|
result['del'] = result['del'] + 1
|
|
|
|
|
result['lab'].insert(0, lab[i])
|
|
|
|
|
result['rec'].insert(0, "")
|
|
|
|
|
i = i - 1
|
|
|
|
|
elif self.space[i][j]['error'] == 'ins': # insertion
|
|
|
|
|
if len(rec[j]) > 0:
|
|
|
|
|
self.data[rec[j]]['ins'] = self.data[rec[j]]['ins'] + 1
|
|
|
|
|
result['ins'] = result['ins'] + 1
|
|
|
|
|
result['lab'].insert(0, "")
|
|
|
|
|
result['rec'].insert(0, rec[j])
|
|
|
|
|
j = j - 1
|
|
|
|
|
elif self.space[i][j]['error'] == 'non': # starting point
|
|
|
|
|
break
|
|
|
|
|
else: # shouldn't reach here
|
|
|
|
|
print(
|
|
|
|
|
'this should not happen , i = {i} , j = {j} , error = {error}'.
|
|
|
|
|
format(i=i, j=j, error=self.space[i][j]['error']))
|
|
|
|
|
return result
|
|
|
|
|
|
|
|
|
|
def overall(self):
|
|
|
|
|
result = {'all': 0, 'cor': 0, 'sub': 0, 'ins': 0, 'del': 0}
|
|
|
|
|
for token in self.data:
|
|
|
|
|
result['all'] = result['all'] + self.data[token]['all']
|
|
|
|
|
result['cor'] = result['cor'] + self.data[token]['cor']
|
|
|
|
|
result['sub'] = result['sub'] + self.data[token]['sub']
|
|
|
|
|
result['ins'] = result['ins'] + self.data[token]['ins']
|
|
|
|
|
result['del'] = result['del'] + self.data[token]['del']
|
|
|
|
|
return result
|
|
|
|
|
|
|
|
|
|
def cluster(self, data):
|
|
|
|
|
result = {'all': 0, 'cor': 0, 'sub': 0, 'ins': 0, 'del': 0}
|
|
|
|
|
for token in data:
|
|
|
|
|
if token in self.data:
|
|
|
|
|
result['all'] = result['all'] + self.data[token]['all']
|
|
|
|
|
result['cor'] = result['cor'] + self.data[token]['cor']
|
|
|
|
|
result['sub'] = result['sub'] + self.data[token]['sub']
|
|
|
|
|
result['ins'] = result['ins'] + self.data[token]['ins']
|
|
|
|
|
result['del'] = result['del'] + self.data[token]['del']
|
|
|
|
|
return result
|
|
|
|
|
|
|
|
|
|
def keys(self):
|
|
|
|
|
return list(self.data.keys())
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def width(string):
|
|
|
|
|
return sum(1 + (unicodedata.east_asian_width(c) in "AFW") for c in string)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def default_cluster(word):
|
|
|
|
|
unicode_names = [unicodedata.name(char) for char in word]
|
|
|
|
|
for i in reversed(range(len(unicode_names))):
|
|
|
|
|
if unicode_names[i].startswith('DIGIT'): # 1
|
|
|
|
|
unicode_names[i] = 'Number' # 'DIGIT'
|
|
|
|
|
elif (unicode_names[i].startswith('CJK UNIFIED IDEOGRAPH') or
|
|
|
|
|
unicode_names[i].startswith('CJK COMPATIBILITY IDEOGRAPH')):
|
|
|
|
|
# 明 / 郎
|
|
|
|
|
unicode_names[i] = 'Mandarin' # 'CJK IDEOGRAPH'
|
|
|
|
|
elif (unicode_names[i].startswith('LATIN CAPITAL LETTER') or
|
|
|
|
|
unicode_names[i].startswith('LATIN SMALL LETTER')):
|
|
|
|
|
# A / a
|
|
|
|
|
unicode_names[i] = 'English' # 'LATIN LETTER'
|
|
|
|
|
elif unicode_names[i].startswith('HIRAGANA LETTER'): # は こ め
|
|
|
|
|
unicode_names[i] = 'Japanese' # 'GANA LETTER'
|
|
|
|
|
elif (unicode_names[i].startswith('AMPERSAND') or
|
|
|
|
|
unicode_names[i].startswith('APOSTROPHE') or
|
|
|
|
|
unicode_names[i].startswith('COMMERCIAL AT') or
|
|
|
|
|
unicode_names[i].startswith('DEGREE CELSIUS') or
|
|
|
|
|
unicode_names[i].startswith('EQUALS SIGN') or
|
|
|
|
|
unicode_names[i].startswith('FULL STOP') or
|
|
|
|
|
unicode_names[i].startswith('HYPHEN-MINUS') or
|
|
|
|
|
unicode_names[i].startswith('LOW LINE') or
|
|
|
|
|
unicode_names[i].startswith('NUMBER SIGN') or
|
|
|
|
|
unicode_names[i].startswith('PLUS SIGN') or
|
|
|
|
|
unicode_names[i].startswith('SEMICOLON')):
|
|
|
|
|
# & / ' / @ / ℃ / = / . / - / _ / # / + / ;
|
|
|
|
|
del unicode_names[i]
|
|
|
|
|
else:
|
|
|
|
|
return 'Other'
|
|
|
|
|
if len(unicode_names) == 0:
|
|
|
|
|
return 'Other'
|
|
|
|
|
if len(unicode_names) == 1:
|
|
|
|
|
return unicode_names[0]
|
|
|
|
|
for i in range(len(unicode_names) - 1):
|
|
|
|
|
if unicode_names[i] != unicode_names[i + 1]:
|
|
|
|
|
return 'Other'
|
|
|
|
|
return unicode_names[0]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def usage():
|
|
|
|
|
print(
|
|
|
|
|
"compute-wer.py : compute word error rate (WER) and align recognition results and references."
|
|
|
|
|
)
|
|
|
|
|
print(
|
|
|
|
|
" usage : python compute-wer.py [--cs={0,1}] [--cluster=foo] [--ig=ignore_file] [--char={0,1}] [--v={0,1}] [--padding-symbol={space,underline}] test.ref test.hyp > test.wer"
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def main():
|
|
|
|
|
# python utils/compute-wer.py --char=1 --v=1 ref hyp > rsl.error
|
|
|
|
|
if len(sys.argv) == 1:
|
|
|
|
|
usage()
|
|
|
|
|
sys.exit(0)
|
|
|
|
|
calculator = Calculator()
|
|
|
|
|
cluster_file = ''
|
|
|
|
|
ignore_words = set()
|
|
|
|
|
tochar = False
|
|
|
|
|
verbose = 1
|
|
|
|
|
padding_symbol = ' '
|
|
|
|
|
case_sensitive = False
|
|
|
|
|
max_words_per_line = sys.maxsize
|
|
|
|
|
split = None
|
|
|
|
|
while len(sys.argv) > 3:
|
|
|
|
|
a = '--maxw='
|
|
|
|
|
if sys.argv[1].startswith(a):
|
|
|
|
|
b = sys.argv[1][len(a):]
|
|
|
|
|
del sys.argv[1]
|
|
|
|
|
max_words_per_line = int(b)
|
|
|
|
|
continue
|
|
|
|
|
a = '--rt='
|
|
|
|
|
if sys.argv[1].startswith(a):
|
|
|
|
|
b = sys.argv[1][len(a):].lower()
|
|
|
|
|
del sys.argv[1]
|
|
|
|
|
remove_tag = (b == 'true') or (b != '0')
|
|
|
|
|
continue
|
|
|
|
|
a = '--cs='
|
|
|
|
|
if sys.argv[1].startswith(a):
|
|
|
|
|
b = sys.argv[1][len(a):].lower()
|
|
|
|
|
del sys.argv[1]
|
|
|
|
|
case_sensitive = (b == 'true') or (b != '0')
|
|
|
|
|
continue
|
|
|
|
|
a = '--cluster='
|
|
|
|
|
if sys.argv[1].startswith(a):
|
|
|
|
|
cluster_file = sys.argv[1][len(a):]
|
|
|
|
|
del sys.argv[1]
|
|
|
|
|
continue
|
|
|
|
|
a = '--splitfile='
|
|
|
|
|
if sys.argv[1].startswith(a):
|
|
|
|
|
split_file = sys.argv[1][len(a):]
|
|
|
|
|
del sys.argv[1]
|
|
|
|
|
split = dict()
|
|
|
|
|
with codecs.open(split_file, 'r', 'utf-8') as fh:
|
|
|
|
|
for line in fh: # line in unicode
|
|
|
|
|
words = line.strip().split()
|
|
|
|
|
if len(words) >= 2:
|
|
|
|
|
split[words[0]] = words[1:]
|
|
|
|
|
continue
|
|
|
|
|
a = '--ig='
|
|
|
|
|
if sys.argv[1].startswith(a):
|
|
|
|
|
ignore_file = sys.argv[1][len(a):]
|
|
|
|
|
del sys.argv[1]
|
|
|
|
|
with codecs.open(ignore_file, 'r', 'utf-8') as fh:
|
|
|
|
|
for line in fh: # line in unicode
|
|
|
|
|
line = line.strip()
|
|
|
|
|
if len(line) > 0:
|
|
|
|
|
ignore_words.add(line)
|
|
|
|
|
continue
|
|
|
|
|
a = '--char='
|
|
|
|
|
if sys.argv[1].startswith(a):
|
|
|
|
|
b = sys.argv[1][len(a):].lower()
|
|
|
|
|
del sys.argv[1]
|
|
|
|
|
tochar = (b == 'true') or (b != '0')
|
|
|
|
|
continue
|
|
|
|
|
a = '--v='
|
|
|
|
|
if sys.argv[1].startswith(a):
|
|
|
|
|
b = sys.argv[1][len(a):].lower()
|
|
|
|
|
del sys.argv[1]
|
|
|
|
|
verbose = 0
|
|
|
|
|
try:
|
|
|
|
|
verbose = int(b)
|
|
|
|
|
except:
|
|
|
|
|
if b == 'true' or b != '0':
|
|
|
|
|
verbose = 1
|
|
|
|
|
continue
|
|
|
|
|
a = '--padding-symbol='
|
|
|
|
|
if sys.argv[1].startswith(a):
|
|
|
|
|
b = sys.argv[1][len(a):].lower()
|
|
|
|
|
del sys.argv[1]
|
|
|
|
|
if b == 'space':
|
|
|
|
|
padding_symbol = ' '
|
|
|
|
|
elif b == 'underline':
|
|
|
|
|
padding_symbol = '_'
|
|
|
|
|
continue
|
|
|
|
|
if True or sys.argv[1].startswith('-'):
|
|
|
|
|
#ignore invalid switch
|
|
|
|
|
del sys.argv[1]
|
|
|
|
|
continue
|
|
|
|
|
|
|
|
|
|
if not case_sensitive:
|
|
|
|
|
ig = set([w.upper() for w in ignore_words])
|
|
|
|
|
ignore_words = ig
|
|
|
|
|
|
|
|
|
|
default_clusters = {}
|
|
|
|
|
default_words = {}
|
|
|
|
|
|
|
|
|
|
ref_file = sys.argv[1]
|
|
|
|
|
hyp_file = sys.argv[2]
|
|
|
|
|
rec_set = {}
|
|
|
|
|
if split and not case_sensitive:
|
|
|
|
|
newsplit = dict()
|
|
|
|
|
for w in split:
|
|
|
|
|
words = split[w]
|
|
|
|
|
for i in range(len(words)):
|
|
|
|
|
words[i] = words[i].upper()
|
|
|
|
|
newsplit[w.upper()] = words
|
|
|
|
|
split = newsplit
|
|
|
|
|
|
|
|
|
|
with codecs.open(hyp_file, 'r', 'utf-8') as fh:
|
|
|
|
|
for line in fh:
|
|
|
|
|
if tochar:
|
|
|
|
|
array = characterize(line)
|
|
|
|
|
else:
|
|
|
|
|
array = line.strip().split()
|
|
|
|
|
if len(array) == 0: continue
|
|
|
|
|
fid = array[0]
|
|
|
|
|
rec_set[fid] = normalize(array[1:], ignore_words, case_sensitive,
|
|
|
|
|
split)
|
|
|
|
|
|
|
|
|
|
# compute error rate on the interaction of reference file and hyp file
|
|
|
|
|
for line in open(ref_file, 'r', encoding='utf-8'):
|
|
|
|
|
if tochar:
|
|
|
|
|
array = characterize(line)
|
|
|
|
|
else:
|
|
|
|
|
array = line.rstrip('\n').split()
|
|
|
|
|
if len(array) == 0: continue
|
|
|
|
|
fid = array[0]
|
|
|
|
|
if fid not in rec_set:
|
|
|
|
|
continue
|
|
|
|
|
lab = normalize(array[1:], ignore_words, case_sensitive, split)
|
|
|
|
|
rec = rec_set[fid]
|
|
|
|
|
if verbose:
|
|
|
|
|
print('\nutt: %s' % fid)
|
|
|
|
|
|
|
|
|
|
for word in rec + lab:
|
|
|
|
|
if word not in default_words:
|
|
|
|
|
default_cluster_name = default_cluster(word)
|
|
|
|
|
if default_cluster_name not in default_clusters:
|
|
|
|
|
default_clusters[default_cluster_name] = {}
|
|
|
|
|
if word not in default_clusters[default_cluster_name]:
|
|
|
|
|
default_clusters[default_cluster_name][word] = 1
|
|
|
|
|
default_words[word] = default_cluster_name
|
|
|
|
|
|
|
|
|
|
result = calculator.calculate(lab, rec)
|
|
|
|
|
if verbose:
|
|
|
|
|
if result['all'] != 0:
|
|
|
|
|
wer = float(result['ins'] + result['sub'] + result[
|
|
|
|
|
'del']) * 100.0 / result['all']
|
|
|
|
|
else:
|
|
|
|
|
wer = 0.0
|
|
|
|
|
print('WER: %4.2f %%' % wer, end=' ')
|
|
|
|
|
print('N=%d C=%d S=%d D=%d I=%d' %
|
|
|
|
|
(result['all'], result['cor'], result['sub'], result['del'],
|
|
|
|
|
result['ins']))
|
|
|
|
|
space = {}
|
|
|
|
|
space['lab'] = []
|
|
|
|
|
space['rec'] = []
|
|
|
|
|
for idx in range(len(result['lab'])):
|
|
|
|
|
len_lab = width(result['lab'][idx])
|
|
|
|
|
len_rec = width(result['rec'][idx])
|
|
|
|
|
length = max(len_lab, len_rec)
|
|
|
|
|
space['lab'].append(length - len_lab)
|
|
|
|
|
space['rec'].append(length - len_rec)
|
|
|
|
|
upper_lab = len(result['lab'])
|
|
|
|
|
upper_rec = len(result['rec'])
|
|
|
|
|
lab1, rec1 = 0, 0
|
|
|
|
|
while lab1 < upper_lab or rec1 < upper_rec:
|
|
|
|
|
if verbose > 1:
|
|
|
|
|
print('lab(%s):' % fid.encode('utf-8'), end=' ')
|
|
|
|
|
else:
|
|
|
|
|
print('lab:', end=' ')
|
|
|
|
|
lab2 = min(upper_lab, lab1 + max_words_per_line)
|
|
|
|
|
for idx in range(lab1, lab2):
|
|
|
|
|
token = result['lab'][idx]
|
|
|
|
|
print('{token}'.format(token=token), end='')
|
|
|
|
|
for n in range(space['lab'][idx]):
|
|
|
|
|
print(padding_symbol, end='')
|
|
|
|
|
print(' ', end='')
|
|
|
|
|
print()
|
|
|
|
|
if verbose > 1:
|
|
|
|
|
print('rec(%s):' % fid.encode('utf-8'), end=' ')
|
|
|
|
|
else:
|
|
|
|
|
print('rec:', end=' ')
|
|
|
|
|
rec2 = min(upper_rec, rec1 + max_words_per_line)
|
|
|
|
|
for idx in range(rec1, rec2):
|
|
|
|
|
token = result['rec'][idx]
|
|
|
|
|
print('{token}'.format(token=token), end='')
|
|
|
|
|
for n in range(space['rec'][idx]):
|
|
|
|
|
print(padding_symbol, end='')
|
|
|
|
|
print(' ', end='')
|
|
|
|
|
print('\n', end='\n')
|
|
|
|
|
lab1 = lab2
|
|
|
|
|
rec1 = rec2
|
|
|
|
|
|
|
|
|
|
if verbose:
|
|
|
|
|
print(
|
|
|
|
|
'==========================================================================='
|
|
|
|
|
)
|
|
|
|
|
print()
|
|
|
|
|
|
|
|
|
|
result = calculator.overall()
|
|
|
|
|
if result['all'] != 0:
|
|
|
|
|
wer = float(result['ins'] + result['sub'] + result[
|
|
|
|
|
'del']) * 100.0 / result['all']
|
|
|
|
|
else:
|
|
|
|
|
wer = 0.0
|
|
|
|
|
print('Overall -> %4.2f %%' % wer, end=' ')
|
|
|
|
|
print('N=%d C=%d S=%d D=%d I=%d' %
|
|
|
|
|
(result['all'], result['cor'], result['sub'], result['del'],
|
|
|
|
|
result['ins']))
|
|
|
|
|
if not verbose:
|
|
|
|
|
print()
|
|
|
|
|
|
|
|
|
|
if verbose:
|
|
|
|
|
for cluster_id in default_clusters:
|
|
|
|
|
result = calculator.cluster(
|
|
|
|
|
[k for k in default_clusters[cluster_id]])
|
|
|
|
|
if result['all'] != 0:
|
|
|
|
|
wer = float(result['ins'] + result['sub'] + result[
|
|
|
|
|
'del']) * 100.0 / result['all']
|
|
|
|
|
else:
|
|
|
|
|
wer = 0.0
|
|
|
|
|
print('%s -> %4.2f %%' % (cluster_id, wer), end=' ')
|
|
|
|
|
print('N=%d C=%d S=%d D=%d I=%d' %
|
|
|
|
|
(result['all'], result['cor'], result['sub'], result['del'],
|
|
|
|
|
result['ins']))
|
|
|
|
|
if len(cluster_file) > 0: # compute separated WERs for word clusters
|
|
|
|
|
cluster_id = ''
|
|
|
|
|
cluster = []
|
|
|
|
|
for line in open(cluster_file, 'r', encoding='utf-8'):
|
|
|
|
|
for token in line.decode('utf-8').rstrip('\n').split():
|
|
|
|
|
# end of cluster reached, like </Keyword>
|
|
|
|
|
if token[0:2] == '</' and token[len(token)-1] == '>' and \
|
|
|
|
|
token.lstrip('</').rstrip('>') == cluster_id :
|
|
|
|
|
result = calculator.cluster(cluster)
|
|
|
|
|
if result['all'] != 0:
|
|
|
|
|
wer = float(result['ins'] + result['sub'] + result[
|
|
|
|
|
'del']) * 100.0 / result['all']
|
|
|
|
|
else:
|
|
|
|
|
wer = 0.0
|
|
|
|
|
print('%s -> %4.2f %%' % (cluster_id, wer), end=' ')
|
|
|
|
|
print('N=%d C=%d S=%d D=%d I=%d' %
|
|
|
|
|
(result['all'], result['cor'], result['sub'],
|
|
|
|
|
result['del'], result['ins']))
|
|
|
|
|
cluster_id = ''
|
|
|
|
|
cluster = []
|
|
|
|
|
# begin of cluster reached, like <Keyword>
|
|
|
|
|
elif token[0] == '<' and token[len(token)-1] == '>' and \
|
|
|
|
|
cluster_id == '' :
|
|
|
|
|
cluster_id = token.lstrip('<').rstrip('>')
|
|
|
|
|
cluster = []
|
|
|
|
|
# general terms, like WEATHER / CAR / ...
|
|
|
|
|
else:
|
|
|
|
|
cluster.append(token)
|
|
|
|
|
print()
|
|
|
|
|
print(
|
|
|
|
|
'==========================================================================='
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
if __name__ == '__main__':
|
|
|
|
|
main()
|