Results


Results are presented in two sets of tables: Unconstrained LM - where teams are allowed to modify the baseline LM; Baseline LM - where the use of the baseline LM is enforced. In the unconstrained case not all systems modified the baseline LM. Teams that did so are marked with an asterisk.

1-Channel, Unconstrained LM

Systems that are not using the baseline RNLMM are marked with an asterisk.
Rank Team Paper Dev Sim Dev Real Eval Sim Eval Real
* Du et al. [paper] 6.61 4.55 11.81 9.15
* Menne et al. [paper] 7.40 5.14 12.36 9.29
* Heymann et al. [paper] 6.69 5.20 11.14 9.34
* Tachioka et al. [paper] 7.15 5.15 12.15 11.13
* Tran et al. [paper] 7.33 6.08 10.88 11.19
Fujita et al. [paper] 7.36 5.89 9.23 11.42
Alam et al. [paper] 9.27 6.79 13.72 12.70
Qian and Tan [paper] 7.93 6.29 12.95 13.91
del-Agua et al. [paper] 11.13 9.95 15.72 16.11
* Tanaka et al. [paper] 10.15 8.06 15.67 16.58
Matassoni et al. [paper] 9.51 9.02 16.09 16.87
Bayestehtashk and Shafran [paper] 12.05 9.83 19.15 18.63
Xiao et al. [paper] 14.25 11.43 21.37 20.94
Baseline - 12.98 11.57 20.84 23.70

[TOP]



2-Channel, Unconstrained LM


Rank Team Paper Dev Sim Dev Real Eval Sim Eval Real
* Du et al. [paper] 3.46 2.33 5.74 3.91
* Menne et al. [paper] 4.04 3.02 5.27 5.32
* Heymann et al. [paper] 4.05 3.54 5.16 5.96
* Tachioka et al. [paper] 4.63 3.50 8.03 7.28
* Tran et al. [paper] 5.10 4.32 7.57 8.33
Fujita et al. [paper] 5.88 4.22 7.32 8.61
Qian and Tan [paper] 5.73 4.79 8.74 9.14
Wang et al. [paper] 7.18 5.58 8.77 9.64
Alam et al. [paper] 6.69 5.13 10.34 9.97
Xiao et al. [paper] 7.06 5.86 10.72 10.46
Zhang et al. [paper] 6.26 5.48 7.80 11.02
del-Agua et al. [paper] 8.93 7.96 12.06 12.82
* Schrank et al. [paper] 8.29 6.46 14.34 13.71
Bayestehtashk and Shafran [paper] 8.78 7.33 13.91 13.81
Baseline - 9.50 8.23 15.33 16.58

[TOP]



6-Channel, Unconstrained LM


Rank Team Paper Dev Sim Dev Real Eval Sim Eval Real
* Du et al. [paper] 1.78 1.69 2.12 2.24
* Menne et al. [paper] 2.34 2.19 2.68 2.91
* Erdogan et al. [paper] 1.95 2.11 1.97 2.98
* Heymann et al. [paper] 2.34 2.73 2.76 3.48
* Tachioka et al. [paper] 2.32 2.30 4.18 4.31
Fujita et al. [paper] 3.33 2.68 4.15 4.68
* Tran et al. [paper] 3.39 3.18 4.97 5.00
Wang et al. [paper] 3.71 3.65 4.38 5.69
Zhang et al. [paper] 3.66 3.62 4.83 6.00
Qian and Tan [paper] 4.37 3.73 6.29 6.41
Xiao et al. [paper] 4.12 4.28 4.99 6.43
* Xiang et al. [paper] 3.78 4.58 4.95 6.55
Alam et al. [paper] 5.07 4.00 7.53 7.23
* Schrank et al. [paper] 4.73 4.22 7.02 8.99
Bayestehtashk and Shafran [paper] 6.34 5.38 10.10 10.09
Baseline - 6.77 5.76 10.90 11.51

[TOP]

Results obtained when using the baseline RNNLM

Teams were asked to report one set of results using the baseline RNNLM.

1-Channel, Baseline LM


Rank Team Paper Dev Sim Dev Real Eval Sim Eval Real
Du et al. [paper] 8.24 6.10 13.62 11.15
Heymann et al. [paper] 7.18 5.50 11.68 9.88
Tran et al. [paper] - - - 12.92
Fujita et al. [paper] 7.36 5.89 9.23 11.42
Alam et al. [paper] 9.27 6.79 13.72 12.70
Qian and Tan [paper] 7.93 6.29 12.95 13.91
del-Agua et al. [paper] 11.13 9.95 15.72 16.11
Tanaka et al. [paper] 10.86 9.09 16.49 17.40
Matassoni et al. [paper] 9.51 9.02 16.09 16.87
Bayestehtashk and Shafran [paper] 12.05 9.83 19.15 18.63
Xiao et al. [paper] 14.25 11.43 21.37 20.94
Baseline - 12.98 11.57 20.84 23.70

[TOP]



2-Channel, Baseline LM


Rank Team Paper Dev Sim Dev Real Eval Sim Eval Real
Du et al. [paper] 4.89 3.56 7.30 5.41
Heymann et al. [paper] 4.45 3.80 5.38 6.44
Tran et al. [paper] - - - 9.77
Fujita et al. [paper] 5.88 4.22 7.32 8.61
Qian and Tan [paper] 5.73 4.79 8.74 9.14
Wang et al. [paper] 7.18 5.58 8.77 9.64
Alam et al. [paper] 6.69 5.13 10.34 9.97
Xiao et al. [paper] 7.06 5.86 10.72 10.46
Zhang et al. [paper] 6.26 5.48 7.80 11.02
del-Agua et al. [paper] 8.93 7.96 12.06 12.82
Schrank et al. [paper] 8.54 6.66 14.46 14.02
Bayestehtashk and Shafran [paper] 8.78 7.33 13.91 13.81
Baseline - 9.50 8.23 15.33 16.58

[TOP]



6-Channel, Baseline LM


Rank Team Paper Dev Sim Dev Real Eval Sim Eval Real
Du et al. [paper] 2.61 2.55 3.06 3.24
Menne et al. [paper] 3.79 4.16 4.04 5.17
Erdogan et al. [paper] 3.28 3.34 3.35 4.44
Heymann et al. [paper] 2.75 2.84 3.11 3.85
Fujita et al. [paper] 3.33 2.68 4.15 4.68
Tran et al. [paper] - - - 6.18
Wang et al. [paper] 3.71 3.65 4.38 5.69
Zhang et al. [paper] 3.66 3.62 4.83 6.00
Qian and Tan [paper] 4.37 3.73 6.29 6.41
Xiao et al. [paper] 4.12 4.28 4.99 6.43
Xiang et al. [paper] 4.22 4.64 5.51 7.15
Alam et al. [paper] 5.07 4.00 7.53 7.23
Schrank et al. [paper] 4.14 4.83 6.16 9.17
Bayestehtashk and Shafran [paper] 6.34 5.38 10.10 10.09
Baseline - 6.77 5.76 10.90 11.51

[TOP]