A Multilingual , Multimodal Speech Training System, SPECO

Eurospeech 2001 - Scandinavia

$0XOWLOLQJXDO0XOWLPRGDO6SHHFK7UDLQLQJ6\VWHP 63(&2 .9LFVL 35RDFK‚$gVWHUÁ=.DFLFA)&VDWiUL $6IDNLDQDNL‚59HURQLNA

%XGDSHVW8QLYHUVLW\RI7HFKQRORJ\DQG(FRQRPLFV+XQJDU\‚8QLYHUVLW\RI5HDGLQJ8QLWHG .LQJGRPÁ.XQJO7HNQLVND+|JVNRODQ6ZHGHQA8QLYHUVLW\RI0DULERU6ORYHQLD

YLFVL#DOSKDWWWEPHKX  

$&2867,&0($685(0(1702'8/(

$EVWUDFW

7KLVPRGXOHLVDODQJXDJHLQGHSHQGHQWV\VWHPZKLFKPHDVXUHV

7KH63(&23URMHFWZDVIXQGHGE\WKH(8WKURXJKWKH,1&2

DQG YLVXDOL]HV WKH FKDUDFWHULVWLF SDUDPHWHUV RI VSHHFK VXFK DV

&23(51,&86SURJUDP&RQWUDFWQR LQ,QWKH

DXGLWRU\

IUDPH RI WKH SURMHFW D V\VWHP KDV EHHQ GHYHORSHG ZKLFK LV DQ

VRXQG HQHUJ\ SLWFK YRLFHG  XQYRLFHG GHWHFWLRQ LQWRQDWLRQ

DXGLRYLVXDO SURQXQFLDWLRQ WHDFKLQJ DQGWUDLQLQJ V\VWHP IRU 

UK\WKPHWFDVGHVFULEHGLQDQHDUOLHUSDSHU>@

\HDUROGFKLOGUHQ&RUUHFWLRQRIGLVRUGHUHGDVSHFWVRIVSHHFK

5(&25',1*02'8/(

LV GRQH E\ UHDO WLPH YLVXDO SUHVHQWDWLRQ RI WKH VSHHFK

7KH UHFRUGLQJ PRGXOH RI WKH HGLWRU JLYHV WKH SRVVLELOLW\ RI

SDUDPHWHUVLQD ZD\ WKDWLV XQGHUVWDQGDEOH DQG LQWHUHVWLQJ IRU

FUHDWLQJWKHUHIHUHQFHVSHHFKYRFDEXODU\ RIWKH9RZHODQGWKH

\RXQJ FKLOGUHQ ZKLOH UHPDLQLQJ FRUUHFW IURP WKH DFRXVWLF

6LELODQW 6XSSRUW 7KLV YRFDEXODU\ FRQWDLQV WKH UHIHUHQFH

SKRQHWLFSRLQWRIYLHZ7KHGHYHORSPHQWRIWKHVSHHFKE\ RXU

H[DPSOHV VRXQG VHTXHQFHV ZRUGV ZRUGSDLUV DQG VHQWHQFHV

WHDFKLQJ PHWKRG LV PDGH PDLQO\ RQ WKH EDVLV RI YLVXDO

$ VSHFLDO YRFDEXODU\ KDV EHHQ FRQVWUXFWHG IRU HDFK ODQJXDJH

LQIRUPDWLRQ XVLQJ WKH LQWDFW YLVXDO FKDQQHO RI WKH KHDULQJ

&DUHIXOO\ VHOHFWHG FKLOGUHQ XVLQJ QRUPDO VRXQG LQWHQVLW\

LPSDLUHG FKLOG +RZHYHU GXULQJ SUDFWLFH ZH XVH WKHLU OLPLWHG

SURGXFHG WKH UHIHUHQFH H[DPSOHV E\ SURQRXQFLQJ DOO RI WKH

DXGLWRU\

SKRQHPHV LQ WKH H[SUHVVLRQ LQ D FRUUHFW ZD\ LQ VLOHQW

FKDQQHO

WRR

E\

JLYLQJ

DXGLWRU\

LQIRUPDWLRQ

VSHFWUXP

VSHFWURJUDP

VSHFWURJUDP

GLIIHUHQFHV

V\QFKURQLVHGZLWKWKHYLVXDORQH7KLVPXOWLPRGDOWUDLQLQJDQG

VXUURXQGLQJVWKHVLJQDOWRQRLVHUDWLRPLQG% 

WHDFKLQJ V\VWHP KDV EHHQ GHYHORSHG IRU ODQJXDJHV RI DOO WKH

6(*0(17(502'8/(

63(&2 SDUWQHUV WKHVH DUH (QJOLVK 6ZHGLVK 6ORYHQLDQ DQG

7KH SODFH RI WKH SKRQHPHV LV PDUNHG RQ WKH DXGLWRU\

+XQJDULDQ>@

VSHFWURJUDPVVHH )LJ   7KXVVHJPHQWDWLRQ DQGODEHOOLQJ RI WKHUHIHUHQFHH[DPSOHVZDVQHFHVVDU\DWWKHSKRQHPHOHYHO,Q



WKHGLDORJER[RIWKHVHJPHQWHGPRGXOHRIWKH(',725WKHUH

,QWURGXFWLRQ

DUHWZRLQSXWILHOGVWRILOOHGLQ2QHLVIRUDIUHHIRUPVWULQJWKDW 7KHJHQHUDOFRQFHSWRIWKH63(&2V\VWHPDQGWKHFRPSDULVRQ

ZLOO EH YLVLEOH RQ WKH VSHFWURJUDP 7KH RWKHU ZLOO QHYHU EH

RILWZLWKWKHRWKHUFRPSXWHUEDVHGSURQXQFLDWLRQWUDLQLQJWRROV

GLVSOD\HG ,W LV XVHG WR LGHQWLI\ WKH GLIIHUHQW VRXQGV IRU WKH

ZHUHGHVFULEHGIRUWKHODVW(85263((&+>@,QWKHSUHVHQW

DXWRPDWLFIHHGEDFN

SDSHUZHSUHVHQWGHYHORSPHQWVPDGHGXULQJWKHODVWWZR\HDUV

$Q DXWRPDWLF VHJPHQWDWLRQ ZDV GHYHORSHG E\ WKH +XQJDULDQ

WKHJHQHUDOVWUXFWXUHRIWKHV\VWHPWKHVSHHFKSLFWXUHVDQGWKH

SDUWQHUVDQGEXLOWLQWRWKHVHJPHQWHUPRGXOHRIWKH(',725WR

GDWDEDVHV ZKLFK KDG WR EH FRQVWUXFWHG IRU WKH IRXU GLIIHUHQW

KHOSWKHZRUNRIWKHSDUWQHUV>@

ODQJXDJHV

*5$3+,&$/(',72502'8/(

7KHV\VWHPLVFRPSRVHGRIWZRSDUWVRQHLVDJHQHUDOODQJXDJH

7KH YLVXDO SUHVHQWDWLRQ RI WKH DFRXVWLF SDUDPHWHUV FDOOHG

LQGHSHQGHQW

VSHHFKSLFWXUHV JLYHVWKHSRVVLELOLW\WRWKHXVHUWRSURFHVVWKH

IUDPH

,1'(3(1'(17

SURJUDP

WKH

'$7$%$6(

/$1*8$*(

(',725

$1'

VSHHFK IXUWKHU RQ WKH EDVLV RI WKLV YLVXDO SUHVHQWDWLRQ $

0($685,1* 6 SKRQHPH@ L





SLFWXUHV





$ Q > SKRQHPH@



 ZKHUH

7KXV WKH\ XVH WKH KXPDQ YLVXDO DQG DXGLWRU\ IHHGEDFN GXULQJ WKHLU VSHHFK OHDUQLQJ LQ DGGLWLRQ WR WKH SURSULRFHSWLYH DQG



$ = {D  D D

WDFWLOHVHQVDWLRQ

L

2QH SDUW RI WKH WUDLQLQJ LV WKH FRPSDULVRQ RI WKH DFWXDO SURQXQFLDWLRQRIWKHLWHPVZLWKWKHVWRUHGUHIHUHQFHV&KLOGUHQ

I

ILUVWPDNHWKHFRPSDULVRQYLVXDOO\$QDXWRPDWLFIHHGEDFNKDV EHHQ GHYHORSHG WR KHOS FRPSDULVRQ EDVHG RQ GLVWDQFH FDOFXODWLRQ EHWZHHQ WKH VSHFWUDO FRPSRQHQWV RI WKH W\SLFDO DXGLWRU\VSHFWUXPVRUVSHFWURJUDPVDQGWKHDFWXDOVSHHFK 2QH LPSRUWDQW IHDWXUH RI WKLV V\VWHP KDV EHHQ WR ILQG JRRG GLVWDQFHPHDVXUHVWKDWFDQKHOSWKHVHFRPSDULVRQVEHWZHHQWKH VSHFWUDO FRPSRQHQWV RI WKH UHIHUHQFH DQG WKH DFWXDO VSHHFK

=

= 1=

,

I

}



QXPEHURIWKHFULWLFDOEDQGILOWHUV

WRWDOQXPEHURIIUDPHVLQWKHSKRQHPH

QXPEHURIVSHDNHUVLQFKLOGUHQVSHHFKGDWDEDVH



7KLV GLVWDQFH PHDVXUH PXVW UHIOHFW WKH MXGJHPHQW RI WKH

7KH FDOFXODWHG W\SLFDO DYHUDJH VSHFWUXP SDWWHUQV RI VLELODQWV

OLVWHQHUVLHWKHMXGJHPHQWRIPRVWRIWKHSHRSOHLQWKHFKLOG¶V

DUHSUHVHQWHGRQWKH)LJ7KH+XQJDULDQSDUWQHUPDGHWKHVH

HGXFDWLRQDOSXEOLFDQGFXOWXUDOOLIH>@

FDOFXODWLRQV DQG LQFRUSRUDWHG WKH VSHFWUXP SDWWHUQV LQWR WKH

7KHUROHRIWKHDXWRPDWLFIHHGEDFNLVWKHPRVWLPSRUWDQWLQWKH

6XSSRUW IDFLOLW\ $OO RI WKHVH VSHFWUXP SDWWHUQV DUH SUHVHQWHG

WHDFKLQJ RI VHYHUHO\ KHDULQJ LPSDLUHG FKLOGUHQ 7KLV IHHGEDFN

IRUWKHFKLOGUHQXVLQJRQHW\SH RIVSHHFKSLFWXUHVKRZQDVLW

SURYLGHVDJRRGRSSRUWXQLW\IRUSUDFWLFHZLWKQRWHDFKHUEXWWKH

LVSUHVHQWHGLQ FDVH RI WKH YRZHO¶X¶ LQWKH )LJ  'XULQJ WKH

V\VWHPDORQH

WUDLQLQJ WKH DFWXDO VSHFWUXP OLQHV PXVW IDOO ZLWKLQ WKH WZR VSHFWUXPOLQHVRQWKHVSHHFKSLFWXUH



&ROOHFWLRQRIWKHFRUSXVRIFKLOGUHQ VVSHHFK IRUVWDWLVWLFDOH[DPLQDWLRQ

$OO SDUWQHUV FROOHFWHG D FRUSXV RI FKLOGUHQ V VSHHFK ZKLFK FRQWDLQV H[DPSOHV RI WKH SURQXQFLDWLRQ RI IULFDWLYHV DIIULFDWHV DQGYRZHOVWKDWKDYHEHHQHYDOXDWHGDVJRRGDFFHSWDEOHRU ZURQJ :H UHIHU WR WKLV GDWDEDVH DV µWKH FKLOG VSHHFK

 

Eurospeech 2001 - Scandinavia



 G% 



 

+XQJDULDQ

(QJOLVK



6ORYHQLDQ





]>]@

%DUN

]V>=@

V]>V@

V>6@

>]@

>=@

>V@

>6@

]>]@

å>=@

V>V@

ã>6@

  )LJXUH$QH[DPSOHRIWKHVSHFWUXPW\SHGVSHHFKSLFWXUHRI

6ZHGLVK



WM>&@

M>M@

V>V@

VM>6"@

YRZHOVRXQGµX¶ &DOOLQJFDUG &XIIOLQNV  &DUGWRVKRZWKHUHVXOWRIWKHDXWRPDWLFIHHGEDFNZLWKWKH

)LJXUH7\SLFDODXGLWRU\VSHFWUDRIIULFDWLYHVRIIRXU

IORZHU,WLVEORRPVLQWKHFDVHRIJRRGSURQXQFLDWLRQ 

ODQJXDJHV  7KHW\SLFDODXGLWRU\VSHFWURJUDPDQGWKHSRVLWLRQLQJ

7KH W\SLFDO DYHUDJH VSHFWURJUDPV XSSHU SDUW RI )LJ   KHOS WKH FRQVWUXFWLRQ DQG WKH SRVLWLRQLQJ RI WKH EDFNJURXQG

RIWKHEDFNJURXQGSLFWXUHV

SLFWXUHV RI HDFK VRXQG ORZHU SDUW RI )LJ   7KH W\SLFDO 2QH SDUW RI WKH WUDLQLQJ LV WKH FRPSDULVRQ RI WKH DFWXDO

DYHUDJH DXGLWRU\ VSHFWURJUDPV KDYH EHHQ FDOFXODWHG E\

SURQXQFLDWLRQ RI WKH LWHPV ZLWK WKH VWRUHG UHIHUHQFHV 7KH

DYHUDJLQJ WKH HQHUJ\ YDOXHV RI VSHFWURJUDPV RI WKH SKRQHPH

DXGLWRU\ VSHFWURJUDP RI WKH VWRUHG VSHHFK RI WKH UHIHUHQFH

H[DPSOHVLQZRUGVRIWKHJRRGDQGDFFHSWDEOHH[DPSOHVIURP

VSHDNHU WKH UHIHUHQFH DXGLWRU\ VSHFWURJUDP  LV SUHVHQWHG LQ

WKH FKLOG VSHHFK GDWDEDVH >@ 7KLV W\SLFDO DYHUDJH DXGLWRU\

WKHXSSHUSDUWRIWKHVFUHHQDQGWKHVSHFWURJUDPRIWKHDFWXDO

VSHFWURJUDP

SURQXQFLDWLRQ RQ WKH ORZHU SDUW RI WKH VFUHHQ %DFNJURXQG

VSHFWURJUDP EXW WKH VSHFWURJUDP RI D JRRG UHIHUHQFH VSHDNHU

SLFWXUHVKDYHEHHQFRQVWUXFWHGDQGSUHVHQWHGWRJHWKHUZLWKWKH

LVFORVHWRLW

LV

QRW

LGHQWLFDO WR

WKH

UHIHUHQFH

DXGLWRU\

UHIHUHQFH DXGLWRU\ VSHFWURJUDP WR HPSKDVL]H WKH LPSRUWDQW SDUWVRILW 

 )LJXUH7KHW\SLFDODYHUDJHDXGLWRU\VSHFWURJUDPVXSSHU DQGWKHSRVLWLRQRIWKHEDFNJURXQGSLFWXUHVORZHU IRU(QJOLVKVLELODQWV 7H[WRIWKHVSHHFKDQGSLFWXUHGDWDEDVH

Eurospeech 2001 - Scandinavia

7KHGHYHORSHGV\VWHPLVQRWRQO\DPHDVXULQJWRROEXWSURYLGHVD

:KHQ WKH V\VWHP ZDV EHLQJ GHYHORSHG D GHWDLOHG HYDOXDWLRQ

PHWKRGIRUWHDFKLQJWKHFKLOGXVLQJWKLVQHZPXOWLPRGDOSURJUDP

H[DPLQDWLRQ ZDV RUJDQL]HG LQ WKH +DUG RI +HDULQJ 6FKRRO RI

$OOSDUWQHUVKDYHEHHQLQYROYHGLQWKHFRQVWUXFWLRQRIWKHWH[WDQG

%XGDSHVW *URXSV RI FKLOGUHQ ZLWK GLIIHUHQW SUREOHPV ZHUH

SLFWXUHV RI WKH GDWDEDVH LHWKH GHYHORSPHQW RI H[HUFLVHV IRU WKH

WDXJKW ZLWK WKH KHOS RI WKH QHZ V\VWHP DQG D FRQWURO JURXS

9RZHO DQG 6LELODQW 6XSSRUW IURP VRXQG SUHSDUDWLRQ  WKURXJK WR

WUDLQHGE\WKHWUDGLWLRQDOZD\ZDVFRPSDUHG7KHLQWHOOLJLELOLW\

IOXHQF\  7KH H[HUFLVHV DUH DSSURSULDWH WR WKH SKRQRORJLFDO

RIWKHVSHHFKRIWKHVHFKLOGUHQDIWHUDKDOI\HDURIWUDLQLQJZDV

VWUXFWXUH RI WKHLU ODQJXDJHV DQG WUDGLWLRQDO WHDFKLQJ SUDFWLFHV

FRPSDUHG,QWKRVHJURXSVZKLFKXVHGWKH6XSSRUWPDWHULDOWKH

KDYHEHHQWDNHQLQWRDFFRXQWLQWKHLUFRQVWUXFWLRQ

LQWHOOLJLELOLW\ZDVWLPHVEHWWHURQDYHUDJHDIWHUWKHWUDLQLQJ

7KHVWUXFWXUHRIWKHH[HUFLVHVLVWKHIROORZLQJ

WKDQ EHIRUH ,Q WKH FRQWURO JURXS WKLV UDWH ZDV RQO\  $



GHWDLOHGSUHVHQWDWLRQRIWKLVHYDOXDWLRQZLOOEHJLYHQGXULQJWKH



FRQIHUHQFH

6281'35(3$5$7,21



ORXGQHVV

$FNQRZOHGJHPHQW

VSHFWUXP 7KLVUHVHDUFKKDVEHHQVXSSRUWHGE\WKH(XURSHDQ8QLRQLQWKH

SLWFK

IUDPHZRUN RI WKH ,QFR&RSHUQLFXV 3URJUDP &RQWUDFW QR

UK\WKP

 DQGE\WKH+XQJDULDQ6FLHQWLILF5HVHDUFK)RXQGDWLRQ

 92:(/6 



VRXQGGHYHORSPHQW



DUWLFXODWLRQ



LVRODWHGSURQXQFLDWLRQ

 

>@

VRXQGVHTXHQFHV

RQHV\OODEOH



SRO\V\OODELF



IOXHQF\ VHQWHQFHV



7UDLQLQJ

DERXWLWHPVSKRQHPH



ZRUGSDLUV

>@ SKRQHPH SKRQHPH

&KLOGUHQ

&VDWiUL ) %DNFVL =VDQG 9LFVL . $ +XQJDULDQ &KLOG

9LFVL . &VDWiUL ) %DNFVL =V DQG 7DQWRV $ 'LVWDQFH

$XGLRYLVXDO

$UWLFXODWLRQ

7UDLQLQJ

3URFHHGLQJV

RI

(85263((&+¶SS 

DUWLFXODWLRQ

LVRODWHGSURQXQFLDWLRQ



VRXQGVHTXHQFHV

>@

WUDLQLQJLQZRUGV



SUDFWLVHGVRXQGLVLQWKHPLGGOH

ZRUGSDLUV

6DPSD

/DEHOOLQJ

RI

.DFLF=9LFVL.5RDFK36LQND,gVWHU$2JQHU 0 DQG %DUF]LND\ 3 'HYHORSPHQW 3HUVSHFWLYHV RI 0XOWLPHGLD 0XOWLOLQJXDO 7HDFKLQJ DQG 7UDLQLQJ 6\VWHP

SKRQHPH SKRQHPH

8VLQJ

5HVRXUFHV(GXFDWLRQ*UDQDGD6SDLQ >@

SUDFWLVHGVRXQGLVDWWKHHQGRIWKHZRUG IOXHQF\ VHQWHQFHV

7HFKQLTXH

3KRQHPHV )LUVW ,QWHUQDWLRQDO &RQIHUHQFH RQ /DQJXDJH

DERXWLWHPVSKRQHPH

SUDFWLVHGVRXQGLVDWWKHEHJLQQLQJ

9LFVL.9LJ$/,$6/DQJXDJH,QGHSHQGHQWDXWRPDWLF 6HJPHQWDWLRQ

DERXWLWHPVSKRQHPH





+DQGLFDSSHG

RI(85263((&+¶SS  >@

VRXQGGHYHORSPHQW





6SHHFK

6FRUH (YDOXDWLRQ RI WKH 9LVXDOLVHG 6SHHFK 6SHFWUD DW





IRU

'DWDEDVHIRU6SHHFK3URFHVVLQJ$SSOLFDWLRQV3URFHHGLQJV

6,%,/$176



6\VWHP

3URFHHGLQJVRI(85263((&+¶SS 





9LFVL . 5RDFK 3 gVWHU $ .DFLF = %DUF]LND\ 3 DQG 6LQND , $ 0XOWLPHGLD 0XOWLOLQJXDO 7HDFKLQJ DQG

DERXWLWHPVSKRQHPH

WUDLQLQJLQZRUGV

5HIHUHQFHV

IRU6SHHFKKDQGLFDSSHG&KLOGUHQ >@



gVWHU $ 9LFVL . 5RDFK 3 .DFLF = DQG %DUF]LND\ 3 $ 0XOWLPHGLD 0XOWLOLQJXDO 7HDFKLQJ DQG 7UDLQLQJ

,1721$7,21

6\VWHP IRU 6SHHFK DQG +HDULQJ ,PSDLUHG &KLOGUHQ 



63(&2)21(7,.¶SS 



>@



-/

:DOODFH

DW

DO



$SSOLFDWLRQV

RI

6SHHFK

5HFRJQLWLRQ LQ WKH 3ULPDU\ 6FKRRO &ODVVURRP (6&$ ±

7KH GHYHORSHUV ZHUH LQ GLUHFW FRPPXQLFDWLRQ ZLWK VSHHFK WKHUDSLVWV WKURXJKRXW WKH GHYHORSPHQW DQG WKH DGYLFH RI WKH

6WLOO:RUNVKRS3URFHHGLQJV0DUKROPHQ >@

WKHUDSLVWV ZDV WDNHQ LQWR FRQVLGHUDWLRQ LQ WKH FRQVWUXFWLRQ RI H[HUFLVHV DQG HVSHFLDOO\ LQ WKH FRQVWUXFWLRQ RI WKH WH[WV IRU WKH GLIIHUHQWODQJXDJHV



(YDOXDWLRQ

7KH 63(&2 JURXS KDV FRQVWUXFWHG DQ HGXFDWLRQDO PXOWLPHGLD V\VWHP IRU IRXU ODQJXDJHV ZKLFK LV VXLWDEOH IRU XVH ZLWK LQGLYLGXDOV SDLUV DQG VPDOO JURXSV ,W IROORZV PRVW RI WKH FULWHULDRI(YDOX7HFK>@6RIWZDUH(YDOXDWLRQ2UJDQLVDWLRQRI 65(%LQ86 'XULQJWKHGHYHORSPHQWDOOSDUWQHUVNHSWLQFORVH FRQWDFW ZLWK VSHHFK WKHUDSLVWV 7KH\ KDYH EHHQ SURYLGHG ZLWK WKH6XSSRUWPDWHULDOUHFHQWO\DQGWKHLURSLQLRQKDVEHHQVRXJKW IURP WLPH WR WLPH 4XHVWLRQHUV ZHUH ILOOHG RXW E\ WKHUDSLVWV IURPGLIIHUHQWHGXFDWLRQDOILHOGVWRUHYHDODQ\SUREOHPVRIWKH XVDJH DQG VKRZ WKH QHHG IRU DQ\ FRUUHFWLRQV 7KLV KDV LPSURYHGWKHV\VWHP

(YDOX7HFK6RIWZDUH(YDOXDWLRQ2UJDQLVDWLRQRI65(%LQ 86 KWWSZZZHYDOXWHFKVUHERUJ