Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode -> summary(?)

From: "Shanmugham, Saravanan" <[email-masked]>
Date: Thu, 7 Jul 2005 09:52:08 -0700

I don't think generating multiple START-OF-SPEECH events is a solution. 
We still haven't addressed, what constitutes a barge-in, for the
optimised case. That should also be the single point when a single
START-OF-SPEECH(or whathever else you want to name it) should be
generated.

That point, in my opinion should be
   1. For "dtmf-recog" resources should be the beginning of a DTMF key
press.
   2. For "speech-recog" resources should be the beginning of a DTMF key
press or the beginning of speech. This is should be irrespective of what
type of grammar is being used. Coz even numbers only grammars can still
be spoken and hence cannot be assumed to be a cue for DTMF only
recognition.
   3. For "speech-only-recog" resources(which are not defined today) the
time to barge-in is the beginning of speech. I don't see a need for such
a resource today. But I am mentioning this for completeness.

Sarvi

     -----Original Message-----
     From: [email-masked] 
     [mailto:[email-masked]] On Behalf Of Eric Burger
     Sent: Thursday, July 07, 2005 5:32 AM
     To: [email-masked]
     Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode 
     -> summary(?)
     
     Can we declare consensus? 
     
     > -----Original Message-----
     > From: [email-masked] 
     [mailto:[email-masked]]
     > Sent: Wednesday, July 06, 2005 10:45 AM
     > To: Dave Burke
     > Cc: [email-masked]; Shanmugham, Saravanan; Klaus Reifenrath
     > Subject: Re: [Speechsc] START-OF-SPEECH in DTMF-only mode ->
     > summary(?)
     > 
     > I think we're getting close. I though about snipping out 
     some pieces 
     > to cut down the text, but I realized the context is 
     still needed. See 
     > inline.
     > 
     > On Jul 6, 2005, at 10:29 AM, Dave Burke wrote:
     > 
     > > Inline.
     > >
     > > Dave
     > >
     > > ----- Original Message ----- From: "David R Oran" 
     <[email-masked]>
     > > To: "Dave Burke" <[email-masked]>
     > > Cc: <[email-masked]>; "Shanmugham, Saravanan" 
     <[email-masked]>; 
     > > "Klaus Reifenrath" <[email-masked]>
     > > Sent: Wednesday, July 06, 2005 1:07 PM
     > > Subject: Re: [Speechsc] START-OF-SPEECH in DTMF-only 
     mode -> summary
     > > (?)
     > >
     > >
     > >
     > >>
     > >> On Jul 6, 2005, at 5:31 AM, Dave Burke wrote:
     > >>
     > >>
     > >>> + Attempting to summarise:
     > >>>
     > >>> 1. START-OF-SPEECH is useful for the client to know when
     > to stop   
     > >>> playing prompts the in non-optimised case 2. 
     START-OF-SPEECH is 
     > >>> useful for the client to calculate the bargin  time 
     (e.g. VoiceXML 
     > >>> 2.1 <mark>)
     > >>> 3. In the optimised case, a bargin automatically 
     stops prompt   
     > >>> playing (assuming prompts barginable) 4. Because of 
     the previous 
     > >>> point, the question of what input type  caused 
     bargin is different 
     > >>> and less important to
     > what input
     > >>> type(s)  the recogniser is listening for
     > >>>
     > >> I'm not sure I follow point 4. Could you elaborate?
     > >>
     > >
     > > DB> Adding a parameter to the START-OF-SPEECH event would
     > certainly
     > > allow the client to ignore (i.e. let prompts continue 
     playing) the 
     > > event if the event type is not of interest (e.g. the 
     client would 
     > > ignore speech start events when it is interested only 
     in  DTMF start 
     > > events). This _only_ works for the non-optimised case, 
     however. For 
     > > the optimised case, assuming START-OF-SPEECH
     > coincides
     > > with the bargin signal to the speechsynth, prompts 
     will stop playing 
     > > for inputs that the client might not be interested
     > in (e.g.  
     > > a speech input will stop prompts playing even if the client
     > is only
     > > interested in DTMF).
     > >
     > OK, now I get it. There's a need for the client to both 
     handle the 
     > non-optimized case itself, and influence or at least 
     have a clue what 
     > the server is going to do in the optimized case.
     > 
     > >
     > >>
     > >>
     > >>> 5. Currently, MRCPv2 has no way of indicating what 
     input type(s) a 
     > >>> recogniser is listening for
     > >>>
     > >>>
     > >> Do you mean exactly this, or do you mean "for the client to 
     > >> indicate  to the resource what input types it should 
     look for"?
     > >>
     > >
     > > DB> Yes exactly - apologies for not being clear.
     > >
     > >
     > >>
     > >>
     > >>> + Why implement 5?
     > >>>
     > >>> i. Noisy case: Need DTMF-only recognition (and may 
     only have a
     > >>> speechrecog)
     > >>>
     > >> I'm having difficulty following the logic of why 
     noise would   
     > >> necessarily trigger START-OF-SPEECH if you were listening for 
     > >> speech  but not DTMF. I suppose you can use a more forgiving 
     > >> discriminator if  all you need to tell is if you're 
     getting DTMF, 
     > >> but I've had a number  of real-world cases where wind 
     noise was 
     > >> detected as DTMF, and  there's always the ambiguity 
     when you have 
     > >> Captain Crunch on the  phone. In either case in the 
     non-optimized 
     > >> case it's the client who  gets to decide whether an 
     event should be 
     > >> interpreted as barge-in or  not, so it seems an 
     aesthetic protocol 
     > >> design decision whether the  client tells the server 
     ahead of time 
     > >> what circumstances to generate  the START-OF-SPEECH 
     event for, or 
     > >> whether the event gets generated  and the client 
     decides based on 
     > >> what's in the event whether it should  be
     > treated
     > >> as barge- in.
     > >>
     > >> I suppose one could make the argument that because 
     the spec implies  
     > >> that the event can only be generated once per request 
     that if a 
     > >> DTMF/ speech capable recognizer first hears
     > enough noise
     > >> to think it's  hearing speech and later hears DTMF, 
     the client will 
     > >> declare barge-in  when the event comes and not when he DTMF 
     > >> actually gets heard.
     > >>
     > >> If that's deemed a problem, we can still handle that 
     in the design  
     > >> where the server just reports what it's hearing by allowing 
     > >> multiple  events to be generated during a single request.
     > >>
     > >> Between the approach just outlined above, and an 
     approach where the 
     > >> client provides a filter for whether to generate the 
     event or not, 
     > >> I  have a mild preference (based on aesthetics rather 
     than some 
     > >> hard  engineering tradeoff) for the approach where
     > the server
     > >> just reports  what it's hearing.
     > >>
     > >>
     > >> Having had some useful exchanges on this topic, it 
     also is becoming 
     > >> apparent to me that this event is poorly named, and 
     we should  
     > >> consider renaming it to "INTERESTING-INPUT-HEARD" or 
     something akin  
     > >> to that, because as others have pointed out, a 
     DTMF-only recognizer  
     > >> will never detect "start of speech".
     > >>
     > >> Another consideration to fold into the design choice is   
     > >> extensibility. Bear with me through a little 
     gedankenexperiment.
     > >>
     > >> Suppose we want to define a new recognizer type, 
     which I'll call 
     > >> the "name that tune" recognizer. The client plays 
     music to the 
     > >> server and  the server recognizes musical notes. The 
     grammar is a 
     > >> standard  musical notation, augmented with a semantic 
     > >> interpretation that  transforms the notes into the 
     title of the 
     > >> tune and provides that as  an answer.
     > >>
     > >> First, there's no speech involved (or is there...hang 
     on a minute). 
     > >> Second, in order to accommodate the "name that tune"
     > >> recognizer, we'd have to extend both the client and 
     the server to 
     > >> undetstand a  directive as to whether to recognize 
     music or now, 
     > >> inaddition to what  the server already knows what to 
     do based on 
     > >> the grammar. If you  follow my logic above, whether 
     or not we do 
     > >> that, we have to extend  "start-of-speech" to say 
     "I'm hearing 
     > >> music". So far fairly  straightforward, but let me 
     now throw in the 
     > >> pathological twist.
     > >>
     > >> Suppose what I feed to a  combined music/speech 
     recognizer is a 
     > >> work  in sprechstimme (spoken music), like the 
     "Geographical Fugue" 
     > >> (aside:  this is a wonderful piece of music I highly 
     recommend to 
     > >> anyone  interested in small ensemble singing). In 
     this case, the 
     > >> tune could  be named by either doing speech or music 
     recognition. 
     > >> Why is there  any need for the client to constrain 
     the server as to 
     > >> which it tries  to do when it's
     > already
     > >> told the server what it wants through the  grammar?
     > >>
     > >> A few other comments below
     > >>
     > >>
     > >>> ii. Flexibility: Want speech-only recognition 
     (because a second 
     > >>> recogniser is doing hotword on DTMF)
     > >>>
     > >>>
     > >> I don't see how flexibility is affected by this deisgn
     > choice. If   
     > >> that's what you want, feed the speech-only recognizer 
     a grammar   
     > >> without any DTMF rules.
     > >>
     > >>
     > >>> + How to implement 5?
     > >>>
     > >>> a. Implicitly:
     > >>>    - dtmfrecog: always DTMF-only recognition
     > >>>    - speechrecog: depends on active grammar type
     > >>>        > if a dtmf grammar is active then DTMF input is "on"
     > >>>        > if a speech grammar is active then speech 
     input is "on"
     > >>>
     > >>> b. Explicitly:
     > >>>    - Add inputmodes header to RECOGNIZE
     > >>>
     > >>> Option a is David's "do what I mean case"; option b is
     > the extra   
     > >>> dial for the client.
     > >>>
     > >>>
     > >> Actually, that's not the point I was making with "do 
     what I mean",  
     > >> but it's not essential to the discussion so let's move on.
     > >>
     > >>
     > >>> It is worth noting that VoiceXML uses option b. This 
     allows one to 
     > >>> activate both speech grammars and DTMF grammars (and
     > therefore
     > >>> be informed of any errors in the grammars at activation
     > time) but
     > >>> independently turn on whichever input mode you like e.g. 
     > perhaps   
     > >>> start with "both" then change to "dtmf".
     > >>>
     > >>>
     > >> I'm not sure the VXML precedent is relevant here, 
     because the   
     > >> application behind VXML is working a different part of the
     > problem
     > >> -  how to traverse a TUI dialog based on different 
     parts of the 
     > >> input  space. In fact, I suspect that the VXML: choice was 
     > >> conditioned more  by limitations at the time it was
     > specified than
     > >> an underlying good  design choice. Clearly having to 
     specify this 
     > >> in VXML make the job of  handling a TUI with nodes 
     like "Say or 
     > >> press 5" harder rather than  easier.
     > >>
     > >
     > > DB> The VoiceXML edge-case is pretty weird so it's not a major
     > > concern. My main concern is that the client can 
     indicate, somehow, 
     > > what the input modes are.
     > >
     > >
     > >>
     > >> Summing up, while I don't feel strongly one way or 
     the other, I 
     > >> have  a preference for handling this as follows:
     > >>
     > >> a) Rename "START-OF-SPEECH" to 
     "INTERESTING-INPUT-RECEIVED" or   
     > >> something equivalent.
     > >> b) Include a parameter in the event saying what was 
     interesting 
     > >> about  the input you received, with a registry of values which
     > >> includes:
     > >>     - signal above noise floor
     > >>     - speech
     > >>     - dtmf
     > >>     - (possibly) music
     > >> c) allow the event to be generated multiple times 
     during a request
     > >>
     > >>
     > >
     > > DB> I like these suggestions (START-OF-INPUT?). 
     However, I don't
     > > see how the problem of the optimised case is not 
     solved by them. I 
     > > think the optimised case is fine if we have the 
     following rules:
     > >
     > Yes, I hadn't thought through the optimized case as 
     thoroughly as you. 
     > Your suggested method name is fine by me as well
     > 
     > > 1. START-OF-SPEECH (and optimised bargin) is only generated
     > for the
     > > input type that is being listened for 2. A speechrecog 
     listens for 
     > > DTMF if DTMF grammars are active, speech if speech 
     grammars are 
     > > active, or speech and DTMF if both grammar types are active.
     > >
     > Works for me.
     > 
     > >
     > >> Note that all of the above I'm saying with my technical
     > hat on and
     > >> my chair hat off.
     > >> Putting my chair hat on for a moment, we really need 
     to get this 
     > >> spec  to last call, so at some point Eric or I is going to
     > declare
     > >> rough  consensus so we can move on.
     > >>
     > >
     > > DB> Agreed!
     > >
     > >
     > >>
     > >> Dave Oran.
     > >>
     > >>> Dave
     > >>>
     > >>>
     > >>>
     > >>>
     > >>>
     > >>>
     > >>>
     > >>>
     > >>>
     > >>>
     > >>>
     > >>>
     > >>> Sarvi makes a good point that adding the reason why the
     > START-OF-
     > >>> SPEECH occurred does not fix the optimised bargin case.
     > >>>
     > >>> dtmfrecog - listens for DTMF only
     > >>> speechrecog - listens for DTMF only, or speech only, 
     or speech & 
     > >>> DTMF
     > >>>
     > >>>
     > >>>
     > >>>
     > >>>
     > >>>
     > >>>
     > >>>
     > >>> ----- Original Message ----- From: "Shanmugham, Saravanan"  
     > >>> <[email-masked]>
     > >>> To: "David R Oran" <[email-masked]>; "Dave Burke"  
     > >>> <[email-masked]>
     > >>> Cc: <[email-masked]>; "Klaus Reifenrath"  
     > >>> <[email-masked]>
     > >>> Sent: Tuesday, July 05, 2005 8:51 PM
     > >>> Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode
     > >>>
     > >>>
     > >>> inline.
     > >>>
     > >>>     -----Original Message-----
     > >>>     From: David R Oran [mailto:[email-masked]]
     > >>>     Sent: Tuesday, July 05, 2005 11:08 AM
     > >>>     To: Dave Burke
     > >>>     Cc: Shanmugham, Saravanan; Klaus Reifenrath; 
     [email-masked]
     > >>>     Subject: Re: [Speechsc] START-OF-SPEECH in DTMF-only mode
     > >>>
     > >>>
     > >>>     On Jul 5, 2005, at 12:54 PM, Dave Burke wrote:
     > >>>
     > >>>     > Inline.
     > >>>     >
     > >>>     > Dave
     > >>>     >
     > >>>     > ----- Original Message ----- From: 
     "Shanmugham, Saravanan"
     > >>>     > <[email-masked]>
     > >>>     > To: "David R Oran" <[email-masked]>; "Klaus Reifenrath"
     > >>>     > <[email-masked]>
     > >>>     > Cc: <[email-masked]>
     > >>>     > Sent: Tuesday, July 05, 2005 5:22 PM
     > >>>     > Subject: RE: [Speechsc] START-OF-SPEECH in 
     DTMF-only mode
     > >>>     >
     > >>>     >
     > >>>     > I agree with Dave's analysis. The purpose of this event
     > >>>     was barge-in.
     > >>>     > And barge-in should happen for both DTMF and speech.
     > >>>     >
     > >>>     > Is there a case where you think it should not behave
     > >>>     this way. If soe,
     > >>>     > please provide a scenario where you think
     > >>>     >    1. Barge-in should happen for DTMF and not voice or
     > >>>     vice-versa.
     > >>>     >
     > >>>     > DB> You want to do a DTMF recognition only 
     because it is 
     > >>> noisy.
     > >>>     > While waiting for DTMF input, the speechrecog resource
     > >>>     (or advanced
     > >>>     > dtmfrecog) generates a START-OF-SPEECH because it heard
     > >>>     some speech.
     > >>>     > The client does not want to stop prompt playing unless
     > >>>     DTMF was heard
     > >>>     > but it can't tell by the START-OF-SPEECH whether speech
     > >>>     or DTMF was
     > >>>     > heard. Similarly vice versa.
     > >>>     >
     > >>>     It's an interesting design question what part of the
     > >>>     policy resides at the client and what at the server, and
     > >>>     who makes the "final decision" about whether what was
     > >>>     heard was relevant to the control channel. Right now we
     > >>>     (IMO) have a weird partitioning in many cases where the
     > >>>     client basically says "do what I mean", but there are no
     > >>>     constraints of what the server actually does, and no
     > >>>     normalized basis for the client to figure out what to set
     > >>>     various magic numbers to (e.g. sensitivity).
     > >>>
     > >>>     In this case the only thing the client needs to decide is
     > >>>     whether to kill the prompt because the server thinks
     > >>>     something that would interfere with the feedback
     > >>>     ear/mouth/finger control happened. What this 
     says to me is
     > >>>     that it isn't necessarily a good idea for the client to
     > >>>     have more knobs to control the server 
     (especially if those
     > >>>     knows are just more value/policy input ungrounded in any
     > >>>     physics/ acoustics). On the other hand, having the server
     > >>>     tell the client more about what it thinks is going on is
     > >>>     probably valuable.
     > >>>
     > >>>     So, Coming to the point after this long rambling
     > >>>     introduction, I think it would in fact be useful for the
     > >>>     START-Of-SPEECH event to indicate some extra information,
     > >>>     for example:
     > >>>     a) I got something enough above the noise floor 
     to qualify
     > >>>     for exceeding the "Sensisitvity" parameter you sent in on
     > >>>     the request but I really can't tell what it is 
     (could be a
     > >>>     hippopatmus fart, or a siren in the background, 
     or captain
     > >>>     crunch trying to whistle DTMF).
     > >>>     b) I think I'm hearing speech
     > >>>     c) I think I'm hearing DTMF
     > >>>
     > >>> Though I agree with your former part of your 
     response. I am not 
     > >>> sure I agree with your proposed solution.
     > >>> The way I see this problem is that, it is more of what
     > constitues a
     > >>> barge-in event. This boils down to whether it is 
     speech, DTMF or 
     > >>> both.
     > >>> This is inturn boils down to what type of recognizer
     > resource we are
     > >>> using, dtmf-recog, speech-recog, and speech-only-recog(we
     > don't have
     > >>> this and I don't think we should add it, but think 
     of this as a 
     > >>> place holder that explains the concept).
     > >>>
     > >>> A client knowing what type of barge-in happenned, does
     > not impact
     > >>> the
     > >>> barge-in operation itself as it may be too late(for 
     the optimized 
     > >>> barge-in case). It may have other use cases, and if we
     > can identify
     > >>> them, I don't mind adding support for the 
     START-OF-SPEECH event to 
     > >>> say what type of barge-in happenned. But that itself does not
     > solve the
     > >>> original problem raised. Refer to my previous response.
     > >>>
     > >>> The solution lies in defining what what is a 
     barge-in event.  
     > >>> That  boils
     > >>> down to what type of recognition is happenning,
     > dtmf-only, speech-
     > >>> dtmf
     > >>> or speech-only. We do not support speech-only as a 
     resource today, 
     > >>> the question is do we need a header to force it.
     > >>>
     > >>> Sarvi
     > >>>
     > >>>
     > >>>     >    2. You would benefit from the client knowing 
     > what caused
     > >>> the
     > >>>     > barge-in, DTMF Vs speech.
     > >>>     >
     > >>>     > DB> See previous comment. And previous e-mail: 
     either add an
     > >>>     > inputmodes header (taking value speech, dtmf, both) to
     > >>>     the RECOGNIZE
     > >>>     > request or add a header to the START-OF-SPEECH event
     > >>>     indicating DTMF
     > >>>     > or speech.
     > >>>     >
     > >>>     I'm leaning in your direction on this latter point - as
     > >>>     should be evident from what I wrote above.
     > >>>
     > >>>     > Sarvi
     > >>>     >
     > >>>     >     -----Original Message-----
     > >>>     >     From: [email-masked]
     > >>>     >     [mailto:[email-masked]] On Behalf Of 
     > David R
     > >>> Oran
     > >>>     >     Sent: Tuesday, July 05, 2005 5:24 AM
     > >>>     >     To: Klaus Reifenrath
     > >>>     >     Cc: [email-masked]'
     > >>>     >     Subject: Re: [Speechsc] START-OF-SPEECH in 
     > DTMF-only mode
     > >>>     >
     > >>>     >
     > >>>     >     On Jul 5, 2005, at 3:46 AM, Reifenrath, 
     Klaus wrote:
     > >>>     >
     > >>>     >     > The current spec is not clear when 
     > START-OF-SPEECH need
     > >>>     >     to be send in
     > >>>     >     > the following scenarios:
     > >>>     >     > A) The client requested a DTMF Recognizer. Is the
     > >>>     >     START-OF-SPEECH
     > >>>     >     > event send to the client also if speech 
     was detected?
     > >>>     >     I suspect so, since one of the prime purposes 
     > is to enable
     > >>>     >     client- mediated barge-in handling. However, if the
     > >>>     >     recognizer is in fact only capable of 
     recognizing DTMF
     > >>>     >     then it may in fact not report anythin 
     unless it's using
     > >>>     >     some primitive thresholding machinery, like a SN  
     > >>> threshold.
     > >>>     >     > B) The client requested a Speech 
     Recognizer, but only
     > >>>     >     activated DTMF
     > >>>     >     > grammars. Is the START-OF-SPEECH event 
     send to the
     > >>>     >     client also if
     > >>>     >     > speech was detected?
     > >>>     >     Again, I'd say yes, for the same reason as above.
     > >>>     >     > I think in both cases START-OF-SPEECH should only
     > >>>     be send after
     > >>>     >     > detecting a DTMF digit (see Figure 12 of VoiceXML
     > >>>     2.0: http://
     > >>>     >     > www.w3.org/TR/voicexml20/#dmlATiming).
     > >>>     >     We seem to have reached different 
     conclusions. I'd be
     > >>>     >     interested in why you think my analysis 
     above is wrong.
     > >>>     >
     > >>>     >     Dave.
     > >>>     >
     > >>>     >     > Klaus
     > >>>     >     >
     > >>>     >     > _______________________________________________
     > >>>     >     > Speechsc mailing list
     > >>>     >     > [email-masked]
     > >>>     >     > https://www1.ietf.org/mailman/listinfo/speechsc
     > >>>     >     >
     > >>>     >
     > >>>     >     _______________________________________________
     > >>>     >     Speechsc mailing list
     > >>>     >     [email-masked]
     > >>>     >     https://www1.ietf.org/mailman/listinfo/speechsc
     > >>>     >
     > >>>     >
     > >>>     > _______________________________________________
     > >>>     > Speechsc mailing list
     > >>>     > [email-masked]
     > >>>     > https://www1.ietf.org/mailman/listinfo/speechsc
     > >>>     >
     > >>>
     > >>>
     > >>> _______________________________________________
     > >>> Speechsc mailing list
     > >>> [email-masked]
     > >>> https://www1.ietf.org/mailman/listinfo/speechsc
     > >>>
     > >>>
     > >>
     > >> _______________________________________________
     > >> Speechsc mailing list
     > >> [email-masked]
     > >> https://www1.ietf.org/mailman/listinfo/speechsc
     > >
     > 
     > _______________________________________________
     > Speechsc mailing list
     > [email-masked]
     > https://www1.ietf.org/mailman/listinfo/speechsc
     > 

     _______________________________________________
     Speechsc mailing list
     [email-masked]
     https://www1.ietf.org/mailman/listinfo/speechsc

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: "Dave Burke" <[email-masked]>
Date: Fri, 8 Jul 2005 11:59:44 +0100

I agree with your opinion on what constitutes a barge-in. Based on Klaus' 
e-mail, I am more concerned that using the grammar type to decide what 
constitutes a barge-in is going to make VoiceXML implementations difficult.

It is easy to map VoiceXML application selected inputmodes to MRCP 
resources:
a. inputmodes="dtmf" -> Use a dtmfrecog
b. inputmodes="speech" -> Use a "speech-only-recog"
c. inputmodes="both" -> Use a speechrecog (or a combination of 
"speech-only-recog" + dtmfrecog)

The only problem is (b). While not as important as being able to do 
DTMF-only recognition, I believe we DO need to support speech-only 
recongition so as (a) to avoid unnecessary limitations in VUI design, and 
(b) to facilitate VoiceXML implementations. I think we should add a header 
to RECOGNIZE so the client is able to always _explicitly_ set the 
inputmodes.

In retrospect, I don't like the idea of multiple START-OF-SPEECH events 
being generated from the same media resource. This is because one assumes 
that the START-OF-SPEECH should be of the same type as the hypothesis 
returned in the RECOGNITION-COMPETE message - most implementations, on 
hearing one input mode type disable the recogniser of the other type. I do 
like David's idea of renaming START-OF-SPEECH to something like 
START-OF-INPUT and carrying a type header because it is neater and more 
extensible.

So in summary, I propose we modify the spec to:

1. Clarify what constitutes a barge-in for a dtmfrecog and speechrecog as 
per Sarvi's e-mail (and in agreement with Klaus' for dtmfrecog).

2. Specifiy an InputModes header to RECOGNIZE (defaults to "both", can also 
be "speech" or "DTMF"). Setting to speech for a speechrecog results in the 
hypothesised "speech-only-recog". Setting to DTMF for a speechrecog is 
equivalent to using a dtmfrecog. Edge cases: Setting to speech for a 
dtmfrecog will result in a noinput as will setting to DTMF for a speechrecog 
which does not support DTMF.

3. Change START-OF-SPEECH to START-OF-INPUT. Clarify it occurs once and 
coincides with barge-in.

4. Add a header of InputType to START-OF-INPUT. Current specified values are 
"dtmf" or "speech".

Dave

----- Original Message ----- 
From: "Shanmugham, Saravanan" <[email-masked]>
To: "Eric Burger" <[email-masked]>; <[email-masked]>
Sent: Thursday, July 07, 2005 5:52 PM
Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode -> summary(?)

Sarvi

     -----Original Message-----
     From: [email-masked]
     [mailto:[email-masked]] On Behalf Of Eric Burger
     Sent: Thursday, July 07, 2005 5:32 AM
     To: [email-masked]
     Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode
     -> summary(?)

     some pieces
     still needed. See
     <[email-masked]>
     <[email-masked]>;
     mode -> summary
     START-OF-SPEECH is
     (e.g. VoiceXML
     stops prompt
     the previous
     bargin is different
     playing) the
     client would
     in  DTMF start
     however. For
     will stop playing
     handle the
     have a clue what
     input type(s) a
     look for"?
     only have a
     noise would
     getting DTMF,
     noise was
     when you have
     non-optimized
     event should be
     aesthetic protocol
     ahead of time
     event for, or
     decides based on
     the spec implies
     that if a
     the client will
     in the design
     approach where the
     event or not,
     than some
     also is becoming
     we should
     something akin
     DTMF-only recognizer
     gedankenexperiment.
     which I'll call
     music to the
     grammar is a
     title of the
     on a minute).
     the server to
     music or now,
     do based on
     or not we do
     "I'm hearing
     now throw in the
     recognizer is a
     "Geographical Fugue"
     recommend to
     this case, the
     recognition.
     the server as to
     (because a second
     a grammar
     input is "on"
     what I mean",
     allows one to
     because the
     parts of the
     specify this
     like "Say or
     indicate, somehow,
     the other, I
     interesting
     during a request
     However, I don't
     solved by them. I
     thoroughly as you.
     listens for
     grammars are
     to get this
     or speech &
     [email-masked]
     "Shanmugham, Saravanan"
     DTMF-only mode
     because it is
     says to me is
     (especially if those
     to qualify
     (could be a
     or captain
     response. I am not
     speech, DTMF or
     of this as a
     the optimized
     START-OF-SPEECH event to
     barge-in event.
     resource today,
     either add an
     was detected?
     recognizing DTMF
     unless it's using
     Recognizer, but only
     send to the
     conclusions. I'd be
     above is wrong.

---

From: David R Oran <[email-masked]>
Date: Fri, 8 Jul 2005 10:05:22 -0400

On Jul 8, 2005, at 6:59 AM, Dave Burke wrote:

> I agree with your opinion on what constitutes a barge-in. Based on  
> Klaus' e-mail, I am more concerned that using the grammar type to  
> decide what constitutes a barge-in is going to make VoiceXML  
> implementations difficult.
>
> recongition so as (a) to avoid unnecessary limitations in VUI  
> design, and (b) to facilitate VoiceXML implementations. I think we  
> should add a header to RECOGNIZE so the client is able to always  
> _explicitly_ set the inputmodes.
>
I'm not sure I buy this, since what the recognizer is looking for and  
what constitutes an input that should be considered a potential barge- 
in strike me as independent. I'm similarly not persuaded that we need  
the flexibility to set the input mode independently of the grammar,  
since it leads to all sorts of inconsistent states (e.g. speech-only  
grammar with an input-mode of dtmf). On the other hand I can see the  
need for the client to specify what sorts of input out to generate  
the start-of-input (nee start-of-speech) event on.

> In retrospect, I don't like the idea of multiple START-OF-SPEECH  
> events being generated from the same media resource. This is  
> because one assumes that the START-OF-SPEECH should be of the same  
> type as the hypothesis returned in the RECOGNITION-COMPETE message  
> - most implementations, on hearing one input mode type disable the  
> recogniser of the other type.
I'm not sure I follow this logic, but I'm not wedded to the idea of  
allowing multiple events. It was trying to solve the problem of  
ambiguity around what the recognizer was hearing and the idea that  
the client might care. If you don't think the client will ever care,  
then we don't need the capability.

> I do like David's idea of renaming START-OF-SPEECH to something  
> like START-OF-INPUT and carrying a type header because it is neater  
> and more extensible.
>
> 1. Clarify what constitutes a barge-in for a dtmfrecog and  
> speechrecog as per Sarvi's e-mail (and in agreement with Klaus' for  
> dtmfrecog).
>
Hmmm, ok, but I think the issue is actually clarifying what the  
recognizer declares as "interesting input", which it may decide also  
constitutes a barge-in in the optimized case, and in either case  
reports to the client that it heard.

> 2. Specifiy an InputModes header to RECOGNIZE (defaults to "both",  
> can also be "speech" or "DTMF"). Setting to speech for a  
> speechrecog results in the hypothesised "speech-only-recog".  
> Setting to DTMF for a speechrecog is equivalent to using a  
> dtmfrecog. Edge cases: Setting to speech for a dtmfrecog will  
> result in a noinput as will setting to DTMF for a speechrecog which  
> does not support DTMF.
>
I'm ok with having a header for the client to tell the server when it  
would like the start-of-input event to be generated and what the  
client considers to be the "interesting input" that the recognizer  
should use to do the discrimination, and possibly do barge-in  
processing in the optimized case.

I'm less ok with the proposed domain of values, since it will have  
extensibility problems. Especially problematical is have a code point  
of "both" since that will be ambiguous if we even define a "music"  
recognizer, or a "gesture" recognizer using video input. Here's my  
counter-proposal:

Create a header on the recognizer resource requests called "start- 
input-on:". Define a registry of values, with the following three  
values initially defined:
     - dtmf
     - speech
     - music

The semantics would be that the resource is to generate a "start-of- 
input" event, apply the defined grammars to what follows, and do any  
local optimized barge-in processing if ANY of the enumerated input  
types is detected. That way you can say
     "start-input-on: dtmf" if you want to just  dtmf,
     "start-input-on: speech" if you want just speech (dtmf input  
would be ignored even if the grammar supported dtmf)
     "start-input- on: speech, dtmf" if you wanted both
etc.

> 3. Change START-OF-SPEECH to START-OF-INPUT. Clarify it occurs once  
> and coincides with barge-in.
>
Ok for the "only once", but I'd like to tighten it up to talk about  
more than just barge-in, as I suggested above.

> 4. Add a header of InputType to START-OF-INPUT. Current specified  
> values are "dtmf" or "speech".
>
Ok, with slight modification. Say that the syntax of the "input-type"  
header is a single-valued subset of the registered value(s) that are  
defined for the start-input-on: header.

Comments?

Dave O. (technical hat on, chair hat off).

> mode