Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode -> summary(?)

From: "Reifenrath, Klaus" <[email-masked]>
Date: Thu, 7 Jul 2005 09:03:56 -0400

I) START-XYZ for the new event is misleading if we allow the event to be
generated multiple times.
Do we really need to introduce a new (optional) event? Why should a DTMF
only resource do speech detection? 

II)
  1. START-OF-SPEECH (and optimised bargin) is only generated for the  
  input type that is being listened for
  2. A speechrecog listens for DTMF if DTMF grammars are active,  
  speech if speech grammars are active, or speech and DTMF if both  
  grammar types are active.
I personally like this, BUT doesn't it conflict with VoiceXML 2.0 section
4.1.5.1 (bargein type speech):
"The prompt will be stopped as soon as speech or DTMF input is detected. The
prompt is stopped irrespective of whether or not the input matches a grammar
and irrespective of which grammars are active."  

Klaus

-----Original Message-----
From: Eric Burger [mailto:[email-masked]] 
Sent: Donnerstag, 7. Juli 2005 14:32
To: [email-masked]
Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode -> summary(?)

Can we declare consensus? 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, July 06, 2005 10:45 AM
> To: Dave Burke
> Cc: [email-masked]; Shanmugham, Saravanan; Klaus Reifenrath
> Subject: Re: [Speechsc] START-OF-SPEECH in DTMF-only mode ->
> summary(?)
> 
> I think we're getting close. I though about snipping out some pieces 
> to cut down the text, but I realized the context is still needed. See 
> inline.
> 
> On Jul 6, 2005, at 10:29 AM, Dave Burke wrote:
> 
> > Inline.
> >
> > Dave
> >
> > ----- Original Message ----- From: "David R Oran" <[email-masked]>
> > To: "Dave Burke" <[email-masked]>
> > Cc: <[email-masked]>; "Shanmugham, Saravanan" <[email-masked]>; 
> > "Klaus Reifenrath" <[email-masked]>
> > Sent: Wednesday, July 06, 2005 1:07 PM
> > Subject: Re: [Speechsc] START-OF-SPEECH in DTMF-only mode -> summary
> > (?)
> >
> >
> >
> >>
> >> On Jul 6, 2005, at 5:31 AM, Dave Burke wrote:
> >>
> >>
> >>> + Attempting to summarise:
> >>>
> >>> 1. START-OF-SPEECH is useful for the client to know when
> to stop   
> >>> playing prompts the in non-optimised case 2. START-OF-SPEECH is 
> >>> useful for the client to calculate the bargin  time (e.g. VoiceXML 
> >>> 2.1 <mark>)
> >>> 3. In the optimised case, a bargin automatically stops prompt   
> >>> playing (assuming prompts barginable) 4. Because of the previous 
> >>> point, the question of what input type  caused bargin is different 
> >>> and less important to
> what input
> >>> type(s)  the recogniser is listening for
> >>>
> >> I'm not sure I follow point 4. Could you elaborate?
> >>
> >
> > DB> Adding a parameter to the START-OF-SPEECH event would
> certainly
> > allow the client to ignore (i.e. let prompts continue playing) the 
> > event if the event type is not of interest (e.g. the client would 
> > ignore speech start events when it is interested only in  DTMF start 
> > events). This _only_ works for the non-optimised case, however. For 
> > the optimised case, assuming START-OF-SPEECH
> coincides
> > with the bargin signal to the speechsynth, prompts will stop playing 
> > for inputs that the client might not be interested
> in (e.g.  
> > a speech input will stop prompts playing even if the client
> is only
> > interested in DTMF).
> >
> OK, now I get it. There's a need for the client to both handle the 
> non-optimized case itself, and influence or at least have a clue what 
> the server is going to do in the optimized case.
> 
> >
> >>
> >>
> >>> 5. Currently, MRCPv2 has no way of indicating what input type(s) a 
> >>> recogniser is listening for
> >>>
> >>>
> >> Do you mean exactly this, or do you mean "for the client to 
> >> indicate  to the resource what input types it should look for"?
> >>
> >
> > DB> Yes exactly - apologies for not being clear.
> >
> >
> >>
> >>
> >>> + Why implement 5?
> >>>
> >>> i. Noisy case: Need DTMF-only recognition (and may only have a
> >>> speechrecog)
> >>>
> >> I'm having difficulty following the logic of why noise would   
> >> necessarily trigger START-OF-SPEECH if you were listening for 
> >> speech  but not DTMF. I suppose you can use a more forgiving 
> >> discriminator if  all you need to tell is if you're getting DTMF, 
> >> but I've had a number  of real-world cases where wind noise was 
> >> detected as DTMF, and  there's always the ambiguity when you have 
> >> Captain Crunch on the  phone. In either case in the non-optimized 
> >> case it's the client who  gets to decide whether an event should be 
> >> interpreted as barge-in or  not, so it seems an aesthetic protocol 
> >> design decision whether the  client tells the server ahead of time 
> >> what circumstances to generate  the START-OF-SPEECH event for, or 
> >> whether the event gets generated  and the client decides based on 
> >> what's in the event whether it should  be
> treated
> >> as barge- in.
> >>
> >> I suppose one could make the argument that because the spec implies  
> >> that the event can only be generated once per request that if a 
> >> DTMF/ speech capable recognizer first hears
> enough noise
> >> to think it's  hearing speech and later hears DTMF, the client will 
> >> declare barge-in  when the event comes and not when he DTMF 
> >> actually gets heard.
> >>
> >> If that's deemed a problem, we can still handle that in the design  
> >> where the server just reports what it's hearing by allowing 
> >> multiple  events to be generated during a single request.
> >>
> >> Between the approach just outlined above, and an approach where the 
> >> client provides a filter for whether to generate the event or not, 
> >> I  have a mild preference (based on aesthetics rather than some 
> >> hard  engineering tradeoff) for the approach where
> the server
> >> just reports  what it's hearing.
> >>
> >>
> >> Having had some useful exchanges on this topic, it also is becoming 
> >> apparent to me that this event is poorly named, and we should  
> >> consider renaming it to "INTERESTING-INPUT-HEARD" or something akin  
> >> to that, because as others have pointed out, a DTMF-only recognizer  
> >> will never detect "start of speech".
> >>
> >> Another consideration to fold into the design choice is   
> >> extensibility. Bear with me through a little gedankenexperiment.
> >>
> >> Suppose we want to define a new recognizer type, which I'll call 
> >> the "name that tune" recognizer. The client plays music to the 
> >> server and  the server recognizes musical notes. The grammar is a 
> >> standard  musical notation, augmented with a semantic 
> >> interpretation that  transforms the notes into the title of the 
> >> tune and provides that as  an answer.
> >>
> >> First, there's no speech involved (or is there...hang on a minute). 
> >> Second, in order to accommodate the "name that tune"
> >> recognizer, we'd have to extend both the client and the server to 
> >> undetstand a  directive as to whether to recognize music or now, 
> >> inaddition to what  the server already knows what to do based on 
> >> the grammar. If you  follow my logic above, whether or not we do 
> >> that, we have to extend  "start-of-speech" to say "I'm hearing 
> >> music". So far fairly  straightforward, but let me now throw in the 
> >> pathological twist.
> >>
> >> Suppose what I feed to a  combined music/speech recognizer is a 
> >> work  in sprechstimme (spoken music), like the "Geographical Fugue" 
> >> (aside:  this is a wonderful piece of music I highly recommend to 
> >> anyone  interested in small ensemble singing). In this case, the 
> >> tune could  be named by either doing speech or music recognition. 
> >> Why is there  any need for the client to constrain the server as to 
> >> which it tries  to do when it's
> already
> >> told the server what it wants through the  grammar?
> >>
> >> A few other comments below
> >>
> >>
> >>> ii. Flexibility: Want speech-only recognition (because a second 
> >>> recogniser is doing hotword on DTMF)
> >>>
> >>>
> >> I don't see how flexibility is affected by this deisgn
> choice. If   
> >> that's what you want, feed the speech-only recognizer a grammar   
> >> without any DTMF rules.
> >>
> >>
> >>> + How to implement 5?
> >>>
> >>> a. Implicitly:
> >>>    - dtmfrecog: always DTMF-only recognition
> >>>    - speechrecog: depends on active grammar type
> >>>        > if a dtmf grammar is active then DTMF input is "on"
> >>>        > if a speech grammar is active then speech input is "on"
> >>>
> >>> b. Explicitly:
> >>>    - Add inputmodes header to RECOGNIZE
> >>>
> >>> Option a is David's "do what I mean case"; option b is
> the extra   
> >>> dial for the client.
> >>>
> >>>
> >> Actually, that's not the point I was making with "do what I mean",  
> >> but it's not essential to the discussion so let's move on.
> >>
> >>
> >>> It is worth noting that VoiceXML uses option b. This allows one to 
> >>> activate both speech grammars and DTMF grammars (and
> therefore
> >>> be informed of any errors in the grammars at activation
> time) but
> >>> independently turn on whichever input mode you like e.g. 
> perhaps   
> >>> start with "both" then change to "dtmf".
> >>>
> >>>
> >> I'm not sure the VXML precedent is relevant here, because the   
> >> application behind VXML is working a different part of the
> problem
> >> -  how to traverse a TUI dialog based on different parts of the 
> >> input  space. In fact, I suspect that the VXML: choice was 
> >> conditioned more  by limitations at the time it was
> specified than
> >> an underlying good  design choice. Clearly having to specify this 
> >> in VXML make the job of  handling a TUI with nodes like "Say or 
> >> press 5" harder rather than  easier.
> >>
> >
> > DB> The VoiceXML edge-case is pretty weird so it's not a major
> > concern. My main concern is that the client can indicate, somehow, 
> > what the input modes are.
> >
> >
> >>
> >> Summing up, while I don't feel strongly one way or the other, I 
> >> have  a preference for handling this as follows:
> >>
> >> a) Rename "START-OF-SPEECH" to "INTERESTING-INPUT-RECEIVED" or   
> >> something equivalent.
> >> b) Include a parameter in the event saying what was interesting 
> >> about  the input you received, with a registry of values which
> >> includes:
> >>     - signal above noise floor
> >>     - speech
> >>     - dtmf
> >>     - (possibly) music
> >> c) allow the event to be generated multiple times during a request
> >>
> >>
> >
> > DB> I like these suggestions (START-OF-INPUT?). However, I don't
> > see how the problem of the optimised case is not solved by them. I 
> > think the optimised case is fine if we have the following rules:
> >
> Yes, I hadn't thought through the optimized case as thoroughly as you. 
> Your suggested method name is fine by me as well
> 
> > 1. START-OF-SPEECH (and optimised bargin) is only generated
> for the
> > input type that is being listened for 2. A speechrecog listens for 
> > DTMF if DTMF grammars are active, speech if speech grammars are 
> > active, or speech and DTMF if both grammar types are active.
> >
> Works for me.
> 
> >
> >> Note that all of the above I'm saying with my technical
> hat on and
> >> my chair hat off.
> >> Putting my chair hat on for a moment, we really need to get this 
> >> spec  to last call, so at some point Eric or I is going to
> declare
> >> rough  consensus so we can move on.
> >>
> >
> > DB> Agreed!
> >
> >
> >>
> >> Dave Oran.
> >>
> >>> Dave
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Sarvi makes a good point that adding the reason why the
> START-OF-
> >>> SPEECH occurred does not fix the optimised bargin case.
> >>>
> >>> dtmfrecog - listens for DTMF only
> >>> speechrecog - listens for DTMF only, or speech only, or speech & 
> >>> DTMF
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> ----- Original Message ----- From: "Shanmugham, Saravanan"  
> >>> <[email-masked]>
> >>> To: "David R Oran" <[email-masked]>; "Dave Burke"  
> >>> <[email-masked]>
> >>> Cc: <[email-masked]>; "Klaus Reifenrath"  
> >>> <[email-masked]>
> >>> Sent: Tuesday, July 05, 2005 8:51 PM
> >>> Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode
> >>>
> >>>
> >>> inline.
> >>>
> >>>     -----Original Message-----
> >>>     From: David R Oran [mailto:[email-masked]]
> >>>     Sent: Tuesday, July 05, 2005 11:08 AM
> >>>     To: Dave Burke
> >>>     Cc: Shanmugham, Saravanan; Klaus Reifenrath; [email-masked]
> >>>     Subject: Re: [Speechsc] START-OF-SPEECH in DTMF-only mode
> >>>
> >>>
> >>>     On Jul 5, 2005, at 12:54 PM, Dave Burke wrote:
> >>>
> >>>     > Inline.
> >>>     >
> >>>     > Dave
> >>>     >
> >>>     > ----- Original Message ----- From: "Shanmugham, Saravanan"
> >>>     > <[email-masked]>
> >>>     > To: "David R Oran" <[email-masked]>; "Klaus Reifenrath"
> >>>     > <[email-masked]>
> >>>     > Cc: <[email-masked]>
> >>>     > Sent: Tuesday, July 05, 2005 5:22 PM
> >>>     > Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode
> >>>     >
> >>>     >
> >>>     > I agree with Dave's analysis. The purpose of this event
> >>>     was barge-in.
> >>>     > And barge-in should happen for both DTMF and speech.
> >>>     >
> >>>     > Is there a case where you think it should not behave
> >>>     this way. If soe,
> >>>     > please provide a scenario where you think
> >>>     >    1. Barge-in should happen for DTMF and not voice or
> >>>     vice-versa.
> >>>     >
> >>>     > DB> You want to do a DTMF recognition only because it is 
> >>> noisy.
> >>>     > While waiting for DTMF input, the speechrecog resource
> >>>     (or advanced
> >>>     > dtmfrecog) generates a START-OF-SPEECH because it heard
> >>>     some speech.
> >>>     > The client does not want to stop prompt playing unless
> >>>     DTMF was heard
> >>>     > but it can't tell by the START-OF-SPEECH whether speech
> >>>     or DTMF was
> >>>     > heard. Similarly vice versa.
> >>>     >
> >>>     It's an interesting design question what part of the
> >>>     policy resides at the client and what at the server, and
> >>>     who makes the "final decision" about whether what was
> >>>     heard was relevant to the control channel. Right now we
> >>>     (IMO) have a weird partitioning in many cases where the
> >>>     client basically says "do what I mean", but there are no
> >>>     constraints of what the server actually does, and no
> >>>     normalized basis for the client to figure out what to set
> >>>     various magic numbers to (e.g. sensitivity).
> >>>
> >>>     In this case the only thing the client needs to decide is
> >>>     whether to kill the prompt because the server thinks
> >>>     something that would interfere with the feedback
> >>>     ear/mouth/finger control happened. What this says to me is
> >>>     that it isn't necessarily a good idea for the client to
> >>>     have more knobs to control the server (especially if those
> >>>     knows are just more value/policy input ungrounded in any
> >>>     physics/ acoustics). On the other hand, having the server
> >>>     tell the client more about what it thinks is going on is
> >>>     probably valuable.
> >>>
> >>>     So, Coming to the point after this long rambling
> >>>     introduction, I think it would in fact be useful for the
> >>>     START-Of-SPEECH event to indicate some extra information,
> >>>     for example:
> >>>     a) I got something enough above the noise floor to qualify
> >>>     for exceeding the "Sensisitvity" parameter you sent in on
> >>>     the request but I really can't tell what it is (could be a
> >>>     hippopatmus fart, or a siren in the background, or captain
> >>>     crunch trying to whistle DTMF).
> >>>     b) I think I'm hearing speech
> >>>     c) I think I'm hearing DTMF
> >>>
> >>> Though I agree with your former part of your response. I am not 
> >>> sure I agree with your proposed solution.
> >>> The way I see this problem is that, it is more of what
> constitues a
> >>> barge-in event. This boils down to whether it is speech, DTMF or 
> >>> both.
> >>> This is inturn boils down to what type of recognizer
> resource we are
> >>> using, dtmf-recog, speech-recog, and speech-only-recog(we
> don't have
> >>> this and I don't think we should add it, but think of this as a 
> >>> place holder that explains the concept).
> >>>
> >>> A client knowing what type of barge-in happenned, does
> not impact
> >>> the
> >>> barge-in operation itself as it may be too late(for the optimized 
> >>> barge-in case). It may have other use cases, and if we
> can identify
> >>> them, I don't mind adding support for the START-OF-SPEECH event to 
> >>> say what type of barge-in happenned. But that itself does not
> solve the
> >>> original problem raised. Refer to my previous response.
> >>>
> >>> The solution lies in defining what what is a barge-in event.  
> >>> That  boils
> >>> down to what type of recognition is happenning,
> dtmf-only, speech-
> >>> dtmf
> >>> or speech-only. We do not support speech-only as a resource today, 
> >>> the question is do we need a header to force it.
> >>>
> >>> Sarvi
> >>>
> >>>
> >>>     >    2. You would benefit from the client knowing 
> what caused
> >>> the
> >>>     > barge-in, DTMF Vs speech.
> >>>     >
> >>>     > DB> See previous comment. And previous e-mail: either add an
> >>>     > inputmodes header (taking value speech, dtmf, both) to
> >>>     the RECOGNIZE
> >>>     > request or add a header to the START-OF-SPEECH event
> >>>     indicating DTMF
> >>>     > or speech.
> >>>     >
> >>>     I'm leaning in your direction on this latter point - as
> >>>     should be evident from what I wrote above.
> >>>
> >>>     > Sarvi
> >>>     >
> >>>     >     -----Original Message-----
> >>>     >     From: [email-masked]
> >>>     >     [mailto:[email-masked]] On Behalf Of 
> David R
> >>> Oran
> >>>     >     Sent: Tuesday, July 05, 2005 5:24 AM
> >>>     >     To: Klaus Reifenrath
> >>>     >     Cc: [email-masked]'
> >>>     >     Subject: Re: [Speechsc] START-OF-SPEECH in 
> DTMF-only mode
> >>>     >
> >>>     >
> >>>     >     On Jul 5, 2005, at 3:46 AM, Reifenrath, Klaus wrote:
> >>>     >
> >>>     >     > The current spec is not clear when 
> START-OF-SPEECH need
> >>>     >     to be send in
> >>>     >     > the following scenarios:
> >>>     >     > A) The client requested a DTMF Recognizer. Is the
> >>>     >     START-OF-SPEECH
> >>>     >     > event send to the client also if speech was detected?
> >>>     >     I suspect so, since one of the prime purposes 
> is to enable
> >>>     >     client- mediated barge-in handling. However, if the
> >>>     >     recognizer is in fact only capable of recognizing DTMF
> >>>     >     then it may in fact not report anythin unless it's using
> >>>     >     some primitive thresholding machinery, like a SN  
> >>> threshold.
> >>>     >     > B) The client requested a Speech Recognizer, but only
> >>>     >     activated DTMF
> >>>     >     > grammars. Is the START-OF-SPEECH event send to the
> >>>     >     client also if
> >>>     >     > speech was detected?
> >>>     >     Again, I'd say yes, for the same reason as above.
> >>>     >     > I think in both cases START-OF-SPEECH should only
> >>>     be send after
> >>>     >     > detecting a DTMF digit (see Figure 12 of VoiceXML
> >>>     2.0: http://
> >>>     >     > www.w3.org/TR/voicexml20/#dmlATiming).
> >>>     >     We seem to have reached different conclusions. I'd be
> >>>     >     interested in why you think my analysis above is wrong.
> >>>     >
> >>>     >     Dave.
> >>>     >
> >>>     >     > Klaus
> >>>     >     >
> >>>     >     > _______________________________________________
> >>>     >     > Speechsc mailing list
> >>>     >     > [email-masked]
> >>>     >     > https://www1.ietf.org/mailman/listinfo/speechsc
> >>>     >     >
> >>>     >
> >>>     >     _______________________________________________
> >>>     >     Speechsc mailing list
> >>>     >     [email-masked]
> >>>     >     https://www1.ietf.org/mailman/listinfo/speechsc
> >>>     >
> >>>     >
> >>>     > _______________________________________________
> >>>     > Speechsc mailing list
> >>>     > [email-masked]
> >>>     > https://www1.ietf.org/mailman/listinfo/speechsc
> >>>     >
> >>>
> >>>
> >>> _______________________________________________
> >>> Speechsc mailing list
> >>> [email-masked]
> >>> https://www1.ietf.org/mailman/listinfo/speechsc
> >>>
> >>>
> >>
> >> _______________________________________________
> >> Speechsc mailing list
> >> [email-masked]
> >> https://www1.ietf.org/mailman/listinfo/speechsc
> >
> 
> _______________________________________________
> Speechsc mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/speechsc
> 

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: "Pierre Forgues" <[email-masked]>
Date: Thu, 7 Jul 2005 09:21:48 -0400

I agree with Klaus's position.  

Moreover, when the SOS event is generated, it is possible exact nature
of the barge-in method is not known at that time.  It is critical to
notify the user as soon as possible to stop the prompt or this impacts
recognition accuracy.  In some cases, the resource might not even have
exact determination at that ninstant in time whether this is DTMF or
speech.

/Pierre

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Klaus Reifenrath
Sent: Thursday, July 07, 2005 9:04 AM
To: 'Eric Burger'; [email-masked]
Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode -> summary(?)

II)
  1. START-OF-SPEECH (and optimised bargin) is only generated for the  
  input type that is being listened for
  2. A speechrecog listens for DTMF if DTMF grammars are active,  
  speech if speech grammars are active, or speech and DTMF if both  
  grammar types are active.
I personally like this, BUT doesn't it conflict with VoiceXML 2.0
section
4.1.5.1 (bargein type speech):
"The prompt will be stopped as soon as speech or DTMF input is detected.
The
prompt is stopped irrespective of whether or not the input matches a
grammar
and irrespective of which grammars are active."  

Klaus