Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode

From: "Pierre Forgues" <[email-masked]>
Date: Tue, 5 Jul 2005 11:38:07 -0400

The START-OF-SPEECH event should be sent to the client whether the user
spoke or pressed DTMF.  I agree with Dave Oran's comment that it applies
to both scenarios mentioned below.

The action to take on the client side is likely going to be the same in
both cases, i.e. to stop the prompt.

Pierre

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of David R Oran
Sent: Tuesday, July 05, 2005 8:24 AM
To: Klaus Reifenrath
Cc: [email-masked]'
Subject: Re: [Speechsc] START-OF-SPEECH in DTMF-only mode

On Jul 5, 2005, at 3:46 AM, Reifenrath, Klaus wrote:

> The current spec is not clear when START-OF-SPEECH need to be send  
> in the following scenarios:
> A) The client requested a DTMF Recognizer. Is the START-OF-SPEECH  
> event send to the client also if speech was detected?
I suspect so, since one of the prime purposes is to enable client- 
mediated barge-in handling. However, if the recognizer is in fact  
only capable of recognizing DTMF then it may in fact not report  
anythin unless it's using some primitive thresholding machinery, like  
a SN threshold.
> B) The client requested a Speech Recognizer, but only activated  
> DTMF grammars. Is the START-OF-SPEECH event send to the client also  
> if speech was detected?
Again, I'd say yes, for the same reason as above.
> I think in both cases START-OF-SPEECH should only be send after  
> detecting a DTMF digit (see Figure 12 of VoiceXML 2.0: http:// 
> www.w3.org/TR/voicexml20/#dmlATiming).
We seem to have reached different conclusions. I'd be interested in  
why you think my analysis above is wrong.

Dave.

> Klaus
>
> _______________________________________________
> Speechsc mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/speechsc
>

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: "Shanmugham, Saravanan" <[email-masked]>
Date: Tue, 5 Jul 2005 10:18:03 -0700

Barge-in does not necessarily happen only through the START-OF-SPEECH
event. Note that if both the ASR and TTS resources are on the same box
they could optimize internally for better performance.

So I think what you are suggesting that is that the recognizer should be
sensitive to what input is being chosen, when deciding when to generate
the START-OF-SPEECH event.

I was going to say, this could be based on the grammar and whether it
contains tokens other than digits or not. But I guess even if they
contained only digits we could still either speak it or press it and the
recognizer should be able to deal with both.

In which case the problem you are raising becomes less of a barge-in
issue. It more about restricting the recognizer to do recognition in a
speech-only or dtmf-only or both modes and whether we want to allow it. 

If we want to DTMF-only, we could always allocate a DTMF-only recognizer
and thus not waste precious speech resources for that channel. If we
want to do both a regular recognizer resource can do both. The question
then becomes, is there a case where you may want to do speech only
recognition or not. Is there a use case for this restriction. The noisy
channel argument does not apply here I am guessing. 

To address the barge-in problem we could just add some text to clarify
that a dtmf-recog resource generates the barge-in event only for DTMF
digits while the speech-recog resource generates the barge-in event for
both speech or DTMF.

Sarvi

     -----Original Message-----
     From: Dave Burke [mailto:[email-masked]] 
     Sent: Tuesday, July 05, 2005 9:54 AM
     To: Shanmugham, Saravanan; David R Oran; Klaus Reifenrath
     Cc: [email-masked]
     Subject: Re: [Speechsc] START-OF-SPEECH in DTMF-only mode
     
     Inline.
     
     Dave
     
     ----- Original Message -----
     From: "Shanmugham, Saravanan" <[email-masked]>
     To: "David R Oran" <[email-masked]>; "Klaus Reifenrath" 
     <[email-masked]>
     Cc: <[email-masked]>
     Sent: Tuesday, July 05, 2005 5:22 PM
     Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode

     I agree with Dave's analysis. The purpose of this event 
     was barge-in.
     And barge-in should happen for both DTMF and speech.
     
     Is there a case where you think it should not behave this 
     way. If soe,
     please provide a scenario where you think
         1. Barge-in should happen for DTMF and not voice or vice-versa.
     
     DB> You want to do a DTMF recognition only because it is 
     noisy. While 
     waiting for DTMF input, the speechrecog resource (or 
     advanced dtmfrecog) 
     generates a START-OF-SPEECH because it heard some speech. 
     The client does 
     not want to stop prompt playing unless DTMF was heard but 
     it can't tell by 
     the START-OF-SPEECH whether speech or DTMF was heard. 
     Similarly vice versa.
     
         2. You would benefit from the client knowing what caused the
     barge-in, DTMF Vs speech.
     
     DB> See previous comment. And previous e-mail: either add 
     an inputmodes 
     header (taking value speech, dtmf, both) to the RECOGNIZE 
     request or add a 
     header to the START-OF-SPEECH event indicating DTMF or speech.
     
     Sarvi
     
          -----Original Message-----
          From: [email-masked]
          [mailto:[email-masked]] On Behalf Of David R Oran
          Sent: Tuesday, July 05, 2005 5:24 AM
          To: Klaus Reifenrath
          Cc: [email-masked]'
          Subject: Re: [Speechsc] START-OF-SPEECH in DTMF-only mode

          On Jul 5, 2005, at 3:46 AM, Reifenrath, Klaus wrote:
     
          > The current spec is not clear when START-OF-SPEECH need
          to be send in
          > the following scenarios:
          I suspect so, since one of the prime purposes is to enable
          client- mediated barge-in handling. However, if the
          recognizer is in fact only capable of recognizing DTMF
          then it may in fact not report anythin unless it's using
          some primitive thresholding machinery, like a SN threshold.
          > B) The client requested a Speech Recognizer, but only
          activated DTMF
          > grammars. Is the START-OF-SPEECH event send to the
          client also if
          > speech was detected?
          > www.w3.org/TR/voicexml20/#dmlATiming).
          We seem to have reached different conclusions. I'd be
          interested in why you think my analysis above is wrong.
     
          Dave.

---

From: "Pierre Forgues" <[email-masked]>
Date: Tue, 5 Jul 2005 16:12:27 -0400

The client will be told in the recognition results exactly what
triggered the START-OF-SPEECH event (dtmf or speech).  At the time the
SOS event is reported, the knowledge of what triggered the event is not
needed to make a decision to stop a prompt (nor does the server even
know at that time what triggered it).

Pierre

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On
Behalf Of Shanmugham, Saravanan
Sent: Tuesday, July 05, 2005 3:52 PM
To: David R Oran; Dave Burke
Cc: [email-masked]; Klaus Reifenrath
Subject: RE: [Speechsc] START-OF-SPEECH in DTMF-only mode

 inline.

     -----Original Message-----
     From: David R Oran [mailto:[email-masked]] 
     Sent: Tuesday, July 05, 2005 11:08 AM
     To: Dave Burke
     Cc: Shanmugham, Saravanan; Klaus Reifenrath; [email-masked]
     Subject: Re: [Speechsc] START-OF-SPEECH in DTMF-only mode

     On Jul 5, 2005, at 12:54 PM, Dave Burke wrote:
     
     > Inline.
     >
     > Dave
     >
     > ----- Original Message ----- From: "Shanmugham, Saravanan"  
     > <[email-masked]>
     was barge-in.
     > Is there a case where you think it should not behave 
     this way. If soe, 
     > DB> You want to do a DTMF recognition only because it is noisy.  
     > While waiting for DTMF input, the speechrecog resource 
     (or advanced
     > dtmfrecog) generates a START-OF-SPEECH because it heard 
     some speech. 
     > The client does not want to stop prompt playing unless 
     DTMF was heard 
     > but it can't tell by the START-OF-SPEECH whether speech 
     or DTMF was 
     > heard. Similarly vice versa.
     >
     It's an interesting design question what part of the 
     policy resides at the client and what at the server, and 
     who makes the "final decision" about whether what was 
     heard was relevant to the control channel. Right now we 
     (IMO) have a weird partitioning in many cases where the 
     client basically says "do what I mean", but there are no 
     constraints of what the server actually does, and no 
     normalized basis for the client to figure out what to set 
     various magic numbers to (e.g. sensitivity).
     
     In this case the only thing the client needs to decide is 
     whether to kill the prompt because the server thinks 
     something that would interfere with the feedback 
     ear/mouth/finger control happened. What this says to me is 
     that it isn't necessarily a good idea for the client to 
     have more knobs to control the server (especially if those 
     knows are just more value/policy input ungrounded in any 
     physics/ acoustics). On the other hand, having the server 
     tell the client more about what it thinks is going on is 
     probably valuable.
     
     So, Coming to the point after this long rambling 
     introduction, I think it would in fact be useful for the 
     START-Of-SPEECH event to indicate some extra information, 
     for example:
     a) I got something enough above the noise floor to qualify 
     for exceeding the "Sensisitvity" parameter you sent in on 
     the request but I really can't tell what it is (could be a 
     hippopatmus fart, or a siren in the background, or captain 
     crunch trying to whistle DTMF).
     b) I think I'm hearing speech
     c) I think I'm hearing DTMF

Though I agree with your former part of your response. I am not sure I
agree with your proposed solution. 
The way I see this problem is that, it is more of what constitues a
barge-in event. This boils down to whether it is speech, DTMF or both.
This is inturn boils down to what type of recognizer resource we are
using, dtmf-recog, speech-recog, and speech-only-recog(we don't have
this and I don't think we should add it, but think of this as a place
holder that explains the concept).

A client knowing what type of barge-in happenned, does not impact the
barge-in operation itself as it may be too late(for the optimized
barge-in case). It may have other use cases, and if we can identify
them, I don't mind adding support for the START-OF-SPEECH event to say
what type of barge-in happenned. But that itself does not solve the
original problem raised. Refer to my previous response.

The solution lies in defining what what is a barge-in event. That boils
down to what type of recognition is happenning, dtmf-only, speech-dtmf
or speech-only. We do not support speech-only as a resource today, the
question is do we need a header to force it. 

Sarvi     

     > DB> See previous comment. And previous e-mail: either add an
     > inputmodes header (taking value speech, dtmf, both) to 
     the RECOGNIZE 
     > request or add a header to the START-OF-SPEECH event 
     indicating DTMF 
     > or speech.
     I'm leaning in your direction on this latter point - as 
     should be evident from what I wrote above.