Subject: RE: [speechsc] Accurate determination of START-OF-INPUT time

From: "Andrew Wahbe" <[email-masked]>
Date: Fri, 7 Jul 2006 07:44:55 -0700

Right -- actually, I'd imagine by suggesting an NTP timestamp, Dave was
implying that this is the timestamp in the RTP stream where start of
speech was detected. As the input-waveform-uri begins at the start of
the RECOGNIZE, I can see how absolute times don't work for what you are
proposing unless you also have the timestamp of when recognition
started.
 
How about the following:
A "Timestamp" header is added that has a timestamp as its value.
The header may appear on a response or an event. On a response, it
indicates the time in the stream at which the associated request was
applied (e.g. when did the RECOGNIZE start). On an event, it indicates
the time at which the event occured (e.g. when did start of speech
occur).
 
This is a more general solution to what is being done with the
speech-marker (see:
http://www1.ietf.org/mail-archive/web/speechsc/current/msg01813.html) If
we really want to know when these things are happening in relation to
the RTP stream in general then I think a separate timestamp header makes
sense rather than sticking the timestamp information in different
headers for each resource or message type.
 
Now there is the question of what format to use for the timestamp. It
seems what we are really interested in is the "point of time" in the
stream as opposed to the clock time. In the discussion related to the
speech marker timestamp it was pointed out that the RTP time can be
derived from the NTP time using an RTCP sender report (which includes
the RTP and NTP time when the report was sent). However, it is common
practice for "bursty" or faster-than-real-time transmission of RTP
between voice platforms and voice resources. For example, this can be
done in synthesis to allow telephony card buffers to be filled up, and
in recognition if the voice platform buffers audio while a speech
session is established. Doesn't this destroy the relationship between
NTP time and the RTP timestamp? I think it makes more sense to use the
RTP timestamp here.

Andrew Wahbe

________________________________

From: Corby Anderson [mailto:[email-masked]] 
Sent: July 6, 2006 1:59 PM
To: Andrew Wahbe
Cc: Dave Burke; [email-masked]
Subject: Re: [speechsc] Accurate determination of START-OF-INPUT time

Yes, yes, very useful!

Two things:

1) Consider a client and server that have poor time synchronization.
The client wouldn't be able to make much sense of the server's absolute
timestamp.  Would it be possible to also provide an absolute timestamp
of the start of audio (from the server's perspective) so that the client
could determine with certainty how far in to the audio sample the start
of speech occurred?

2) What about audio samples provided by input-waveform-uri?  In our
case, we're much more interested in the sample offset where start and
end of speech occured; not so much interested in the absolute time when
they occurred.

Corby Anderson
Tellme Networks, Inc.

Andrew Wahbe wrote: 

	I agree that this is useful. I can also think of many instances
where an accurate determination of the "end of input" time would be
useful as well. We should consider if the emma:start and emma:end
attributes are sufficient though. Or do you think there is a reason to
have this information (well the start time) before the recognition
completes?

________________________________

	From: Dave Burke [mailto:[email-masked]] 
	Sent: July 6, 2006 7:01 AM
	To: [email-masked]
	Subject: [speechsc] Accurate determination of START-OF-INPUT
time

	It is useful to know the START-OF-INPUT time in many speech
applications (e.g. to be able to resume from where one left off or to
offset from the point of barge-in, e.g. "skip forward two seconds").
While this can be *estimated* from the time of receipt of the event, it
's accuracy is subject to network transit times.  We've got an NTP
timestamp for SPEECH-MARKER and I believe we should also have one for
START-OF-INPUT. Note that, implementation of VoiceXML's marktime
requires one subtract the last SPEECH-MARKER time from the
START-OF-INPUT time.
	 
	We could simply add a ;timestamp attribute to Input-Type a la
Speech-Marker. 
	 
	Dave
	
________________________________

	_______________________________________________
	Speechsc mailing list
	[email-masked]
	https://www1.ietf.org/mailman/listinfo/speechsc

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: Brett Gavagni <[email-masked]>
Date: Fri, 7 Jul 2006 11:05:45 -0400

I agree, a client would probably obtain the most usefulness from a 
"Timestamp" value which represents the point in time w.r.t client's RTP 
stream fed to the server for the recognition request. 

I would also suggest that if this request is added to a future version of 
the MRCPv2 draft that the wording clearly articulates/references the 
wording in the RTP spec.

Specifically, some MRCPv2 draft readers may interpret the value as clock 
time, and not the terminology defined in RFC 3550.

>From RFC 3550:
   timestamp: 32 bits
      The timestamp reflects the sampling instant of the first octet in
      the RTP data packet.  The sampling instant MUST be derived from a
      clock that increments monotonically and linearly in time to allow
      synchronization and jitter calculations (see Section 6.4.1).  The
      resolution of the clock MUST be sufficient for the desired
      synchronization accuracy and for measuring packet arrival jitter
      (one tick per video frame is typically not sufficient).  The clock
      frequency is dependent on the format of data carried as payload
      and is specified statically in the profile or payload format
      specification that defines the format, or MAY be specified
      dynamically for payload formats defined through non-RTP means.  If
      RTP packets are generated periodically, the nominal sampling
      instant as determined from the sampling clock is to be used, not a
      reading of the system clock.  As an example, for fixed-rate audio
      the timestamp clock would likely increment by one for each
      sampling period.  If an audio application reads blocks covering
      160 sampling periods from the input device, the timestamp would be
      increased by 160 for each such block, regardless of whether the
      block is transmitted in a packet or dropped as silent.

Thanks,

Brett Gavagni 
WebSphere Voice Server Development 
http://www-306.ibm.com/software/pervasive/voice_server/
(561) 862-2097 T/L (975) 
[email-masked]

"Andrew Wahbe" <[email-masked]> 
07/07/2006 10:44 AM

To
"Corby Anderson" <[email-masked]>
cc
[email-masked], Dave Burke <[email-masked]>
Subject
RE: [speechsc] Accurate determination of START-OF-INPUT time

Right -- actually, I'd imagine by suggesting an NTP timestamp, Dave was 
implying that this is the timestamp in the RTP stream where start of 
speech was detected. As the input-waveform-uri begins at the start of the 
RECOGNIZE, I can see how absolute times don't work for what you are 
proposing unless you also have the timestamp of when recognition started.
 
How about the following:
A "Timestamp" header is added that has a timestamp as its value.
The header may appear on a response or an event. On a response, it 
indicates the time in the stream at which the associated request was 
applied (e.g. when did the RECOGNIZE start). On an event, it indicates the 
time at which the event occured (e.g. when did start of speech occur).
 
This is a more general solution to what is being done with the 
speech-marker (see: 
http://www1.ietf.org/mail-archive/web/speechsc/current/msg01813.html) If 
we really want to know when these things are happening in relation to the 
RTP stream in general then I think a separate timestamp header makes sense 
rather than sticking the timestamp information in different headers for 
each resource or message type.
 
Now there is the question of what format to use for the timestamp. It 
seems what we are really interested in is the "point of time" in the 
stream as opposed to the clock time. In the discussion related to the 
speech marker timestamp it was pointed out that the RTP time can be 
derived from the NTP time using an RTCP sender report (which includes the 
RTP and NTP time when the report was sent). However, it is common practice 
for "bursty" or faster-than-real-time transmission of RTP between voice 
platforms and voice resources. For example, this can be done in synthesis 
to allow telephony card buffers to be filled up, and in recognition if the 
voice platform buffers audio while a speech session is established. 
Doesn't this destroy the relationship between NTP time and the RTP 
timestamp? I think it makes more sense to use the RTP timestamp here.

Andrew Wahbe

Yes, yes, very useful!

Two things:

1) Consider a client and server that have poor time synchronization.  The 
client wouldn't be able to make much sense of the server's absolute 
timestamp.  Would it be possible to also provide an absolute timestamp of 
the start of audio (from the server's perspective) so that the client 
could determine with certainty how far in to the audio sample the start of 
speech occurred?

2) What about audio samples provided by input-waveform-uri?  In our case, 
we're much more interested in the sample offset where start and end of 
speech occured; not so much interested in the absolute time when they 
occurred.

Corby Anderson
Tellme Networks, Inc.

Andrew Wahbe wrote: 
I agree that this is useful. I can also think of many instances where an 
accurate determination of the "end of input" time would be useful as well. 
We should consider if the emma:start and emma:end attributes are 
sufficient though. Or do you think there is a reason to have this 
information (well the start time) before the recognition completes?

From: Dave Burke [mailto:[email-masked]] 
Sent: July 6, 2006 7:01 AM
To: [email-masked]
Subject: [speechsc] Accurate determination of START-OF-INPUT time

It is useful to know the START-OF-INPUT time in many speech applications 
(e.g. to be able to resume from where one left off or to offset from the 
point of barge-in, e.g. "skip forward two seconds"). While this can be 
*estimated* from the time of receipt of the event, it 's accuracy is 
subject to network transit times.  We've got an NTP timestamp for 
SPEECH-MARKER and I believe we should also have one for START-OF-INPUT. 
Note that, implementation of VoiceXML's marktime requires one subtract the 
last SPEECH-MARKER time from the START-OF-INPUT time.

Dave

---

From: "Dave Burke" <[email-masked]>
Date: Fri, 7 Jul 2006 16:43:08 +0100

I'm in support of a separate Timestamp header. 

On the subject of fast-than-real-time RTP. For stored data, the NTP value in the RTCP SR indicates the presentation time (when the sample indicated by the RTP timestamp is current)  and so if you speed up transmission rates, you accelerate the two clocks. 

Dave
  ----- Original Message ----- 
  From: Andrew Wahbe 
  To: Corby Anderson 
  Cc: Dave Burke ; [email-masked] 
  Sent: Friday, July 07, 2006 3:44 PM
  Subject: RE: [speechsc] Accurate determination of START-OF-INPUT time

  Right -- actually, I'd imagine by suggesting an NTP timestamp, Dave was implying that this is the timestamp in the RTP stream where start of speech was detected. As the input-waveform-uri begins at the start of the RECOGNIZE, I can see how absolute times don't work for what you are proposing unless you also have the timestamp of when recognition started.

  How about the following:
  A "Timestamp" header is added that has a timestamp as its value.
  The header may appear on a response or an event. On a response, it indicates the time in the stream at which the associated request was applied (e.g. when did the RECOGNIZE start). On an event, it indicates the time at which the event occured (e.g. when did start of speech occur).

  This is a more general solution to what is being done with the speech-marker (see: http://www1.ietf.org/mail-archive/web/speechsc/current/msg01813.html) If we really want to know when these things are happening in relation to the RTP stream in general then I think a separate timestamp header makes sense rather than sticking the timestamp information in different headers for each resource or message type.

  Now there is the question of what format to use for the timestamp. It seems what we are really interested in is the "point of time" in the stream as opposed to the clock time. In the discussion related to the speech marker timestamp it was pointed out that the RTP time can be derived from the NTP time using an RTCP sender report (which includes the RTP and NTP time when the report was sent). However, it is common practice for "bursty" or faster-than-real-time transmission of RTP between voice platforms and voice resources. For example, this can be done in synthesis to allow telephony card buffers to be filled up, and in recognition if the voice platform buffers audio while a speech session is established. Doesn't this destroy the relationship between NTP time and the RTP timestamp? I think it makes more sense to use the RTP timestamp here.

  Andrew Wahbe

------------------------------------------------------------------------------

  Yes, yes, very useful!

  Two things:

  1) Consider a client and server that have poor time synchronization.  The client wouldn't be able to make much sense of the server's absolute timestamp.  Would it be possible to also provide an absolute timestamp of the start of audio (from the server's perspective) so that the client could determine with certainty how far in to the audio sample the start of speech occurred?

  2) What about audio samples provided by input-waveform-uri?  In our case, we're much more interested in the sample offset where start and end of speech occured; not so much interested in the absolute time when they occurred.

  Corby Anderson
  Tellme Networks, Inc.

  Andrew Wahbe wrote: 
    I agree that this is useful. I can also think of many instances where an accurate determination of the "end of input" time would be useful as well. We should consider if the emma:start and emma:end attributes are sufficient though. Or do you think there is a reason to have this information (well the start time) before the recognition completes?

----------------------------------------------------------------------------
    From: Dave Burke [mailto:[email-masked]] 
    Sent: July 6, 2006 7:01 AM
    To: [email-masked]
    Subject: [speechsc] Accurate determination of START-OF-INPUT time

    It is useful to know the START-OF-INPUT time in many speech applications (e.g. to be able to resume from where one left off or to offset from the point of barge-in, e.g. "skip forward two seconds"). While this can be *estimated* from the time of receipt of the event, it 's accuracy is subject to network transit times.  We've got an NTP timestamp for SPEECH-MARKER and I believe we should also have one for START-OF-INPUT. Note that, implementation of VoiceXML's marktime requires one subtract the last SPEECH-MARKER time from the START-OF-INPUT time.

    We could simply add a ;timestamp attribute to Input-Type a la Speech-Marker. 

    Dave
----------------------------------------------------------------------------
_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc