Subject: RE: [speechsc] Accurate determination of START-OF-INPUT time

From: "Andrew Wahbe" <[email-masked]>
Date: Fri, 7 Jul 2006 10:37:14 -0700

While this makes sense, is it called out in rfc 3550? It seems to be
written from the perspective that the content is real-time, and not
stored or being sent at accelerated rates. It stresses that the most
common wallclock should be preferred over session elapsed time. However,
since the stated intent of the NTP timestamp is for synchronization I
think the behavior you suggest for sped-up transmission makes sense. If
we are going to depend on this behavior though, I think we should call
it out in the MRCP v2 spec (unless this is called out somewhere already
and I missed it).
 
Also, what if you have not received an RTCP SR before you receive a
timestamp? (Dave, you raised this already in the previous speech marker
discussion.
http://www1.ietf.org/mail-archive/web/speechsc/current/msg01821.html)
Isn't the relevance of this issue is increased if we start using the NTP
timestamps more broadly. The MRCP recognizer resource can't even compute
the NTP timestamp if it hasn't got the SR from the client yet.
 
Andrew

________________________________

From: Dave Burke [mailto:[email-masked]] 
Sent: July 7, 2006 11:43 AM
To: Andrew Wahbe; Corby Anderson
Cc: [email-masked]
Subject: Re: [speechsc] Accurate determination of START-OF-INPUT time

I'm in support of a separate Timestamp header. 
 
On the subject of fast-than-real-time RTP. For stored data, the NTP
value in the RTCP SR indicates the presentation time (when the sample
indicated by the RTP timestamp is current)  and so if you speed up
transmission rates, you accelerate the two clocks. 
 
Dave

	----- Original Message ----- 
	From: Andrew Wahbe <mailto:[email-masked]>  
	To: Corby Anderson <mailto:[email-masked]>  
	Cc: Dave Burke <mailto:[email-masked]>  ;
[email-masked] 
	Sent: Friday, July 07, 2006 3:44 PM
	Subject: RE: [speechsc] Accurate determination of START-OF-INPUT
time

	Right -- actually, I'd imagine by suggesting an NTP timestamp,
Dave was implying that this is the timestamp in the RTP stream where
start of speech was detected. As the input-waveform-uri begins at the
start of the RECOGNIZE, I can see how absolute times don't work for what
you are proposing unless you also have the timestamp of when recognition
started.
	 
	How about the following:
	A "Timestamp" header is added that has a timestamp as its value.
	The header may appear on a response or an event. On a response,
it indicates the time in the stream at which the associated request was
applied (e.g. when did the RECOGNIZE start). On an event, it indicates
the time at which the event occured (e.g. when did start of speech
occur).
	 
	This is a more general solution to what is being done with the
speech-marker (see:
http://www1.ietf.org/mail-archive/web/speechsc/current/msg01813.html) If
we really want to know when these things are happening in relation to
the RTP stream in general then I think a separate timestamp header makes
sense rather than sticking the timestamp information in different
headers for each resource or message type.
	 
	Now there is the question of what format to use for the
timestamp. It seems what we are really interested in is the "point of
time" in the stream as opposed to the clock time. In the discussion
related to the speech marker timestamp it was pointed out that the RTP
time can be derived from the NTP time using an RTCP sender report (which
includes the RTP and NTP time when the report was sent). However, it is
common practice for "bursty" or faster-than-real-time transmission of
RTP between voice platforms and voice resources. For example, this can
be done in synthesis to allow telephony card buffers to be filled up,
and in recognition if the voice platform buffers audio while a speech
session is established. Doesn't this destroy the relationship between
NTP time and the RTP timestamp? I think it makes more sense to use the
RTP timestamp here.

	Andrew Wahbe
	
________________________________

	From: Corby Anderson [mailto:[email-masked]] 
	Sent: July 6, 2006 1:59 PM
	To: Andrew Wahbe
	Cc: Dave Burke; [email-masked]
	Subject: Re: [speechsc] Accurate determination of START-OF-INPUT
time

	Yes, yes, very useful!
	
	Two things:
	
	1) Consider a client and server that have poor time
synchronization.  The client wouldn't be able to make much sense of the
server's absolute timestamp.  Would it be possible to also provide an
absolute timestamp of the start of audio (from the server's perspective)
so that the client could determine with certainty how far in to the
audio sample the start of speech occurred?
	
	2) What about audio samples provided by input-waveform-uri?  In
our case, we're much more interested in the sample offset where start
and end of speech occured; not so much interested in the absolute time
when they occurred.
	
	Corby Anderson
	Tellme Networks, Inc.

	Andrew Wahbe wrote: 

		I agree that this is useful. I can also think of many
instances where an accurate determination of the "end of input" time
would be useful as well. We should consider if the emma:start and
emma:end attributes are sufficient though. Or do you think there is a
reason to have this information (well the start time) before the
recognition completes?

________________________________

		From: Dave Burke [mailto:[email-masked]] 
		Sent: July 6, 2006 7:01 AM
		To: [email-masked]
		Subject: [speechsc] Accurate determination of
START-OF-INPUT time

		It is useful to know the START-OF-INPUT time in many
speech applications (e.g. to be able to resume from where one left off
or to offset from the point of barge-in, e.g. "skip forward two
seconds"). While this can be *estimated* from the time of receipt of the
event, it 's accuracy is subject to network transit times.  We've got an
NTP timestamp for SPEECH-MARKER and I believe we should also have one
for START-OF-INPUT. Note that, implementation of VoiceXML's marktime
requires one subtract the last SPEECH-MARKER time from the
START-OF-INPUT time.
		 
		We could simply add a ;timestamp attribute to Input-Type
a la Speech-Marker. 
		 
		Dave
		
________________________________

		_______________________________________________
		Speechsc mailing list
		[email-masked]
		https://www1.ietf.org/mailman/listinfo/speechsc

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: "Dave Burke" <[email-masked]>
Date: Sat, 8 Jul 2006 12:01:10 +0100

RFC 3550 is certainly written from the perspective of real-time sampled content so you need to "read between the lines" a little to infer faster-than-real-time behaviour. I'm not sure we should really attempt to specify this kind of behaviour, however -  seems like a potential can of worms. I would feel more comfortable if left it as an implementation detail...

The transmission time of SR packets can be problematic both from an immediacy and reliability point of view. A practical solution is to send send an SR packet immediately on initiation of a new media stream though there is no guarantee that that packet doesn't get lost. RTSP addresses this through the rtptime parameter in the RTP-Info header, for example...

However, if we use timestamps relatively (e.g. we report the timestamp when recognition or playback started - i.e. IN-PROGRESS -, the instant of barge-in, the occurrence of a speech marker) then we don't need to worry about correlating with the RTP timestamp. Indeed, we can't assume the MRCP client has access to the media source/sink (hence RTP timestamps) in any meaningful way...

Dave

  ----- Original Message ----- 
  From: Andrew Wahbe 
  To: Dave Burke ; Corby Anderson 
  Cc: [email-masked] 
  Sent: Friday, July 07, 2006 6:37 PM
  Subject: RE: [speechsc] Accurate determination of START-OF-INPUT time

  While this makes sense, is it called out in rfc 3550? It seems to be written from the perspective that the content is real-time, and not stored or being sent at accelerated rates. It stresses that the most common wallclock should be preferred over session elapsed time. However, since the stated intent of the NTP timestamp is for synchronization I think the behavior you suggest for sped-up transmission makes sense. If we are going to depend on this behavior though, I think we should call it out in the MRCP v2 spec (unless this is called out somewhere already and I missed it).

  Also, what if you have not received an RTCP SR before you receive a timestamp? (Dave, you raised this already in the previous speech marker discussion. http://www1.ietf.org/mail-archive/web/speechsc/current/msg01821.html) Isn't the relevance of this issue is increased if we start using the NTP timestamps more broadly. The MRCP recognizer resource can't even compute the NTP timestamp if it hasn't got the SR from the client yet.

  Andrew

------------------------------------------------------------------------------

  On the subject of fast-than-real-time RTP. For stored data, the NTP value in the RTCP SR indicates the presentation time (when the sample indicated by the RTP timestamp is current)  and so if you speed up transmission rates, you accelerate the two clocks. 

  Dave
    ----- Original Message ----- 
    From: Andrew Wahbe 
    To: Corby Anderson 
    Cc: Dave Burke ; [email-masked] 
    Sent: Friday, July 07, 2006 3:44 PM
    Subject: RE: [speechsc] Accurate determination of START-OF-INPUT time

    Right -- actually, I'd imagine by suggesting an NTP timestamp, Dave was implying that this is the timestamp in the RTP stream where start of speech was detected. As the input-waveform-uri begins at the start of the RECOGNIZE, I can see how absolute times don't work for what you are proposing unless you also have the timestamp of when recognition started.

    How about the following:
    A "Timestamp" header is added that has a timestamp as its value.
    The header may appear on a response or an event. On a response, it indicates the time in the stream at which the associated request was applied (e.g. when did the RECOGNIZE start). On an event, it indicates the time at which the event occured (e.g. when did start of speech occur).

    This is a more general solution to what is being done with the speech-marker (see: http://www1.ietf.org/mail-archive/web/speechsc/current/msg01813.html) If we really want to know when these things are happening in relation to the RTP stream in general then I think a separate timestamp header makes sense rather than sticking the timestamp information in different headers for each resource or message type.

    Now there is the question of what format to use for the timestamp. It seems what we are really interested in is the "point of time" in the stream as opposed to the clock time. In the discussion related to the speech marker timestamp it was pointed out that the RTP time can be derived from the NTP time using an RTCP sender report (which includes the RTP and NTP time when the report was sent). However, it is common practice for "bursty" or faster-than-real-time transmission of RTP between voice platforms and voice resources. For example, this can be done in synthesis to allow telephony card buffers to be filled up, and in recognition if the voice platform buffers audio while a speech session is established. Doesn't this destroy the relationship between NTP time and the RTP timestamp? I think it makes more sense to use the RTP timestamp here.

    Andrew Wahbe

----------------------------------------------------------------------------

    Yes, yes, very useful!

    Two things:

    1) Consider a client and server that have poor time synchronization.  The client wouldn't be able to make much sense of the server's absolute timestamp.  Would it be possible to also provide an absolute timestamp of the start of audio (from the server's perspective) so that the client could determine with certainty how far in to the audio sample the start of speech occurred?

    2) What about audio samples provided by input-waveform-uri?  In our case, we're much more interested in the sample offset where start and end of speech occured; not so much interested in the absolute time when they occurred.

    Corby Anderson
    Tellme Networks, Inc.

    Andrew Wahbe wrote: 
      I agree that this is useful. I can also think of many instances where an accurate determination of the "end of input" time would be useful as well. We should consider if the emma:start and emma:end attributes are sufficient though. Or do you think there is a reason to have this information (well the start time) before the recognition completes?

--------------------------------------------------------------------------
      From: Dave Burke [mailto:[email-masked]] 
      Sent: July 6, 2006 7:01 AM
      To: [email-masked]
      Subject: [speechsc] Accurate determination of START-OF-INPUT time

      It is useful to know the START-OF-INPUT time in many speech applications (e.g. to be able to resume from where one left off or to offset from the point of barge-in, e.g. "skip forward two seconds"). While this can be *estimated* from the time of receipt of the event, it 's accuracy is subject to network transit times.  We've got an NTP timestamp for SPEECH-MARKER and I believe we should also have one for START-OF-INPUT. Note that, implementation of VoiceXML's marktime requires one subtract the last SPEECH-MARKER time from the START-OF-INPUT time.

      We could simply add a ;timestamp attribute to Input-Type a la Speech-Marker. 

      Dave
--------------------------------------------------------------------------
_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc