Subject: RE: [Speechsc] Managing big grammars

From: "Shanmugham, Saravanan" <[email-masked]>
Date: Tue, 14 Jun 2005 04:04:21 -0700

cool. That makes sense.
The way I though this problem would have been addressed, is the MRCP
caching system to be smart.
That is everytime there is a request a grammar for a grammar URI. the
MRCP server would do a fetch to check if the document has changed. If it
has, then it would download and compile the grammar. While this
happenning existing calls that were using the older compiled grammar
continue to use it. Once this is ready, the previous version of the
grammar is allowed to expire-out.
 
Would this not address your scenario and avoid having to do this work
around you are suggesting?
 
Sarvi 

________________________________

	From: Dave Burke [mailto:[email-masked]] 
	Sent: Tuesday, June 14, 2005 3:16 AM
	To: Shanmugham, Saravanan; Wyss, Felix; Corby Anderson;
[email-masked]
	Subject: Re: [Speechsc] Managing big grammars

	It's not that the URL needs a specific date or time, just that
it has a unique name. This is simply so that you can ensure the new
grammar is downloaded, compiled, and loaded ahead of any calls. Once
compiled and loaded, the application grammar URIs can be updated to
reference the new grammar. This way you avoid the danger that a bunch of
calls, which arrive at the same time as the DEFINE-GRAMMAR is happening,
experience delays.
	 
	Dave

		----- Original Message ----- 
		From: Shanmugham, Saravanan <mailto:[email-masked]>  
		To: Dave Burke <mailto:[email-masked]>  ;
Wyss, Felix <mailto:[email-masked]>  ; Corby Anderson
<mailto:[email-masked]>  ; [email-masked] 
		Sent: Tuesday, June 14, 2005 7:58 AM
		Subject: RE: [Speechsc] Managing big grammars

		I dont understand why the RUL has to encode and date and
time into it. Lets say we are refering to large_grammar.grxml
		 
		When a client does a DEFINE-GRAMMAR on the MRCP server
with this URI. The MRCP server is expected to go download this URI and
compile it. The caching behaviour of this URI and how long it lives in
the cache is dependent on what is returned by the HTTP GET operation
that brought the grammar into the MRCP server. In addition, the
behaviour is also governed by Cache -Control parameters that the client
can specify for the DEFINE-GRAMMAR that would take precedence over what
was returned by the HTTP GET operation. This means that the MRCP client
can force the MRCP server to ignore the cache parameters it got from the
grammar webserver.
		 
		So, for the large_grammar.grxml, if the webserver
returned a large max-age value the grammar and its compiled version in
the cache of the MRCP server should not get timedout. For any future
requests for that grammar, MRCP server should go do fetch of the
document, compare the max-age of the fetched coument and make sure the
webserver says that the document in the cache is not out-of-date. As
long as the grammar document in the cache is not out of date, you don't
need to compile it again and you are fine. If the propoerty of the
large_grammar.grxml on the webserver had changed then the MRCP server
would be expected to refresh the cache and compile the newly downloaded
grammar.

		Am I missing something here.
		 
		Sarvi 

________________________________

			From: [email-masked]
[mailto:[email-masked]] On Behalf Of Dave Burke
			Sent: Sunday, June 12, 2005 12:12 PM
			To: Wyss, Felix; Shanmugham, Saravanan; Corby
Anderson; [email-masked]
			Subject: Re: [Speechsc] Managing big grammars

			There is no doubt that grammar compilation and
loading must be hidden from the caller experience. What is in doubt is
whether the MRCP protocol needs to expose such control. 
			 
			The grammar loading problem is not dissimilar to
cache seeding in the Web world. Taking your use-case, I would implement
this in MRCP as a two-step process:
			 
			(1) Make the large grammar available through
HTTP and date-stamp the URI, e.g.
http://example.com/large_grammar_12June2005.grxml. Set the HTTP response
header Cache-Control: max-age to a value larger than the grammar change
period.
			 
			(2) When a new grammar is required, open an MRCP
channel to each of the MRCP servers. Issue a DEFINE-GRAMMAR with the
relevant URI e.g. http://example.com/large_grammar_13June2005.grxml.
Update the application to point at the new grammar.
			 
			[For a mid-size grammar where loading latency is
negligible, and assuming a centrally located compiled grammar cache,
step (2) might only require a DEFINE-GRAMMAR to one MRCP server].
			 
			You are correct that the current MRCPv2 draft
scopes the DEFINE-GRAMMAR to a channel and this is a concern. However,
this is resolved with Sarvi's suggested text change.
			 
			Dave

				----- Original Message ----- 
				From: Wyss, Felix
<mailto:[email-masked]>  
				To: Dave Burke
<mailto:[email-masked]>  ; Shanmugham, Saravanan
<mailto:[email-masked]>  ; Corby Anderson <mailto:[email-masked]>  ;
[email-masked] 
				Sent: Sunday, June 12, 2005 1:41 AM
				Subject: RE: [Speechsc] Managing big
grammars

				I disagree.  Predictability is *very*
critical for an application using ASR.  It must be able to make sure
that when a recognition operation is started, there is no latency while
the ASR server compiles and/or loads a large precompiled grammar into
memory.  The DEFINE-GRAMMAR command is scoped to a channel, which means
by the time the application has a channel (i.e. a caller is in the
system), it is unacceptable to block for more than fraction of a second
while the grammars are being compiled and/or loaded.  Otherwise, the
first caller after a grammar is changed encounters delays and has a poor
user experience or even hangs up.  Somebody may say: well, just make a
"priming" call after modifying the grammar to ensure the grammar is
cached.  However, that is not very practical for automatically generated
grammars; but worse yet, in large systems there are commonly several
load-balanced ASR severs and it will be rather hard to ensure each of
them is touched at least once.  

				The way we solve this on our platform is
as follows: Large grammars that may change frequently, possibly several
times a day, such as company directory grammars that are generated from
a database, can be registered as "preloaded grammars".  This means, our
platform ensures the grammar is compiled, distributed to all ASR
servers, and loaded into their caches after a new version of that
grammar has been generated.  Each preloaded grammar has a name
associated, which is referenced through a special URI.  The application
logic only knows that URI and not the actual URI/filename of the
underlying grammar data.  The subsystem managing the preloaded grammars
ensures that the preloaded grammar URI resolves to the previous version
of the grammar until the new grammar is loaded and ready on all ASR
servers.  Thus, all callers that call into the system while the grammar
is being generated, compiled, and loaded on the servers will get the
previous version.  This ensures that there is no delay for the first
caller after a change.  We can do this as we are using the native APIs
provided by the ASR engine vendors we support and layer a sophisticated
framework and API on top of it.  

				It would be nice if MRCP provided some
support for schemes like that, without having to open dummy channels to
register grammars and pray the MRCP server doesn't evict them too early.
An MRCP client has to be able to guarantee that when a call comes
through the system, everything is ready and it won't experience any
delay.  As it stands now the MRCP protocol does not appear to provide
support for this.

				Thanks,

				Felix

________________________________

				From: [email-masked]
[mailto:[email-masked]] On Behalf Of Dave Burke
				Sent: Saturday, June 11, 2005 10:42
				To: Shanmugham, Saravanan; Corby
Anderson; [email-masked]
				Subject: Re: [Speechsc] Managing big
grammars

				I agree. Grammar compilation, caching,
and sharing is a problem for the MRCP server. Exposing this in the
protocol puts unnecessary complexity on the client. For example, the
client needs to download grammars, implement its own HTTP caching,
checksum algorithms (some way of not having to push huge data to the
MRCP server to query) etc.

				So I vote for point 1 below and include
in the text that an MRCP server could also, as an optimisation,
implement a centralised common cache shared across multiple MRCP
servers.

				Dave

				----- Original Message ----- 

				From: Shanmugham, Saravanan
<mailto:[email-masked]>  

				To: Corby Anderson
<mailto:[email-masked]>  ; [email-masked] 

				Sent: Saturday, June 11, 2005 12:09 AM

				Subject: RE: [Speechsc] Managing big
grammars

				I would say that a common cache amoung a
farm of MRCP servers to optimize compilations would be a nice server
side optimization and shouldn't affect the MRCP protocol.

				my 2c.

				Sarvi

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: "Dave Burke" <[email-masked]>
Date: Tue, 14 Jun 2005 13:16:20 +0100

That optimisation would be nice alright and is not restricted by HTTP (see note from RFC 2616 below). 

It seems like we have got some good optimisation suggestions (read: informative) that could go into the spec to address large grammars. If memory serves, they are:
    1. An MRCP server may cache grammars used in DEFINE-GRAMMAR or RECOGNIZE to be used over multiple MRCP sessions
    2. An MRCP server may cache the compiled version of a grammar
    3. An MRCP server may cache grammars centrally to allow other servers take advantage of the cache
    4. An MRCP server, in response to a DEFINE-GRAMMAR, may inform other MRCP servers to reload the new grammar into memory
    5. An MRCP server may choose to return a stale response while a new grammar is being downloaded, compiled, and cached

-- Dave

>From RFC 2616:

   In some cases, the operator of a cache MAY choose to configure it to
   return stale responses even when not requested by clients. This
   decision ought not be made lightly, but may be necessary for reasons
   of availability or performance, especially when the cache is poorly
   connected to the origin server. Whenever a cache returns a stale
   response, it MUST mark it as such (using a Warning header) enabling
   the client software to alert the user that there might be a potential
   problem.

----- Original Message ----- 
  From: Shanmugham, Saravanan 
  To: Dave Burke ; Wyss, Felix ; Corby Anderson ; [email-masked] 
  Sent: Tuesday, June 14, 2005 12:04 PM
  Subject: RE: [Speechsc] Managing big grammars

  cool. That makes sense.
  The way I though this problem would have been addressed, is the MRCP caching system to be smart.
  That is everytime there is a request a grammar for a grammar URI. the MRCP server would do a fetch to check if the document has changed. If it has, then it would download and compile the grammar. While this happenning existing calls that were using the older compiled grammar continue to use it. Once this is ready, the previous version of the grammar is allowed to expire-out.

  Would this not address your scenario and avoid having to do this work around you are suggesting?

  Sarvi 

----------------------------------------------------------------------------
    From: Dave Burke [mailto:[email-masked]] 
    Sent: Tuesday, June 14, 2005 3:16 AM
    To: Shanmugham, Saravanan; Wyss, Felix; Corby Anderson; [email-masked]
    Subject: Re: [Speechsc] Managing big grammars

    It's not that the URL needs a specific date or time, just that it has a unique name. This is simply so that you can ensure the new grammar is downloaded, compiled, and loaded ahead of any calls. Once compiled and loaded, the application grammar URIs can be updated to reference the new grammar. This way you avoid the danger that a bunch of calls, which arrive at the same time as the DEFINE-GRAMMAR is happening, experience delays.

    Dave
      ----- Original Message ----- 
      From: Shanmugham, Saravanan 
      To: Dave Burke ; Wyss, Felix ; Corby Anderson ; [email-masked] 
      Sent: Tuesday, June 14, 2005 7:58 AM
      Subject: RE: [Speechsc] Managing big grammars

      I dont understand why the RUL has to encode and date and time into it. Lets say we are refering to large_grammar.grxml

      When a client does a DEFINE-GRAMMAR on the MRCP server with this URI. The MRCP server is expected to go download this URI and compile it. The caching behaviour of this URI and how long it lives in the cache is dependent on what is returned by the HTTP GET operation that brought the grammar into the MRCP server. In addition, the behaviour is also governed by Cache -Control parameters that the client can specify for the DEFINE-GRAMMAR that would take precedence over what was returned by the HTTP GET operation. This means that the MRCP client can force the MRCP server to ignore the cache parameters it got from the grammar webserver.

      So, for the large_grammar.grxml, if the webserver returned a large max-age value the grammar and its compiled version in the cache of the MRCP server should not get timedout. For any future requests for that grammar, MRCP server should go do fetch of the document, compare the max-age of the fetched coument and make sure the webserver says that the document in the cache is not out-of-date. As long as the grammar document in the cache is not out of date, you don't need to compile it again and you are fine. If the propoerty of the large_grammar.grxml on the webserver had changed then the MRCP server would be expected to refresh the cache and compile the newly downloaded grammar.

      Sarvi 

------------------------------------------------------------------------
        From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave Burke
        Sent: Sunday, June 12, 2005 12:12 PM
        To: Wyss, Felix; Shanmugham, Saravanan; Corby Anderson; [email-masked]
        Subject: Re: [Speechsc] Managing big grammars

        There is no doubt that grammar compilation and loading must be hidden from the caller experience. What is in doubt is whether the MRCP protocol needs to expose such control. 

        The grammar loading problem is not dissimilar to cache seeding in the Web world. Taking your use-case, I would implement this in MRCP as a two-step process:

        (1) Make the large grammar available through HTTP and date-stamp the URI, e.g. http://example.com/large_grammar_12June2005.grxml. Set the HTTP response header Cache-Control: max-age to a value larger than the grammar change period.

        (2) When a new grammar is required, open an MRCP channel to each of the MRCP servers. Issue a DEFINE-GRAMMAR with the relevant URI e.g. http://example.com/large_grammar_13June2005.grxml. Update the application to point at the new grammar.

        [For a mid-size grammar where loading latency is negligible, and assuming a centrally located compiled grammar cache, step (2) might only require a DEFINE-GRAMMAR to one MRCP server].

        You are correct that the current MRCPv2 draft scopes the DEFINE-GRAMMAR to a channel and this is a concern. However, this is resolved with Sarvi's suggested text change.

        Dave
          ----- Original Message ----- 
          From: Wyss, Felix 
          To: Dave Burke ; Shanmugham, Saravanan ; Corby Anderson ; [email-masked] 
          Sent: Sunday, June 12, 2005 1:41 AM
          Subject: RE: [Speechsc] Managing big grammars

          I disagree.  Predictability is *very* critical for an application using ASR.  It must be able to make sure that when a recognition operation is started, there is no latency while the ASR server compiles and/or loads a large precompiled grammar into memory.  The DEFINE-GRAMMAR command is scoped to a channel, which means by the time the application has a channel (i.e. a caller is in the system), it is unacceptable to block for more than fraction of a second while the grammars are being compiled and/or loaded.  Otherwise, the first caller after a grammar is changed encounters delays and has a poor user experience or even hangs up.  Somebody may say: well, just make a "priming" call after modifying the grammar to ensure the grammar is cached.  However, that is not very practical for automatically generated grammars; but worse yet, in large systems there are commonly several load-balanced ASR severs and it will be rather hard to ensure each of them is touched at least once.  

          The way we solve this on our platform is as follows: Large grammars that may change frequently, possibly several times a day, such as company directory grammars that are generated from a database, can be registered as "preloaded grammars".  This means, our platform ensures the grammar is compiled, distributed to all ASR servers, and loaded into their caches after a new version of that grammar has been generated.  Each preloaded grammar has a name associated, which is referenced through a special URI.  The application logic only knows that URI and not the actual URI/filename of the underlying grammar data.  The subsystem managing the preloaded grammars ensures that the preloaded grammar URI resolves to the previous version of the grammar until the new grammar is loaded and ready on all ASR servers.  Thus, all callers that call into the system while the grammar is being generated, compiled, and loaded on the servers will get the previous version.  This ensures that there is no delay for the first caller after a change.  We can do this as we are using the native APIs provided by the ASR engine vendors we support and layer a sophisticated framework and API on top of it.  

          It would be nice if MRCP provided some support for schemes like that, without having to open dummy channels to register grammars and pray the MRCP server doesn't evict them too early.  An MRCP client has to be able to guarantee that when a call comes through the system, everything is ready and it won't experience any delay.  As it stands now the MRCP protocol does not appear to provide support for this.

          Thanks,

          Felix

----------------------------------------------------------------------

          From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave Burke
          Sent: Saturday, June 11, 2005 10:42
          To: Shanmugham, Saravanan; Corby Anderson; [email-masked]
          Subject: Re: [Speechsc] Managing big grammars

          I agree. Grammar compilation, caching, and sharing is a problem for the MRCP server. Exposing this in the protocol puts unnecessary complexity on the client. For example, the client needs to download grammars, implement its own HTTP caching, checksum algorithms (some way of not having to push huge data to the MRCP server to query) etc.

          So I vote for point 1 below and include in the text that an MRCP server could also, as an optimisation, implement a centralised common cache shared across multiple MRCP servers.

          Dave

            ----- Original Message ----- 

            To: Corby Anderson ; [email-masked] 

            I would say that a common cache amoung a farm of MRCP servers to optimize compilations would be a nice server side optimization and shouldn't affect the MRCP protocol.

            my 2c.

            Sarvi

---

From: "David R. Oran" <[email-masked]>
Date: Tue, 14 Jun 2005 08:31:27 -0400

I like Dave's formulation. If there are no objections, I'm going to  
craft this into a sub-section on "Grammar Caching"  under the section  
on "Grammar Data". On looking where to put this, I noticed that the  
description is under 9.5.1 "recognizer Grammar Data" which is under  
"Recognizer message body". Some of that material isn't really about  
message body (since it talks about ow indirect content is handled),  
so I'm going to re-jigger things a bit to improve the flow and move  
the parts about indirect content to the new earlier section.

Speak now if you don't like this, because I'm going to do this later  
today...

On Jun 14, 2005, at 8:16 AM, Dave Burke wrote:

> It seems like we have got some good optimisation suggestions (read:  
> informative) that could go into the spec to address large grammars.  
> If memory serves, they are:
>     1. An MRCP server may cache grammars used in DEFINE-GRAMMAR or  
> RECOGNIZE to be used over multiple MRCP sessions
>     4. An MRCP server, in response to a DEFINE-GRAMMAR, may inform  
> other MRCP servers to reload the new grammar into memory
>     5. An MRCP server may choose to return a stale response while a  
> new grammar is being downloaded, compiled, and cached
>
> -- Dave
>
> From RFC 2616:
>
> That is everytime there is a request a grammar for a grammar URI.  
> the MRCP server would do a fetch to check if the document has  
> changed. If it has, then it would download and compile the grammar.  
> While this happenning existing calls that were using the older  
> compiled grammar continue to use it. Once this is ready, the  
> previous version of the grammar is allowed to expire-out.
>
> It's not that the URL needs a specific date or time, just that it  
> has a unique name. This is simply so that you can ensure the new  
> grammar is downloaded, compiled, and loaded ahead of any calls.  
> Once compiled and loaded, the application grammar URIs can be  
> updated to reference the new grammar. This way you avoid the danger  
> that a bunch of calls, which arrive at the same time as the DEFINE- 
> GRAMMAR is happening, experience delays.
>
> Dave
> I dont understand why the RUL has to encode and date and time into  
> it. Lets say we are refering to large_grammar.grxml
>
> When a client does a DEFINE-GRAMMAR on the MRCP server with this  
> URI. The MRCP server is expected to go download this URI and  
> compile it. The caching behaviour of this URI and how long it lives  
> in the cache is dependent on what is returned by the HTTP GET  
> operation that brought the grammar into the MRCP server. In  
> addition, the behaviour is also governed by Cache -Control  
> parameters that the client can specify for the DEFINE-GRAMMAR that  
> would take precedence over what was returned by the HTTP GET  
> operation. This means that the MRCP client can force the MRCP  
> server to ignore the cache parameters it got from the grammar  
> webserver.
>
> So, for the large_grammar.grxml, if the webserver returned a large  
> max-age value the grammar and its compiled version in the cache of  
> the MRCP server should not get timedout. For any future requests  
> for that grammar, MRCP server should go do fetch of the document,  
> compare the max-age of the fetched coument and make sure the  
> webserver says that the document in the cache is not out-of-date.  
> As long as the grammar document in the cache is not out of date,  
> you don't need to compile it again and you are fine. If the  
> propoerty of the large_grammar.grxml on the webserver had changed  
> then the MRCP server would be expected to refresh the cache and  
> compile the newly downloaded grammar.
>
>
> There is no doubt that grammar compilation and loading must be  
> hidden from the caller experience. What is in doubt is whether the  
> MRCP protocol needs to expose such control.
>
> The grammar loading problem is not dissimilar to cache seeding in  
> the Web world. Taking your use-case, I would implement this in MRCP  
> as a two-step process:
>
> (1) Make the large grammar available through HTTP and date-stamp  
> the URI, e.g. http://example.com/large_grammar_12June2005.grxml.  
> Set the HTTP response header Cache-Control: max-age to a value  
> larger than the grammar change period.
>
> (2) When a new grammar is required, open an MRCP channel to each of  
> the MRCP servers. Issue a DEFINE-GRAMMAR with the relevant URI e.g.  
> http://example.com/large_grammar_13June2005.grxml. Update the  
> application to point at the new grammar.
>
> [For a mid-size grammar where loading latency is negligible, and  
> assuming a centrally located compiled grammar cache, step (2) might  
> only require a DEFINE-GRAMMAR to one MRCP server].
>
> You are correct that the current MRCPv2 draft scopes the DEFINE- 
> GRAMMAR to a channel and this is a concern. However, this is  
> resolved with Sarvi's suggested text change.
>
> Dave
> I disagree.  Predictability is *very* critical for an application  
> using ASR.  It must be able to make sure that when a recognition  
> operation is started, there is no latency while the ASR server  
> compiles and/or loads a large precompiled grammar into memory.  The  
> DEFINE-GRAMMAR command is scoped to a channel, which means by the  
> time the application has a channel (i.e. a caller is in the  
> system), it is unacceptable to block for more than fraction of a  
> second while the grammars are being compiled and/or loaded.   
> Otherwise, the first caller after a grammar is changed encounters  
> delays and has a poor user experience or even hangs up.  Somebody  
> may say: well, just make a “priming” call after modifying the  
> grammar to ensure the grammar is cached.  However, that is not very  
> practical for automatically generated grammars; but worse yet, in  
> large systems there are commonly several load-balanced ASR severs  
> and it will be rather hard to ensure each of them is touched at  
> least once.
>
>
>
> The way we solve this on our platform is as follows: Large grammars  
> that may change frequently, possibly several times a day, such as  
> company directory grammars that are generated from a database, can  
> be registered as “preloaded grammars”.  This means, our platform  
> ensures the grammar is compiled, distributed to all ASR servers,  
> and loaded into their caches after a new version of that grammar  
> has been generated.  Each preloaded grammar has a name associated,  
> which is referenced through a special URI.  The application logic  
> only knows that URI and not the actual URI/filename of the  
> underlying grammar data.  The subsystem managing the preloaded  
> grammars ensures that the preloaded grammar URI resolves to the  
> previous version of the grammar until the new grammar is loaded and  
> ready on all ASR servers.  Thus, all callers that call into the  
> system while the grammar is being generated, compiled, and loaded  
> on the servers will get the previous version.  This ensures that  
> there is no delay for the first caller after a change.  We can do  
> this as we are using the native APIs provided by the ASR engine  
> vendors we support and layer a sophisticated framework and API on  
> top of it.
>
>
>
> It would be nice if MRCP provided some support for schemes like  
> that, without having to open dummy channels to register grammars  
> and pray the MRCP server doesn’t evict them too early.  An MRCP  
> client has to be able to guarantee that when a call comes through  
> the system, everything is ready and it won’t experience any delay.   
> As it stands now the MRCP protocol does not appear to provide  
> support for this.
>
>
>
> Thanks,
>
> Felix
>
>
>
> I agree. Grammar compilation, caching, and sharing is a problem for  
> the MRCP server. Exposing this in the protocol puts unnecessary  
> complexity on the client. For example, the client needs to download  
> grammars, implement its own HTTP caching, checksum algorithms (some  
> way of not having to push huge data to the MRCP server to query) etc.
>
>
>
> So I vote for point 1 below and include in the text that an MRCP  
> server could also, as an optimisation, implement a centralised  
> common cache shared across multiple MRCP servers.
>
>
>
> Dave
>
> I would say that a common cache amoung a farm of MRCP servers to  
> optimize compilations would be a nice server side optimization and  
> shouldn't affect the MRCP protocol.
>
>
>
> my 2c.
>
> Sarvi
>