Subject: RE: [Speechsc] Managing big grammars

From: "Shanmugham, Saravanan" <[email-masked]>
Date: Mon, 13 Jun 2005 23:58:25 -0700

I dont understand why the RUL has to encode and date and time into it.
Lets say we are refering to large_grammar.grxml
 
When a client does a DEFINE-GRAMMAR on the MRCP server with this URI.
The MRCP server is expected to go download this URI and compile it. The
caching behaviour of this URI and how long it lives in the cache is
dependent on what is returned by the HTTP GET operation that brought the
grammar into the MRCP server. In addition, the behaviour is also
governed by Cache -Control parameters that the client can specify for
the DEFINE-GRAMMAR that would take precedence over what was returned by
the HTTP GET operation. This means that the MRCP client can force the
MRCP server to ignore the cache parameters it got from the grammar
webserver.
 
So, for the large_grammar.grxml, if the webserver returned a large
max-age value the grammar and its compiled version in the cache of the
MRCP server should not get timedout. For any future requests for that
grammar, MRCP server should go do fetch of the document, compare the
max-age of the fetched coument and make sure the webserver says that the
document in the cache is not out-of-date. As long as the grammar
document in the cache is not out of date, you don't need to compile it
again and you are fine. If the propoerty of the large_grammar.grxml on
the webserver had changed then the MRCP server would be expected to
refresh the cache and compile the newly downloaded grammar.

Am I missing something here.
 
Sarvi 

________________________________

	From: [email-masked]
[mailto:[email-masked]] On Behalf Of Dave Burke
	Sent: Sunday, June 12, 2005 12:12 PM
	To: Wyss, Felix; Shanmugham, Saravanan; Corby Anderson;
[email-masked]
	Subject: Re: [Speechsc] Managing big grammars

	There is no doubt that grammar compilation and loading must be
hidden from the caller experience. What is in doubt is whether the MRCP
protocol needs to expose such control. 
	 
	The grammar loading problem is not dissimilar to cache seeding
in the Web world. Taking your use-case, I would implement this in MRCP
as a two-step process:
	 
	(1) Make the large grammar available through HTTP and date-stamp
the URI, e.g. http://example.com/large_grammar_12June2005.grxml. Set the
HTTP response header Cache-Control: max-age to a value larger than the
grammar change period.
	 
	(2) When a new grammar is required, open an MRCP channel to each
of the MRCP servers. Issue a DEFINE-GRAMMAR with the relevant URI e.g.
http://example.com/large_grammar_13June2005.grxml. Update the
application to point at the new grammar.
	 
	[For a mid-size grammar where loading latency is negligible, and
assuming a centrally located compiled grammar cache, step (2) might only
require a DEFINE-GRAMMAR to one MRCP server].
	 
	You are correct that the current MRCPv2 draft scopes the
DEFINE-GRAMMAR to a channel and this is a concern. However, this is
resolved with Sarvi's suggested text change.
	 
	Dave

		----- Original Message ----- 
		From: Wyss, Felix <mailto:[email-masked]>  
		To: Dave Burke <mailto:[email-masked]>  ;
Shanmugham, Saravanan <mailto:[email-masked]>  ; Corby Anderson
<mailto:[email-masked]>  ; [email-masked] 
		Sent: Sunday, June 12, 2005 1:41 AM
		Subject: RE: [Speechsc] Managing big grammars

		I disagree.  Predictability is *very* critical for an
application using ASR.  It must be able to make sure that when a
recognition operation is started, there is no latency while the ASR
server compiles and/or loads a large precompiled grammar into memory.
The DEFINE-GRAMMAR command is scoped to a channel, which means by the
time the application has a channel (i.e. a caller is in the system), it
is unacceptable to block for more than fraction of a second while the
grammars are being compiled and/or loaded.  Otherwise, the first caller
after a grammar is changed encounters delays and has a poor user
experience or even hangs up.  Somebody may say: well, just make a
"priming" call after modifying the grammar to ensure the grammar is
cached.  However, that is not very practical for automatically generated
grammars; but worse yet, in large systems there are commonly several
load-balanced ASR severs and it will be rather hard to ensure each of
them is touched at least once.  

		The way we solve this on our platform is as follows:
Large grammars that may change frequently, possibly several times a day,
such as company directory grammars that are generated from a database,
can be registered as "preloaded grammars".  This means, our platform
ensures the grammar is compiled, distributed to all ASR servers, and
loaded into their caches after a new version of that grammar has been
generated.  Each preloaded grammar has a name associated, which is
referenced through a special URI.  The application logic only knows that
URI and not the actual URI/filename of the underlying grammar data.  The
subsystem managing the preloaded grammars ensures that the preloaded
grammar URI resolves to the previous version of the grammar until the
new grammar is loaded and ready on all ASR servers.  Thus, all callers
that call into the system while the grammar is being generated,
compiled, and loaded on the servers will get the previous version.  This
ensures that there is no delay for the first caller after a change.  We
can do this as we are using the native APIs provided by the ASR engine
vendors we support and layer a sophisticated framework and API on top of
it.  

		It would be nice if MRCP provided some support for
schemes like that, without having to open dummy channels to register
grammars and pray the MRCP server doesn't evict them too early.  An MRCP
client has to be able to guarantee that when a call comes through the
system, everything is ready and it won't experience any delay.  As it
stands now the MRCP protocol does not appear to provide support for
this.

		Thanks,

		Felix

________________________________

		From: [email-masked]
[mailto:[email-masked]] On Behalf Of Dave Burke
		Sent: Saturday, June 11, 2005 10:42
		To: Shanmugham, Saravanan; Corby Anderson;
[email-masked]
		Subject: Re: [Speechsc] Managing big grammars

		I agree. Grammar compilation, caching, and sharing is a
problem for the MRCP server. Exposing this in the protocol puts
unnecessary complexity on the client. For example, the client needs to
download grammars, implement its own HTTP caching, checksum algorithms
(some way of not having to push huge data to the MRCP server to query)
etc.

		So I vote for point 1 below and include in the text that
an MRCP server could also, as an optimisation, implement a centralised
common cache shared across multiple MRCP servers.

		Dave

			----- Original Message ----- 

			From: Shanmugham, Saravanan
<mailto:[email-masked]>  

			To: Corby Anderson <mailto:[email-masked]>  ;
[email-masked] 

			Sent: Saturday, June 11, 2005 12:09 AM

			Subject: RE: [Speechsc] Managing big grammars

			I would say that a common cache amoung a farm of
MRCP servers to optimize compilations would be a nice server side
optimization and shouldn't affect the MRCP protocol.

			my 2c.

			Sarvi

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: "Dave Burke" <[email-masked]>
Date: Tue, 14 Jun 2005 11:16:19 +0100

It's not that the URL needs a specific date or time, just that it has a unique name. This is simply so that you can ensure the new grammar is downloaded, compiled, and loaded ahead of any calls. Once compiled and loaded, the application grammar URIs can be updated to reference the new grammar. This way you avoid the danger that a bunch of calls, which arrive at the same time as the DEFINE-GRAMMAR is happening, experience delays.

Dave
  ----- Original Message ----- 
  From: Shanmugham, Saravanan 
  To: Dave Burke ; Wyss, Felix ; Corby Anderson ; [email-masked] 
  Sent: Tuesday, June 14, 2005 7:58 AM
  Subject: RE: [Speechsc] Managing big grammars

  I dont understand why the RUL has to encode and date and time into it. Lets say we are refering to large_grammar.grxml

  When a client does a DEFINE-GRAMMAR on the MRCP server with this URI. The MRCP server is expected to go download this URI and compile it. The caching behaviour of this URI and how long it lives in the cache is dependent on what is returned by the HTTP GET operation that brought the grammar into the MRCP server. In addition, the behaviour is also governed by Cache -Control parameters that the client can specify for the DEFINE-GRAMMAR that would take precedence over what was returned by the HTTP GET operation. This means that the MRCP client can force the MRCP server to ignore the cache parameters it got from the grammar webserver.

  So, for the large_grammar.grxml, if the webserver returned a large max-age value the grammar and its compiled version in the cache of the MRCP server should not get timedout. For any future requests for that grammar, MRCP server should go do fetch of the document, compare the max-age of the fetched coument and make sure the webserver says that the document in the cache is not out-of-date. As long as the grammar document in the cache is not out of date, you don't need to compile it again and you are fine. If the propoerty of the large_grammar.grxml on the webserver had changed then the MRCP server would be expected to refresh the cache and compile the newly downloaded grammar.

  Sarvi 

----------------------------------------------------------------------------
    From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave Burke
    Sent: Sunday, June 12, 2005 12:12 PM
    To: Wyss, Felix; Shanmugham, Saravanan; Corby Anderson; [email-masked]
    Subject: Re: [Speechsc] Managing big grammars

    There is no doubt that grammar compilation and loading must be hidden from the caller experience. What is in doubt is whether the MRCP protocol needs to expose such control. 

    The grammar loading problem is not dissimilar to cache seeding in the Web world. Taking your use-case, I would implement this in MRCP as a two-step process:

    (1) Make the large grammar available through HTTP and date-stamp the URI, e.g. http://example.com/large_grammar_12June2005.grxml. Set the HTTP response header Cache-Control: max-age to a value larger than the grammar change period.

    (2) When a new grammar is required, open an MRCP channel to each of the MRCP servers. Issue a DEFINE-GRAMMAR with the relevant URI e.g. http://example.com/large_grammar_13June2005.grxml. Update the application to point at the new grammar.

    [For a mid-size grammar where loading latency is negligible, and assuming a centrally located compiled grammar cache, step (2) might only require a DEFINE-GRAMMAR to one MRCP server].

    You are correct that the current MRCPv2 draft scopes the DEFINE-GRAMMAR to a channel and this is a concern. However, this is resolved with Sarvi's suggested text change.

    Dave
      ----- Original Message ----- 
      From: Wyss, Felix 
      To: Dave Burke ; Shanmugham, Saravanan ; Corby Anderson ; [email-masked] 
      Sent: Sunday, June 12, 2005 1:41 AM
      Subject: RE: [Speechsc] Managing big grammars

      I disagree.  Predictability is *very* critical for an application using ASR.  It must be able to make sure that when a recognition operation is started, there is no latency while the ASR server compiles and/or loads a large precompiled grammar into memory.  The DEFINE-GRAMMAR command is scoped to a channel, which means by the time the application has a channel (i.e. a caller is in the system), it is unacceptable to block for more than fraction of a second while the grammars are being compiled and/or loaded.  Otherwise, the first caller after a grammar is changed encounters delays and has a poor user experience or even hangs up.  Somebody may say: well, just make a "priming" call after modifying the grammar to ensure the grammar is cached.  However, that is not very practical for automatically generated grammars; but worse yet, in large systems there are commonly several load-balanced ASR severs and it will be rather hard to ensure each of them is touched at least once.  

      The way we solve this on our platform is as follows: Large grammars that may change frequently, possibly several times a day, such as company directory grammars that are generated from a database, can be registered as "preloaded grammars".  This means, our platform ensures the grammar is compiled, distributed to all ASR servers, and loaded into their caches after a new version of that grammar has been generated.  Each preloaded grammar has a name associated, which is referenced through a special URI.  The application logic only knows that URI and not the actual URI/filename of the underlying grammar data.  The subsystem managing the preloaded grammars ensures that the preloaded grammar URI resolves to the previous version of the grammar until the new grammar is loaded and ready on all ASR servers.  Thus, all callers that call into the system while the grammar is being generated, compiled, and loaded on the servers will get the previous version.  This ensures that there is no delay for the first caller after a change.  We can do this as we are using the native APIs provided by the ASR engine vendors we support and layer a sophisticated framework and API on top of it.  

      It would be nice if MRCP provided some support for schemes like that, without having to open dummy channels to register grammars and pray the MRCP server doesn't evict them too early.  An MRCP client has to be able to guarantee that when a call comes through the system, everything is ready and it won't experience any delay.  As it stands now the MRCP protocol does not appear to provide support for this.

      Thanks,

      Felix

--------------------------------------------------------------------------

      From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave Burke
      Sent: Saturday, June 11, 2005 10:42
      To: Shanmugham, Saravanan; Corby Anderson; [email-masked]
      Subject: Re: [Speechsc] Managing big grammars

      I agree. Grammar compilation, caching, and sharing is a problem for the MRCP server. Exposing this in the protocol puts unnecessary complexity on the client. For example, the client needs to download grammars, implement its own HTTP caching, checksum algorithms (some way of not having to push huge data to the MRCP server to query) etc.

      So I vote for point 1 below and include in the text that an MRCP server could also, as an optimisation, implement a centralised common cache shared across multiple MRCP servers.

      Dave

        ----- Original Message ----- 

        To: Corby Anderson ; [email-masked] 

        I would say that a common cache amoung a farm of MRCP servers to optimize compilations would be a nice server side optimization and shouldn't affect the MRCP protocol.

        my 2c.

        Sarvi

---

From: "David R. Oran" <[email-masked]>
Date: Tue, 14 Jun 2005 08:23:05 -0400

On Jun 14, 2005, at 6:16 AM, Dave Burke wrote:

> It's not that the URL needs a specific date or time, just that it  
> has a unique name. This is simply so that you can ensure the new  
> grammar is downloaded, compiled, and loaded ahead of any calls.  
> Once compiled and loaded, the application grammar URIs can be  
> updated to reference the new grammar. This way you avoid the danger  
> that a bunch of calls, which arrive at the same time as the DEFINE- 
> GRAMMAR is happening, experience delays.
>
I'm lost - so what happens if the call arrives during this  
hypothetical COMPILE-GRAMMAR extension? How's that any different?

There are LOTS of things that might hold up a call. Setting up a TLS  
connection can sometimes take a few seconds to do all the PK  
operations if the server is busy and you don't have PK crypto  
acceleration hardware. A server might be taken out of service and you  
have to load-balance. The source grammar might have gotten replaced  
due to tuning or a bug.

If you don't want to have any delays, wait until all your setup is  
done before enabling your input ports...

> I dont understand why the RUL has to encode and date and time into  
> it. Lets say we are refering to large_grammar.grxml
>
> When a client does a DEFINE-GRAMMAR on the MRCP server with this  
> URI. The MRCP server is expected to go download this URI and  
> compile it. The caching behaviour of this URI and how long it lives  
> in the cache is dependent on what is returned by the HTTP GET  
> operation that brought the grammar into the MRCP server. In  
> addition, the behaviour is also governed by Cache -Control  
> parameters that the client can specify for the DEFINE-GRAMMAR that  
> would take precedence over what was returned by the HTTP GET  
> operation. This means that the MRCP client can force the MRCP  
> server to ignore the cache parameters it got from the grammar  
> webserver.
>
> max-age value the grammar and its compiled version in the cache of  
> the MRCP server should not get timedout. For any future requests  
> for that grammar, MRCP server should go do fetch of the document,  
> compare the max-age of the fetched coument and make sure the  
> webserver says that the document in the cache is not out-of-date.  
> As long as the grammar document in the cache is not out of date,  
> you don't need to compile it again and you are fine. If the  
> propoerty of the large_grammar.grxml on the webserver had changed  
> then the MRCP server would be expected to refresh the cache and  
> compile the newly downloaded grammar.
>
>
> hidden from the caller experience. What is in doubt is whether the  
> MRCP protocol needs to expose such control.
>
> The grammar loading problem is not dissimilar to cache seeding in  
> the Web world. Taking your use-case, I would implement this in MRCP  
> as a two-step process:
>
> the URI, e.g. http://example.com/large_grammar_12June2005.grxml.  
> Set the HTTP response header Cache-Control: max-age to a value  
> larger than the grammar change period.
>
> (2) When a new grammar is required, open an MRCP channel to each of  
> the MRCP servers. Issue a DEFINE-GRAMMAR with the relevant URI e.g.  
> assuming a centrally located compiled grammar cache, step (2) might  
> only require a DEFINE-GRAMMAR to one MRCP server].
>
> You are correct that the current MRCPv2 draft scopes the DEFINE- 
> GRAMMAR to a channel and this is a concern. However, this is  
> I disagree.  Predictability is *very* critical for an application  
> using ASR.  It must be able to make sure that when a recognition  
> operation is started, there is no latency while the ASR server  
> compiles and/or loads a large precompiled grammar into memory.  The  
> DEFINE-GRAMMAR command is scoped to a channel, which means by the  
> time the application has a channel (i.e. a caller is in the  
> system), it is unacceptable to block for more than fraction of a  
> second while the grammars are being compiled and/or loaded.   
> Otherwise, the first caller after a grammar is changed encounters  
> delays and has a poor user experience or even hangs up.  Somebody  
> may say: well, just make a “priming” call after modifying the  
> grammar to ensure the grammar is cached.  However, that is not very  
> practical for automatically generated grammars; but worse yet, in  
> large systems there are commonly several load-balanced ASR severs  
> and it will be rather hard to ensure each of them is touched at  
> least once.
>
>
>
> The way we solve this on our platform is as follows: Large grammars  
> that may change frequently, possibly several times a day, such as  
> company directory grammars that are generated from a database, can  
> be registered as “preloaded grammars”.  This means, our platform  
> ensures the grammar is compiled, distributed to all ASR servers,  
> and loaded into their caches after a new version of that grammar  
> has been generated.  Each preloaded grammar has a name associated,  
> which is referenced through a special URI.  The application logic  
> only knows that URI and not the actual URI/filename of the  
> underlying grammar data.  The subsystem managing the preloaded  
> grammars ensures that the preloaded grammar URI resolves to the  
> previous version of the grammar until the new grammar is loaded and  
> ready on all ASR servers.  Thus, all callers that call into the  
> system while the grammar is being generated, compiled, and loaded  
> on the servers will get the previous version.  This ensures that  
> there is no delay for the first caller after a change.  We can do  
> this as we are using the native APIs provided by the ASR engine  
> vendors we support and layer a sophisticated framework and API on  
> top of it.
>
>
>
> It would be nice if MRCP provided some support for schemes like  
> that, without having to open dummy channels to register grammars  
> and pray the MRCP server doesn’t evict them too early.  An MRCP  
> client has to be able to guarantee that when a call comes through  
> the system, everything is ready and it won’t experience any delay.   
> As it stands now the MRCP protocol does not appear to provide  
> support for this.
>
>
>
> Thanks,
>
> Felix
>
>
>
> I agree. Grammar compilation, caching, and sharing is a problem for  
> the MRCP server. Exposing this in the protocol puts unnecessary  
> complexity on the client. For example, the client needs to download  
> grammars, implement its own HTTP caching, checksum algorithms (some  
> way of not having to push huge data to the MRCP server to query) etc.
>
>
>
> So I vote for point 1 below and include in the text that an MRCP  
> server could also, as an optimisation, implement a centralised  
> I would say that a common cache amoung a farm of MRCP servers to  
> optimize compilations would be a nice server side optimization and  
> shouldn't affect the MRCP protocol.
>
>
>
> my 2c.
>
> Sarvi
>