Subject: RE: [Speechsc] Managing big grammars

From: "Wyss, Felix" <[email-masked]>
Date: Sat, 11 Jun 2005 19:41:25 -0500

I disagree.  Predictability is *very* critical for an application using
ASR.  It must be able to make sure that when a recognition operation is
started, there is no latency while the ASR server compiles and/or loads
a large precompiled grammar into memory.  The DEFINE-GRAMMAR command is
scoped to a channel, which means by the time the application has a
channel (i.e. a caller is in the system), it is unacceptable to block
for more than fraction of a second while the grammars are being compiled
and/or loaded.  Otherwise, the first caller after a grammar is changed
encounters delays and has a poor user experience or even hangs up.
Somebody may say: well, just make a "priming" call after modifying the
grammar to ensure the grammar is cached.  However, that is not very
practical for automatically generated grammars; but worse yet, in large
systems there are commonly several load-balanced ASR severs and it will
be rather hard to ensure each of them is touched at least once.  

The way we solve this on our platform is as follows: Large grammars that
may change frequently, possibly several times a day, such as company
directory grammars that are generated from a database, can be registered
as "preloaded grammars".  This means, our platform ensures the grammar
is compiled, distributed to all ASR servers, and loaded into their
caches after a new version of that grammar has been generated.  Each
preloaded grammar has a name associated, which is referenced through a
special URI.  The application logic only knows that URI and not the
actual URI/filename of the underlying grammar data.  The subsystem
managing the preloaded grammars ensures that the preloaded grammar URI
resolves to the previous version of the grammar until the new grammar is
loaded and ready on all ASR servers.  Thus, all callers that call into
the system while the grammar is being generated, compiled, and loaded on
the servers will get the previous version.  This ensures that there is
no delay for the first caller after a change.  We can do this as we are
using the native APIs provided by the ASR engine vendors we support and
layer a sophisticated framework and API on top of it.  

It would be nice if MRCP provided some support for schemes like that,
without having to open dummy channels to register grammars and pray the
MRCP server doesn't evict them too early.  An MRCP client has to be able
to guarantee that when a call comes through the system, everything is
ready and it won't experience any delay.  As it stands now the MRCP
protocol does not appear to provide support for this.

Thanks,

Felix

________________________________

From: [email-masked] [mailto:[email-masked]] On
Behalf Of Dave Burke
Sent: Saturday, June 11, 2005 10:42
To: Shanmugham, Saravanan; Corby Anderson; [email-masked]
Subject: Re: [Speechsc] Managing big grammars

I agree. Grammar compilation, caching, and sharing is a problem for the
MRCP server. Exposing this in the protocol puts unnecessary complexity
on the client. For example, the client needs to download grammars,
implement its own HTTP caching, checksum algorithms (some way of not
having to push huge data to the MRCP server to query) etc.

So I vote for point 1 below and include in the text that an MRCP server
could also, as an optimisation, implement a centralised common cache
shared across multiple MRCP servers.

Dave

	----- Original Message ----- 

	From: Shanmugham, Saravanan <mailto:[email-masked]>  

	To: Corby Anderson <mailto:[email-masked]>  ;
[email-masked] 

	Sent: Saturday, June 11, 2005 12:09 AM

	Subject: RE: [Speechsc] Managing big grammars

	I would say that a common cache amoung a farm of MRCP servers to
optimize compilations would be a nice server side optimization and
shouldn't affect the MRCP protocol.

	my 2c.

	Sarvi

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: "Dave Burke" <[email-masked]>
Date: Sun, 12 Jun 2005 20:11:32 +0100

There is no doubt that grammar compilation and loading must be hidden from the caller experience. What is in doubt is whether the MRCP protocol needs to expose such control. 

The grammar loading problem is not dissimilar to cache seeding in the Web world. Taking your use-case, I would implement this in MRCP as a two-step process:

(1) Make the large grammar available through HTTP and date-stamp the URI, e.g. http://example.com/large_grammar_12June2005.grxml. Set the HTTP response header Cache-Control: max-age to a value larger than the grammar change period.

(2) When a new grammar is required, open an MRCP channel to each of the MRCP servers. Issue a DEFINE-GRAMMAR with the relevant URI e.g. http://example.com/large_grammar_13June2005.grxml. Update the application to point at the new grammar.

[For a mid-size grammar where loading latency is negligible, and assuming a centrally located compiled grammar cache, step (2) might only require a DEFINE-GRAMMAR to one MRCP server].

You are correct that the current MRCPv2 draft scopes the DEFINE-GRAMMAR to a channel and this is a concern. However, this is resolved with Sarvi's suggested text change.

Dave
  ----- Original Message ----- 
  From: Wyss, Felix 
  To: Dave Burke ; Shanmugham, Saravanan ; Corby Anderson ; [email-masked] 
  Sent: Sunday, June 12, 2005 1:41 AM
  Subject: RE: [Speechsc] Managing big grammars

  I disagree.  Predictability is *very* critical for an application using ASR.  It must be able to make sure that when a recognition operation is started, there is no latency while the ASR server compiles and/or loads a large precompiled grammar into memory.  The DEFINE-GRAMMAR command is scoped to a channel, which means by the time the application has a channel (i.e. a caller is in the system), it is unacceptable to block for more than fraction of a second while the grammars are being compiled and/or loaded.  Otherwise, the first caller after a grammar is changed encounters delays and has a poor user experience or even hangs up.  Somebody may say: well, just make a "priming" call after modifying the grammar to ensure the grammar is cached.  However, that is not very practical for automatically generated grammars; but worse yet, in large systems there are commonly several load-balanced ASR severs and it will be rather hard to ensure each of them is touched at least once.  

  The way we solve this on our platform is as follows: Large grammars that may change frequently, possibly several times a day, such as company directory grammars that are generated from a database, can be registered as "preloaded grammars".  This means, our platform ensures the grammar is compiled, distributed to all ASR servers, and loaded into their caches after a new version of that grammar has been generated.  Each preloaded grammar has a name associated, which is referenced through a special URI.  The application logic only knows that URI and not the actual URI/filename of the underlying grammar data.  The subsystem managing the preloaded grammars ensures that the preloaded grammar URI resolves to the previous version of the grammar until the new grammar is loaded and ready on all ASR servers.  Thus, all callers that call into the system while the grammar is being generated, compiled, and loaded on the servers will get the previous version.  This ensures that there is no delay for the first caller after a change.  We can do this as we are using the native APIs provided by the ASR engine vendors we support and layer a sophisticated framework and API on top of it.  

  It would be nice if MRCP provided some support for schemes like that, without having to open dummy channels to register grammars and pray the MRCP server doesn't evict them too early.  An MRCP client has to be able to guarantee that when a call comes through the system, everything is ready and it won't experience any delay.  As it stands now the MRCP protocol does not appear to provide support for this.

  Thanks,

  Felix

------------------------------------------------------------------------------

  From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave Burke
  Sent: Saturday, June 11, 2005 10:42
  To: Shanmugham, Saravanan; Corby Anderson; [email-masked]
  Subject: Re: [Speechsc] Managing big grammars

  I agree. Grammar compilation, caching, and sharing is a problem for the MRCP server. Exposing this in the protocol puts unnecessary complexity on the client. For example, the client needs to download grammars, implement its own HTTP caching, checksum algorithms (some way of not having to push huge data to the MRCP server to query) etc.

  So I vote for point 1 below and include in the text that an MRCP server could also, as an optimisation, implement a centralised common cache shared across multiple MRCP servers.

  Dave

    ----- Original Message ----- 

    From: Shanmugham, Saravanan 

    To: Corby Anderson ; [email-masked] 

    I would say that a common cache amoung a farm of MRCP servers to optimize compilations would be a nice server side optimization and shouldn't affect the MRCP protocol.

    my 2c.

    Sarvi