Subject: RE: [Speechsc] Managing big grammars

From: "Wyss, Felix" <[email-masked]>
Date: Sun, 12 Jun 2005 18:24:14 -0500

[inline]

Felix

________________________________

From: Dave Burke [mailto:[email-masked]] 
Sent: Sunday, June 12, 2005 14:12
To: Wyss, Felix; Shanmugham, Saravanan; Corby Anderson;
[email-masked]
Subject: Re: [Speechsc] Managing big grammars

There is no doubt that grammar compilation and loading must be hidden
from the caller experience. What is in doubt is whether the MRCP
protocol needs to expose such control. 

[Wyss, Felix] I believe it is critical that the MRCP standard mandates
behavior that is deterministic and consistent on all conforming MRCP
server implementations.  As it stands now, I am not convinced this is
the case with respect to grammar management.  

The grammar loading problem is not dissimilar to cache seeding in the
Web world. Taking your use-case, I would implement this in MRCP as a
two-step process:

(1) Make the large grammar available through HTTP and date-stamp the
URI, e.g. http://example.com/large_grammar_12June2005.grxml. Set the
HTTP response header Cache-Control: max-age to a value larger than the
grammar change period.

 [Wyss, Felix] This may work in some cases but it is not deterministic.
The HTTP protocol does not mandate that clients honor the cache control
headers.  They are primarily a hint to avoid unnecessary fetches.
Unless there is some way to lock the grammar in memory through the MRCP
protocol, the MRCP server's HTTP client or compiled grammar cache may
decided to evict a large grammar that has not been used for a while
(e.g. overnight).  In my opinion, forcing the application to rely on the
MRCP server's grammar cache behavior is a hack as it will be vendor
specific.  

(2) When a new grammar is required, open an MRCP channel to each of the
MRCP servers. Issue a DEFINE-GRAMMAR with the relevant URI e.g.
http://example.com/large_grammar_13June2005.grxml. Update the
application to point at the new grammar.

[Wyss, Felix] That too is really just a hack.  I can imagine that there
will be load-balancing proxies for MRCP servers.  The client may thus
not reliably know how many actual ASR servers there are or even be able
to individually address them.  

[For a mid-size grammar where loading latency is negligible, and
assuming a centrally located compiled grammar cache, step (2) might only
require a DEFINE-GRAMMAR to one MRCP server].

You are correct that the current MRCPv2 draft scopes the DEFINE-GRAMMAR
to a channel and this is a concern. However, this is resolved with
Sarvi's suggested text change.

[Wyss, Felix] Where is that documented?  I didn't see it in the latest
version I could find (draft-ietf-speechsc-mrcpv2-06.txt).  

Dave

	----- Original Message ----- 

	From: Wyss, Felix <mailto:[email-masked]>  

	To: Dave Burke <mailto:[email-masked]>  ; Shanmugham,
Saravanan <mailto:[email-masked]>  ; Corby Anderson
<mailto:[email-masked]>  ; [email-masked] 

	Sent: Sunday, June 12, 2005 1:41 AM

	Subject: RE: [Speechsc] Managing big grammars

	I disagree.  Predictability is *very* critical for an
application using ASR.  It must be able to make sure that when a
recognition operation is started, there is no latency while the ASR
server compiles and/or loads a large precompiled grammar into memory.
The DEFINE-GRAMMAR command is scoped to a channel, which means by the
time the application has a channel (i.e. a caller is in the system), it
is unacceptable to block for more than fraction of a second while the
grammars are being compiled and/or loaded.  Otherwise, the first caller
after a grammar is changed encounters delays and has a poor user
experience or even hangs up.  Somebody may say: well, just make a
"priming" call after modifying the grammar to ensure the grammar is
cached.  However, that is not very practical for automatically generated
grammars; but worse yet, in large systems there are commonly several
load-balanced ASR severs and it will be rather hard to ensure each of
them is touched at least once.  

	The way we solve this on our platform is as follows: Large
grammars that may change frequently, possibly several times a day, such
as company directory grammars that are generated from a database, can be
registered as "preloaded grammars".  This means, our platform ensures
the grammar is compiled, distributed to all ASR servers, and loaded into
their caches after a new version of that grammar has been generated.
Each preloaded grammar has a name associated, which is referenced
through a special URI.  The application logic only knows that URI and
not the actual URI/filename of the underlying grammar data.  The
subsystem managing the preloaded grammars ensures that the preloaded
grammar URI resolves to the previous version of the grammar until the
new grammar is loaded and ready on all ASR servers.  Thus, all callers
that call into the system while the grammar is being generated,
compiled, and loaded on the servers will get the previous version.  This
ensures that there is no delay for the first caller after a change.  We
can do this as we are using the native APIs provided by the ASR engine
vendors we support and layer a sophisticated framework and API on top of
it.  

	It would be nice if MRCP provided some support for schemes like
that, without having to open dummy channels to register grammars and
pray the MRCP server doesn't evict them too early.  An MRCP client has
to be able to guarantee that when a call comes through the system,
everything is ready and it won't experience any delay.  As it stands now
the MRCP protocol does not appear to provide support for this.

	Thanks,

	Felix

________________________________

	From: [email-masked]
[mailto:[email-masked]] On Behalf Of Dave Burke
	Sent: Saturday, June 11, 2005 10:42
	To: Shanmugham, Saravanan; Corby Anderson; [email-masked]
	Subject: Re: [Speechsc] Managing big grammars

	I agree. Grammar compilation, caching, and sharing is a problem
for the MRCP server. Exposing this in the protocol puts unnecessary
complexity on the client. For example, the client needs to download
grammars, implement its own HTTP caching, checksum algorithms (some way
of not having to push huge data to the MRCP server to query) etc.

	So I vote for point 1 below and include in the text that an MRCP
server could also, as an optimisation, implement a centralised common
cache shared across multiple MRCP servers.

	Dave

		----- Original Message ----- 

		From: Shanmugham, Saravanan <mailto:[email-masked]>  

		To: Corby Anderson <mailto:[email-masked]>  ;
[email-masked] 

		Sent: Saturday, June 11, 2005 12:09 AM

		Subject: RE: [Speechsc] Managing big grammars

		I would say that a common cache amoung a farm of MRCP
servers to optimize compilations would be a nice server side
optimization and shouldn't affect the MRCP protocol.

		my 2c.

		Sarvi

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: "Dave Burke" <[email-masked]>
Date: Mon, 13 Jun 2005 11:13:26 +0100

Inline with DB>

-- Dave
  ----- Original Message ----- 
  From: Wyss, Felix 
  To: Dave Burke ; Shanmugham, Saravanan ; Corby Anderson ; [email-masked] 
  Sent: Monday, June 13, 2005 12:24 AM
  Subject: RE: [Speechsc] Managing big grammars

  [inline]

  Felix

------------------------------------------------------------------------------

  From: Dave Burke [mailto:[email-masked]] 
  Sent: Sunday, June 12, 2005 14:12
  To: Wyss, Felix; Shanmugham, Saravanan; Corby Anderson; [email-masked]
  Subject: Re: [Speechsc] Managing big grammars

  There is no doubt that grammar compilation and loading must be hidden from the caller experience. What is in doubt is whether the MRCP protocol needs to expose such control. 

  [Wyss, Felix] I believe it is critical that the MRCP standard mandates behavior that is deterministic and consistent on all conforming MRCP server implementations.  As it stands now, I am not convinced this is the case with respect to grammar management.  

  DB> I agree that the MRCP standard should not be silent on _optimisations_ for large grammars so that we achieve consistent behavior across implementations. I just don't agree that additional MRCP client control is necessary, warranted, nor solves the problem.

  The grammar loading problem is not dissimilar to cache seeding in the Web world. Taking your use-case, I would implement this in MRCP as a two-step process:

  (1) Make the large grammar available through HTTP and date-stamp the URI, e.g. http://example.com/large_grammar_12June2005.grxml. Set the HTTP response header Cache-Control: max-age to a value larger than the grammar change period.

   [Wyss, Felix] This may work in some cases but it is not deterministic.  The HTTP protocol does not mandate that clients honor the cache control headers.  They are primarily a hint to avoid unnecessary fetches.  Unless there is some way to lock the grammar in memory through the MRCP protocol, the MRCP server's HTTP client or compiled grammar cache may decided to evict a large grammar that has not been used for a while (e.g. overnight).  In my opinion, forcing the application to rely on the MRCP server's grammar cache behavior is a hack as it will be vendor specific.  

  DB> I would not consider using HTTP standard caching mechanisms a hack. An MRCP server that supports large grammars would be ill advised not honor HTTP caching headers. Moreover, an MRCP server that supports large grammars would be ill advised to implement an aggressive LRU approach you describe (rather one would imagine pruning would only happen at cache overflow).

  (2) When a new grammar is required, open an MRCP channel to each of the MRCP servers. Issue a DEFINE-GRAMMAR with the relevant URI e.g. http://example.com/large_grammar_13June2005.grxml. Update the application to point at the new grammar.

  [Wyss, Felix] That too is really just a hack.  I can imagine that there will be load-balancing proxies for MRCP servers.  The client may thus not reliably know how many actual ASR servers there are or even be able to individually address them.  

  DB> The case of "pinging" each server only applies to the case when loading the previously compiled grammar from a central location into memory incurs significant latency. This is the "jumbo" as opposed to "large" case. Adding verbs to the MRCP client to control grammar compilation (as discussed in this thread) does not alleviate this problem. This is a problem for the MRCP server to optimise - not the client to control. There are solutions that can work here (e.g. when a DEFINE-GRAMMAR occurs on any MRCP server, compile and cache in a central location and send a multicast message to inform other servers to reload). 

  [For a mid-size grammar where loading latency is negligible, and assuming a centrally located compiled grammar cache, step (2) might only require a DEFINE-GRAMMAR to one MRCP server].

  You are correct that the current MRCPv2 draft scopes the DEFINE-GRAMMAR to a channel and this is a concern. However, this is resolved with Sarvi's suggested text change.

  [Wyss, Felix] Where is that documented?  I didn't see it in the latest version I could find (draft-ietf-speechsc-mrcpv2-06.txt).  

  DB> In this thread. 

  Dave

    ----- Original Message ----- 

    From: Wyss, Felix 

    To: Dave Burke ; Shanmugham, Saravanan ; Corby Anderson ; [email-masked] 

    I disagree.  Predictability is *very* critical for an application using ASR.  It must be able to make sure that when a recognition operation is started, there is no latency while the ASR server compiles and/or loads a large precompiled grammar into memory.  The DEFINE-GRAMMAR command is scoped to a channel, which means by the time the application has a channel (i.e. a caller is in the system), it is unacceptable to block for more than fraction of a second while the grammars are being compiled and/or loaded.  Otherwise, the first caller after a grammar is changed encounters delays and has a poor user experience or even hangs up.  Somebody may say: well, just make a "priming" call after modifying the grammar to ensure the grammar is cached.  However, that is not very practical for automatically generated grammars; but worse yet, in large systems there are commonly several load-balanced ASR severs and it will be rather hard to ensure each of them is touched at least once.  

    The way we solve this on our platform is as follows: Large grammars that may change frequently, possibly several times a day, such as company directory grammars that are generated from a database, can be registered as "preloaded grammars".  This means, our platform ensures the grammar is compiled, distributed to all ASR servers, and loaded into their caches after a new version of that grammar has been generated.  Each preloaded grammar has a name associated, which is referenced through a special URI.  The application logic only knows that URI and not the actual URI/filename of the underlying grammar data.  The subsystem managing the preloaded grammars ensures that the preloaded grammar URI resolves to the previous version of the grammar until the new grammar is loaded and ready on all ASR servers.  Thus, all callers that call into the system while the grammar is being generated, compiled, and loaded on the servers will get the previous version.  This ensures that there is no delay for the first caller after a change.  We can do this as we are using the native APIs provided by the ASR engine vendors we support and layer a sophisticated framework and API on top of it.  

    It would be nice if MRCP provided some support for schemes like that, without having to open dummy channels to register grammars and pray the MRCP server doesn't evict them too early.  An MRCP client has to be able to guarantee that when a call comes through the system, everything is ready and it won't experience any delay.  As it stands now the MRCP protocol does not appear to provide support for this.

    Thanks,

    Felix

----------------------------------------------------------------------------

    From: [email-masked] [mailto:[email-masked]] On Behalf Of Dave Burke
    Sent: Saturday, June 11, 2005 10:42
    To: Shanmugham, Saravanan; Corby Anderson; [email-masked]
    Subject: Re: [Speechsc] Managing big grammars

    I agree. Grammar compilation, caching, and sharing is a problem for the MRCP server. Exposing this in the protocol puts unnecessary complexity on the client. For example, the client needs to download grammars, implement its own HTTP caching, checksum algorithms (some way of not having to push huge data to the MRCP server to query) etc.

    So I vote for point 1 below and include in the text that an MRCP server could also, as an optimisation, implement a centralised common cache shared across multiple MRCP servers.

    Dave

      ----- Original Message ----- 

      From: Shanmugham, Saravanan 

      To: Corby Anderson ; [email-masked] 

      I would say that a common cache amoung a farm of MRCP servers to optimize compilations would be a nice server side optimization and shouldn't affect the MRCP protocol.

      my 2c.

      Sarvi

------------------------------------------------------------------------------