Subject: RE: [Speechsc] Managing big grammars

From: "Reifenrath, Klaus" <[email-masked]>
Date: Tue, 14 Jun 2005 08:51:49 -0400

Is 4) a realistic scenario? It would require that the MRCP server know each
other, which results in a configuration nightmare. And what happens if one
of them is down just at that moment... 

Slight reformulation of 2):
An MRCP server may cache a compiled version of a grammar

Klaus

-----Original Message-----
From: David R. Oran [mailto:[email-masked]] 
Sent: Dienstag, 14. Juni 2005 14:31
To: Dave Burke
Cc: [email-masked]; Shanmugham, Saravanan; Wyss, Felix
Subject: Re: [Speechsc] Managing big grammars

I like Dave's formulation. If there are no objections, I'm going to craft
this into a sub-section on "Grammar Caching"  under the section on "Grammar
Data". On looking where to put this, I noticed that the description is under
9.5.1 "recognizer Grammar Data" which is under "Recognizer message body".
Some of that material isn't really about message body (since it talks about
ow indirect content is handled), so I'm going to re-jigger things a bit to
improve the flow and move the parts about indirect content to the new
earlier section.

Speak now if you don't like this, because I'm going to do this later
today...

On Jun 14, 2005, at 8:16 AM, Dave Burke wrote:

> That optimisation would be nice alright and is not restricted by HTTP 
> (see note from RFC 2616 below).
>
> It seems like we have got some good optimisation suggestions (read:  
> informative) that could go into the spec to address large grammars.  
> If memory serves, they are:
>     1. An MRCP server may cache grammars used in DEFINE-GRAMMAR or 
> RECOGNIZE to be used over multiple MRCP sessions
>     2. An MRCP server may cache the compiled version of a grammar
>     3. An MRCP server may cache grammars centrally to allow other 
> servers take advantage of the cache
>     4. An MRCP server, in response to a DEFINE-GRAMMAR, may inform 
> other MRCP servers to reload the new grammar into memory
>     5. An MRCP server may choose to return a stale response while a 
> new grammar is being downloaded, compiled, and cached
>
> -- Dave
>
> From RFC 2616:
>
>    In some cases, the operator of a cache MAY choose to configure it 
> to
>    return stale responses even when not requested by clients. This
>    decision ought not be made lightly, but may be necessary for 
> reasons
>    of availability or performance, especially when the cache is poorly
>    connected to the origin server. Whenever a cache returns a stale
>    response, it MUST mark it as such (using a Warning header) enabling
>    the client software to alert the user that there might be a 
> potential
>    problem.
>
>
> ----- Original Message -----
> From: Shanmugham, Saravanan
> To: Dave Burke ; Wyss, Felix ; Corby Anderson ; [email-masked]
> Sent: Tuesday, June 14, 2005 12:04 PM
> Subject: RE: [Speechsc] Managing big grammars
>
> cool. That makes sense.
> The way I though this problem would have been addressed, is the MRCP 
> caching system to be smart.
> That is everytime there is a request a grammar for a grammar URI.  
> the MRCP server would do a fetch to check if the document has changed. 
> If it has, then it would download and compile the grammar.
> While this happenning existing calls that were using the older 
> compiled grammar continue to use it. Once this is ready, the previous 
> version of the grammar is allowed to expire-out.
>
> Would this not address your scenario and avoid having to do this work 
> around you are suggesting?
>
> Sarvi
>
> From: Dave Burke [mailto:[email-masked]]
> Sent: Tuesday, June 14, 2005 3:16 AM
> To: Shanmugham, Saravanan; Wyss, Felix; Corby Anderson; 
> [email-masked]
> Subject: Re: [Speechsc] Managing big grammars
>
> It's not that the URL needs a specific date or time, just that it has 
> a unique name. This is simply so that you can ensure the new grammar 
> is downloaded, compiled, and loaded ahead of any calls.
> Once compiled and loaded, the application grammar URIs can be updated 
> to reference the new grammar. This way you avoid the danger that a 
> bunch of calls, which arrive at the same time as the DEFINE- GRAMMAR 
> is happening, experience delays.
>
> Dave
> ----- Original Message -----
> From: Shanmugham, Saravanan
> To: Dave Burke ; Wyss, Felix ; Corby Anderson ; [email-masked]
> Sent: Tuesday, June 14, 2005 7:58 AM
> Subject: RE: [Speechsc] Managing big grammars
>
> I dont understand why the RUL has to encode and date and time into it. 
> Lets say we are refering to large_grammar.grxml
>
> When a client does a DEFINE-GRAMMAR on the MRCP server with this URI. 
> The MRCP server is expected to go download this URI and compile it. 
> The caching behaviour of this URI and how long it lives in the cache 
> is dependent on what is returned by the HTTP GET operation that 
> brought the grammar into the MRCP server. In addition, the behaviour 
> is also governed by Cache -Control parameters that the client can 
> specify for the DEFINE-GRAMMAR that would take precedence over what 
> was returned by the HTTP GET operation. This means that the MRCP 
> client can force the MRCP server to ignore the cache parameters it got 
> from the grammar webserver.
>
> So, for the large_grammar.grxml, if the webserver returned a large 
> max-age value the grammar and its compiled version in the cache of the 
> MRCP server should not get timedout. For any future requests for that 
> grammar, MRCP server should go do fetch of the document, compare the 
> max-age of the fetched coument and make sure the webserver says that 
> the document in the cache is not out-of-date.
> As long as the grammar document in the cache is not out of date, you 
> don't need to compile it again and you are fine. If the propoerty of 
> the large_grammar.grxml on the webserver had changed then the MRCP 
> server would be expected to refresh the cache and compile the newly 
> downloaded grammar.
>
>
> Am I missing something here.
>
> Sarvi
>
> From: [email-masked] [mailto:[email-masked]]
> On Behalf Of Dave Burke
> Sent: Sunday, June 12, 2005 12:12 PM
> To: Wyss, Felix; Shanmugham, Saravanan; Corby Anderson; 
> [email-masked]
> Subject: Re: [Speechsc] Managing big grammars
>
> There is no doubt that grammar compilation and loading must be hidden 
> from the caller experience. What is in doubt is whether the MRCP 
> protocol needs to expose such control.
>
> The grammar loading problem is not dissimilar to cache seeding in the 
> Web world. Taking your use-case, I would implement this in MRCP as a 
> two-step process:
>
> (1) Make the large grammar available through HTTP and date-stamp the 
> URI, e.g. http://example.com/large_grammar_12June2005.grxml.
> Set the HTTP response header Cache-Control: max-age to a value larger 
> than the grammar change period.
>
> (2) When a new grammar is required, open an MRCP channel to each of 
> the MRCP servers. Issue a DEFINE-GRAMMAR with the relevant URI e.g.
> http://example.com/large_grammar_13June2005.grxml. Update the 
> application to point at the new grammar.
>
> [For a mid-size grammar where loading latency is negligible, and 
> assuming a centrally located compiled grammar cache, step (2) might 
> only require a DEFINE-GRAMMAR to one MRCP server].
>
> You are correct that the current MRCPv2 draft scopes the DEFINE- 
> GRAMMAR to a channel and this is a concern. However, this is resolved 
> with Sarvi's suggested text change.
>
> Dave
> ----- Original Message -----
> From: Wyss, Felix
> To: Dave Burke ; Shanmugham, Saravanan ; Corby Anderson ; 
> [email-masked]
> Sent: Sunday, June 12, 2005 1:41 AM
> Subject: RE: [Speechsc] Managing big grammars
>
> I disagree.  Predictability is *very* critical for an application 
> using ASR.  It must be able to make sure that when a recognition 
> operation is started, there is no latency while the ASR server 
> compiles and/or loads a large precompiled grammar into memory.  The 
> DEFINE-GRAMMAR command is scoped to a channel, which means by the time 
> the application has a channel (i.e. a caller is in the system), it is 
> unacceptable to block for more than fraction of a
> second while the grammars are being compiled and/or loaded.   
> Otherwise, the first caller after a grammar is changed encounters 
> delays and has a poor user experience or even hangs up.  Somebody may 
> say: well, just make a "priming" call after modifying the grammar to 
> ensure the grammar is cached.  However, that is not very practical for 
> automatically generated grammars; but worse yet, in large systems 
> there are commonly several load-balanced ASR severs and it will be 
> rather hard to ensure each of them is touched at least once.
>
>
>
> The way we solve this on our platform is as follows: Large grammars 
> that may change frequently, possibly several times a day, such as 
> company directory grammars that are generated from a database, can be 
> registered as "preloaded grammars".  This means, our platform ensures 
> the grammar is compiled, distributed to all ASR servers, and loaded 
> into their caches after a new version of that grammar has been 
> generated.  Each preloaded grammar has a name associated, which is 
> referenced through a special URI.  The application logic only knows 
> that URI and not the actual URI/filename of the underlying grammar 
> data.  The subsystem managing the preloaded grammars ensures that the 
> preloaded grammar URI resolves to the previous version of the grammar 
> until the new grammar is loaded and ready on all ASR servers.  Thus, 
> all callers that call into the system while the grammar is being 
> generated, compiled, and loaded on the servers will get the previous 
> version.  This ensures that there is no delay for the first caller 
> after a change.  We can do this as we are using the native APIs 
> provided by the ASR engine vendors we support and layer a 
> sophisticated framework and API on top of it.
>
>
>
> It would be nice if MRCP provided some support for schemes like that, 
> without having to open dummy channels to register grammars and pray 
> the MRCP server doesn't evict them too early.  An MRCP client has to 
> be able to guarantee that when a call comes through
> the system, everything is ready and it won't experience any delay.   
> As it stands now the MRCP protocol does not appear to provide support 
> for this.
>
>
>
> Thanks,
>
> Felix
>
>
>
> From: [email-masked] [mailto:[email-masked]]
> On Behalf Of Dave Burke
> Sent: Saturday, June 11, 2005 10:42
> To: Shanmugham, Saravanan; Corby Anderson; [email-masked]
> Subject: Re: [Speechsc] Managing big grammars
>
>
>
> I agree. Grammar compilation, caching, and sharing is a problem for  
> the MRCP server. Exposing this in the protocol puts unnecessary  
> complexity on the client. For example, the client needs to download  
> grammars, implement its own HTTP caching, checksum algorithms (some  
> way of not having to push huge data to the MRCP server to query) etc.
>
>
>
> So I vote for point 1 below and include in the text that an MRCP  
> server could also, as an optimisation, implement a centralised  
> common cache shared across multiple MRCP servers.
>
>
>
> Dave
>
> ----- Original Message -----
>
> From: Shanmugham, Saravanan
>
> To: Corby Anderson ; [email-masked]
>
> Sent: Saturday, June 11, 2005 12:09 AM
>
> Subject: RE: [Speechsc] Managing big grammars
>
>
>
> I would say that a common cache amoung a farm of MRCP servers to  
> optimize compilations would be a nice server side optimization and  
> shouldn't affect the MRCP protocol.
>
>
>
> my 2c.
>
> Sarvi
>
> _______________________________________________
> Speechsc mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/speechsc
>

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: "Dave Burke" <[email-masked]>
Date: Tue, 14 Jun 2005 15:51:59 +0100

For 4) I was thinking of a broadcast idea. Each MRCP server just needs to 
know the (default) multicast address. Each MRCP server (via IGMP) learns of 
a new grammar to potentially load if any one server is asked to 
DEFINE-GRAMMAR. With a well-chosen IP address e.g. in 239.0.0.0 - 
239.255.255.255 range, there is no configuration.

If a server is down, it's got to load the grammar when first required on 
restart. Assuming 1-3 this shouldn't be so bad.

- Dave

----- Original Message ----- 
From: "Reifenrath, Klaus" <[email-masked]>
To: "'David R. Oran'" <[email-masked]>; "Dave Burke" 
<[email-masked]>
Cc: <[email-masked]>; "Shanmugham, Saravanan" <[email-masked]>; "Wyss, 
Felix" <[email-masked]>
Sent: Tuesday, June 14, 2005 1:51 PM
Subject: RE: [Speechsc] Managing big grammars