Subject: RE: [Speechsc] Managing big grammars

From: "Bennett, Patrick" <[email-masked]>
Date: Tue, 14 Jun 2005 16:25:40 -0500

Ok, the message thread is starting to get a little cluttered, so I'll
just pull out individual statements to respond to them if that's ok.  If
there's something I didn't respond to that you felt I should have, feel
free to drop it back in on future replies.

> >What's the point of a standard client protocol if servers can't be
> > equally interchanged and provide equal (not least
common-denominator)
> > functionality?
> >
> Well, I suspect my company would be less than pleased if all IP
> routers could be equally interchangeable simply by conforming to
> RFC795 and friends. :-)

[Bennett, Patrick] Well, I'm sure many (not at Cisco ;>) would disagree,
but I understand what you're saying.  Unfortunately, since server
implementers are the primary forces behind MRCP at this point, it's
possible to question the motives of leaving so many holes in the
specification.  If you can't drop in another server and get at least as
much functionality as any other server then I think the specification is
a failure.  Of course, what that minimum level of functionality is,
well, that's where it gets tricky.  :)

> Distributed systems are distributed for a reason. If you want a
> system which meets specific timing and robustness constraints I don't
> think you can look to the protocol specifications to provide that.

[Bennett, Patrick] No, I just expect for deterministic behavior to be
part of the specification.  I disagree with the large number of
'optional' features in the spec which are crucial to a speech system
that handles more than a couple of users.

> >...By the spec, every
> > grammar
> > would have to be defined for every single session, every single
time,
> > and there would be *no* way of knowing if a specific grammar would
be
> > (re)compiled (it might be a grammar that takes 20 minutes just to
> > compile!) for any given session.

> That's right. If you want to be certain that all the state is in
> place before you fire off a RECOGNIZE request, the client can wait to
> enable input calls/sessions from users until it gets back COMPLETE
> responses for all the grammars it thinks will need to be active for
> the recognizer resource it is talking to. 
> It can further set the
> cache timeouts long enough to have high assurance those grammars
> don't get kicked out, and re-do the DEFINE before the cache timers
> run out. 

[Bennett, Patrick] Not possible.  The MRCP spec specifically allows the
cache information to be ignored.  It's entirely optional.  Even if the
implementor says they adhere to the HTTP 1.1 cache specs, those are
optional too.  Do you see what I'm saying?  I keep seeing messages
saying 'well, no reasonable implementation would do that..' - well, why
wouldn't they!?  The specification doesn't mandate anything of the sort.
If the spec thought it was important, it would specifically state the
importance of it, why it was necessary, and why it was a MUST
requirement for server implementers.

>I don't see how this is more onerous than exposing the
> various grammar formats and states to the client. For example,
> there's nothing which says that the server, even with a compiled
> grammar, can't have really stupid precedence handling and spend
> seconds juggling the grammar subset for a particular request in order
> to do the semantic matching properly.

[Bennett, Patrick] That's another issue.  I think the MRCP spec is
broken there as well.  If two grammars match against the same speech
utterances, that's an intentional grammar ambiguity and both results
should be returned for the client application to process.

> > Technically, it MAY take 20 minutes to
> > compile for every single caller since the server MAY not pay
attention
> > to anything I requested [even ignoring the huge problem of no global
> > grammar contexts].
> >
> I'm not sure what you mean by global grammar contexts, so I can't
> comment on that part. 

[Bennett, Patrick] Just that there is no way to compile large grammar X
such that it would be guaranteed to be cached and *instantly* available
to *all* subsequent sessions on a particular MRCP server.  Grammars have
to be defined for each individual session and it must be assumed that
each session is completely unique in terms of prior state, as there are
no hard caching requirements (caching isn't required at all) nor
cross-session grammar knowledge.

> I certainly do think that a system which takes
> 20 minutes and recompiles grammars for every RECOGNIZE session is
> pretty stupidly implemented, 

[Bennett, Patrick] Exactly.  The MRCP spec allows for this.  As an ISV
providing a platform in which speech applications can be developed using
multiple speech vendors (we allow multiple speech engines to be used
simultaneously actually), when we add MRCP as one of the available
speech integrations, we have to worry about specs that allow vendor X,
Y, or Z to behave *completely* differently.  Particularly with regard to
something as important as speech grammars.

If everyone can agree that no reasonable implementation should behave
this way, then what argument can anyone possibly have for it being
optional?

> > Even if the server MAY have cached the result and the caching is
> > spread
> > across sessions, you still have to have some way of priming the
cache.
> I think this is what DEFINE-GRAMMAR does. Why do you think it doesn't?

[Bennett, Patrick] Because all it does is define a grammar for the
current session to use.  It doesn't define a grammar that could be used
by other sessions (with no delay).
> 
> > But of course, you can't (by the spec) prime the cache from a
> > different
> > session.  You'd have to define the grammar in the context of an
active
> > call!
> What in the specification says anything about MRCPv2 sessions being
> bound to calls?

[Bennett, Patrick] Nothing - I interjected my own term.  Typically,
speech vendors have license restrictions on how their speech 'ports' may
be used [so you can't set up a pool of x resources that handle >x
simultaneous users/callers].  A 'call' is thus associated in our system
(assuming there are no failures) with a speech session at initiation
time and stays with that session for the duration of the call or until
the application developer explicitly releases the speech context.  Thus,
the ability to reference grammars in a larger context is necessary (we
compile the grammars separately from recognition sessions and cache the
vendor-specific results on distributed servers).

> 
> > What happens for the first caller into an automated attendant
> > application as the system he uses takes an indeterminate amount of
> > time
> > to compile the speech grammar before it even allows recognition!
Does
> > the customer have to force a dummy call into the system each time a
> > dynamic grammar generation process runs?
> >
> Only if they have a client which waits for an incoming call before it
> does anything to prime the server state.

[Bennett, Patrick] It's not that simple. You're thinking in terms of a
static application.  Our customers frequently create highly dynamic
systems, with grammars that are composed (or determined) on the fly
based on caller identification or input.  Since we are able to maintain
unique cached state across multiple servers, if caller Y happens to try
to use a grammar that caller X just used (which it compiled
asynchronously), we are able to guarantee that the grammar doesn't have
to be referenced or recompiled.  Even if different sessions compose a
brand new grammar if the contents of the grammar are the same, we detect
that, and use the existing precompiled version.  
With MRCP, none of this is currently possible.  There is nothing in MRCP
that specifies how grammars are kept or cached (remember, caching is
completely optional! :>), nor whether that cache crosses session
boundaries. 

> > The vendors we currently
> > support have well defined means for precompiling grammars which we
can
> > take advantage of and cache appropriately across large numbers of
> > servers.
> How can you cache a compiled grammar if you have a mix of servers
> from different vendors? 

[Bennett, Patrick] It's complicated, but we do it.  We automatically
translate SRGS (ABNF or XML form) grammars to the vendor's own formats
as well.  So, assuming developers (if you want to call them that) on our
system use SRGS for their grammars (and SISR for the semantic tags),
then their applications (we call them handlers) can automatically work
against any vendor's system we support.  Some customers are even looking
at using us because they can use vendor X which supports language X
well, and vendor Y for their language Y support.  

> I don't see how MRCPv2 prevents you from
> getting the same behavior, assuming the server implementors compile
> grammars as a side-effect of fetch, and clients ensure the fetch
> happens before the instant recognition is needed (by using DEFINE-
> GRAMMAR, for example). 

[Bennett, Patrick] Not possible.  The MRCP server may have to compile
the grammar, and it may be a very large one.  Even a modest grammar, if
it takes more than a couple of seconds to compile is a problem since a
user might be waiting at that point.

> Would it help if we inserted a SHOULD strength
> requirement that servers compile grammars when fetched and not
> respond to the associated MRCPv2 request until the compilation is
> finished?

[Bennett, Patrick] My colleague Felix Wyss wrote the following in an
earlier in a thread about caching:

Using a stale grammar for calls arriving while the new version is
fetched and compiled would be acceptable, provided:
  1) There is a flag in the DEFINE-GRAMMAR request to control this
behavior.  I can think of three possible options:
    a. Ensure grammar is always up-to date according to the HTTP
server's caching headers ("accuracy over latency")
    b. Allow stale grammars to be used ("latency over accuracy"), but
block when grammar is not in the cache at all
    c. Like b, but fail immediately if grammar is not already in the
cache; that way, the application can fallback to something else instead
of spuriously blocking.
  2) DEFINE-GRAMMAR returns a header or result-code indicating whether a
stale grammar is being used 
  3) If a DEFINE-GRAMMAR causes a stale grammar to be used, that grammar
must be used for the whole duration of that call or until the next
DEFINE-GRAMMAR is issued for it (i.e. the grammar won't unpredictably
change between recognitions just because a newer version became ready in
the meantime).  
  4) All standard conforming MRCP servers MUST implement the caching
behavior this way (i.e. it is not optional or up to the vendor like in
RFC 2616).  

Of course, this still requires the application to ping the servers to
update their caches; otherwise some calls may get very stale grammars.  

> As WG co-chair I'm responsible for both the consensus process and for
> the technical quality of the specification. So I take this discussion
> very seriously.

[Bennett, Patrick] I'm extremely relieved to hear that you are.  :)

Cheers...
Patrick Bennett

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: "Dave Burke" <[email-masked]>
Date: Wed, 15 Jun 2005 10:57:16 +0100

Here's my 4 cents (that's 1 cent per point):

1. If the MRCP server temporarily uses stale content in an effort to avoid
latency then it is reasonable that the DEFINE-GRAMMAR/RECOGNISE complete
response/event includes a warning. RFC2616 would suggest we return something
like:

    Warning: 110 10.0.0.1 Response is stale

2. Allowing DEFINE-GRAMMAR or RECOGNIZE to prefer reduced latency over
accuracy seems fine. I'd posit that the default action is to prefer reduced
latency. To enforce accuracy (e.g. as one might want during development and
testing of an application), a Cache-Control: max-stale=0 could be used. I
can't convince myself that we need an option to fail if a resource is not in
the cache.

3. Disallowing a grammar in an MRCP session to change outside of that
session's invocation DEFINE-GRAMMAR or RECOGNIZE (due to a revalidation in a
different MRCP session) seems perfectly reasonable.

4. Given that latencies are much more detrimental to VUIs than GUIs, it
seems reasonable to make caching a SHOULD. Fancy optimisations such as a
centralised cache and informing other servers in a cluster to reload seem
like they ought to remain MAYs. Going for SHOULD over MUST seems reasonable
to me to allow dtmfrecog and low-end speechrecogs. Ref. David's comment
about protocol engineering encroaching on system engineering.

-- Dave

----- Original Message ----- 
From: "Bennett, Patrick" <[email-masked]>
To: "David R. Oran" <[email-masked]>
Cc: "Reifenrath, Klaus" <[email-masked]>; "Dave Burke" 
<[email-masked]>; <[email-masked]>; "Shanmugham, Saravanan" 
<[email-masked]>; "Wyss, Felix" <[email-masked]>
Sent: Tuesday, June 14, 2005 10:25 PM
Subject: RE: [Speechsc] Managing big grammars

common-denominator)

time,
be

attention

cache.

active

Does

can

Cheers...
Patrick Bennett