Subject: RE: [Speechsc] Managing big grammars

From: "Bennett, Patrick" <[email-masked]>
Date: Tue, 14 Jun 2005 14:30:35 -0500

[responses inline]

> -----Original Message-----
> From: David R. Oran [mailto:[email-masked]]
> Sent: Tuesday, June 14, 2005 11:53 AM
> To: Bennett, Patrick
> Cc: Reifenrath, Klaus; Dave Burke; [email-masked]; Shanmugham,
> Saravanan; Wyss, Felix
> Subject: Re: [Speechsc] Managing big grammars
> 
> <chair hat>
> <philosophy alert>
> 
...
> 
> If you are a system-builder you need to ensure that the components
> you select (e.g. MRCPv2 servers) meet the performance requirements
> for your applications. 

[Bennett, Patrick] Ah, but there's the rub!  I don't select the MRCP
server.  The customer does!  If they pick vendor Z for their system
because it's an "MRCPv2" server (and we list MRCPv2 as one of our
supported server types), and that vendor implemented the bare minimum of
MRCP (while staying completely within the 'spec'), that customer
wouldn't be too happy.
I couldn't blame it on the vendor either.  They'd be within the spec.

What's the point of a standard client protocol if servers can't be
equally interchanged and provide equal (not least common-denominitor)
functionality?

> The current MRCPv2 design, as elaborated on by
> Sarvi, allows a client to ascertain to a limited degree whether a
> server is going to perform adequately (through the cache control
> parameters), but not to force a server to use any particular means to
> do so. I think this is the right tradeoff. You're certainly entitled
> to disagree.

 [Bennett, Patrick] I guess we'll agree to disagree then.  If the server
'may' pay attention to these parameters, it really doesn't mean a whole
lot.

> > If needs to be to be usable by carrier grade and scale implementors.
> > This isn't possible when the MRCP servers MAY do something.
> >
> I agree in the case where the results are semantically different and
> it is impossible for the client to ascertain what the server does,
> but not in cases where the alternative behaviors are explicitly there
> for flexibility and to avoid over-specification.

[Bennett, Patrick] As currently defined, DEFINE-GRAMMAR is well, pretty
much completely undefined.  I don't mean to be too aggressive here
(writing an RFC is extremely difficult!), but c'mon.  Here is
effectively, the entire definition of DEFINE-GRAMMAR: 
  "The DEFINE-GRAMMAR method, from the client to the server, provides a
  grammar and tells the server to define, download if needed and compile
the 
  grammar."
The rest just defines when it can be called.  By the spec, every grammar
would have to be defined for every single session, every single time,
and there would be *no* way of knowing if a specific grammar would be
(re)compiled (it might be a grammar that takes 20 minutes just to
compile!) for any given session.  Technically, it MAY take 20 minutes to
compile for every single caller since the server MAY not pay attention
to anything I requested [even ignoring the huge problem of no global
grammar contexts].

Even if the server MAY have cached the result and the caching is spread
across sessions, you still have to have some way of priming the cache.
But of course, you can't (by the spec) prime the cache from a different
session.  You'd have to define the grammar in the context of an active
call!  What happens for the first caller into an automated attendant
application as the system he uses takes an indeterminate amount of time
to compile the speech grammar before it even allows recognition!  Does
the customer have to force a dummy call into the system each time a
dynamic grammar generation process runs?  

> > I guess I could just say, the MRCP spec MAY be unusable for anything
> > other a simple test system with trivial, *static* grammars with no
> > more
> > than a few concurrent sessions.
> >
> I think we have deployed experience with MRCPv1 (which says even less
> on the subject) that contradicts this observation.

[Bennett, Patrick] I guess I'll have to take your word for it, but I
doubt those successful (large-scale, right?) deployments can work with
any drop in MRCPv1 server.  If these servers were as you said selected
to "...meet the performance requirements for your applications..." then
what was the point of MRCP?  The implementer might as well have used the
speech vendor's own proprietary API!  I doubt they could truly drop in
another vendor's MRCP server.

Basically, we (Interactive Intelligence, Inc.) as a part of implementing
our own vendor-agnostic speech recognition framework succeeded in
creating an extremely scalable and robust system.  In the near future,
we plan on adding support for MRCP servers in addition to the vendors we
already support.
Unfortunately, as defined, MRCP would be a huge step backwards for us
compared to functionality we already have.  The vendors we currently
support have well defined means for precompiling grammars which we can
take advantage of and cache appropriately across large numbers of
servers.

One of our concerns is that some vendors (e.g. Nuance) have suggested
that they're going to drop their existing APIs and start supporting MRCP
only.  This is why a poorly defined MRCP spec is particularly troubling
for us. 

Patrick Bennett

_______________________________________________
Speechsc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/speechsc

---

From: "David R. Oran" <[email-masked]>
Date: Tue, 14 Jun 2005 16:10:11 -0400

> equally interchanged and provide equal (not least common-denominitor)
Well, I suspect my company would be less than pleased if all IP  
routers could be equally interchangeable simply by conforming to  
RFC795 and friends. :-)

Distributed systems are distributed for a reason. If you want a  
system which meets specific timing and robustness constraints I don't  
think you can look to the protocol specifications to provide that. I  
understand that some other standardization bodies approach the  
problem differently (e.g. ITU with SS7) and specify both the  
protocols and the system engineering as part of the same documents.

There are plusses and minuses to that. The IETF and the W3C tend to  
eschew any attempt to creep over the line into system engineering as  
part of the protocol definitions.
>   grammar and tells the server to define, download if needed and  
well, it's "defined" just fine, it's just that the definition doesn't  
appear to do what you want.
It's a tradeoff about what level of internal server operation to  
expose to the client. You're arguing for exposing a lot, but I don't  
see a whole lot of support for this position from other WG members,  
at least not yet. If you persuade folks that more explicit control by  
clients is necessary we'll adjust the specification accordingly.

> compile!) for any given session.
That's right. If you want to be certain that all the state is in  
place before you fire off a RECOGNIZE request, the client can wait to  
enable input calls/sessions from users until it gets back COMPLETE  
responses for all the grammars it thinks will need to be active for  
the recognizer resource it is talking to. It can further set the  
cache timeouts long enough to have high assurance those grammars  
don't get kicked out, and re-do the DEFINE before the cache timers  
run out. I don't see how this is more onerous than exposing the  
various grammar formats and states to the client. For example,  
there's nothing which says that the server, even with a compiled  
grammar, can't have really stupid precedence handling and spend  
seconds juggling the grammar subset for a particular request in order  
to do the semantic matching properly.

> Technically, it MAY take 20 minutes to
I'm not sure what you mean by global grammar contexts, so I can't  
comment on that part. I certainly do think that a system which takes  
20 minutes and recompiles grammars for every RECOGNIZE session is  
pretty stupidly implemented, just like a JAVA system which doesn't  
cache compiled JAVA code and recompiles it every time a method is  
invoked would be pretty stupid. It might even cause a plane to crash  
if you tried this in an avionics system.

> across sessions, you still have to have some way of priming the cache.
I think this is what DEFINE-GRAMMAR does. Why do you think it doesn't?

> call!
What in the specification says anything about MRCPv2 sessions being  
bound to calls?

> What happens for the first caller into an automated attendant
Only if they have a client which waits for an incoming call before it  
does anything to prime the server state.

> what was the point of MRCP?
Interoperability.

> The implementer might as well have used the
They could if they used MRCP and the set of acceptable MRCP server  
implementations was non-null.

> compared to functionality we already have.
New interoperable protocols are, for better or worse, usually a step  
backward from highly optimized proprietary solutions.  
Interoperability by any regime other than blind superset (with its  
attendant complexity explosion) doesn't come for free.

> The vendors we currently
How can you cache a compiled grammar if you have a mix of servers  
from different vendors? I don't see how MRCPv2 prevents you from  
getting the same behavior, assuming the server implementors compile  
grammars as a side-effect of fetch, and clients ensure the fetch  
happens before the instant recognition is needed (by using DEFINE- 
GRAMMAR, for example). Would it help if we inserted a SHOULD strength  
requirement that servers compile grammars when fetched and not  
respond to the associated MRCPv2 request until the compilation is  
finished?

> only.  This is why a poorly defined MRCP spec is particularly  
<chair hat>
I agree that if MRCPv2 is poorly defined/designed it would be  
troubling, and for all of us - not just for you. Where I think  
there's still some lack of consensus is over the adequacy of the  
existing MRCPv2 machinery.

As WG co-chair I'm responsible for both the consensus process and for  
the technical quality of the specification. So I take this discussion  
very seriously.

</chair hat>