Subject: Re: [mpowr] WG Formation

From: Harald Tveit Alvestrand <[email-masked]>
Date: Sun, 15 Feb 2004 20:08:05 -0800

actually I see at least 3 different types of BOF being run in the IETF:

- Proposals, often externally initiated, for IETF activities. There, 
evidence of an active, open community/discussion is the first thing we ask 
for.

- "This is a problem" BOFs. Examples are various SPAM BOFs, the Anycast 
BOFs  and so on - things where we (members of the community, some set of WG 
chairs, IESG members.....) recognize that there's a problem, but it doesn't 
seem like WG material, and it doesn't belong to any WG's scope.
At the upcoming IETF, it remains to be seen whether MXCOMP will be such a 
BOF or the initation of a WG activity. Here, making a separate mailing list 
may be a result of the decision to have a BOF, rather than a precondition - 
and sometimes a mailing list is even inappropriate.

- "Haven't finished the paperwork" BOFs - things that are obviously going 
to be a WG, but for some reason the charter hasn't been approved (such as 
haven't got WG chairs, can't agree on a name, didn't get the review 
messages sent in time....). Examples were L2VPN and L3VPN at the time of 
the split.

One thing I really liked about draft-iesg-hardie-outline was the discussion 
of the various types of WGs. BOFs, too, don't come in "one size fits all".

When we have a reason to have a meeting at the IETF that doesn't fit the WG 
category, we usually call it a BOF.
That means - some variety is to be expected, and some judgment needs to be 
exercised.
I hope that works, most of the time......

                                Harald

_______________________________________________
mpowr mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/mpowr

---

From: Dave Crocker <[email-masked]>
Date: Sun, 15 Feb 2004 21:45:31 -0800

Harald,

HTA> - Proposals, often externally initiated, for IETF activities. There, 
HTA> evidence of an active, open community/discussion is the first thing we ask 
HTA> for.

Excellent.  Then it is not a big step to make it a requirement, is it?

Except that most of the formative BOFs that I've been to in the last few
years have had little or no prior activity.

And the requirement I am suggesting is for actual work getting done.
That's different from simply having active discussion.  Active
discussion is easy.  Make progress is not.

d/
--
 Dave Crocker <dcrocker-at-brandenburg-dot-com>
 Brandenburg InternetWorking <www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>

---

From: John C Klensin <[email-masked]>
Date: Mon, 16 Feb 2004 01:14:43 -0500

--On Sunday, 15 February, 2004 21:45 -0800 Dave Crocker 
<[email-masked]> wrote:

> Harald,
>
> HTA> - Proposals, often externally initiated, for IETF
> activities. There,  HTA> evidence of an active, open
> community/discussion is the first thing we ask  HTA> for.
>

I.e. --
	"the requirement won't add much, so we should do it"?

Wouldn't a better interpretation be
	"we are doing this already most of the time, and maybe
	flexibility is useful, so why tamper with it" or

	"what we are doing isn't working well enough, so why
	write it into a rule"

Now, from my perspective, the more interesting question to ask 
would have been:
	"And how much real information comes out of these BOFs?
	If the answer is often 'not much' and the IESG has the
	sense that the group will end up chartered as a WG
	sooner or later anyway, might it not be better to just
	set them up as a WG on a trial basis and with a short
	leash?"

> Except that most of the formative BOFs that I've been to in
> the last few years have had little or no prior activity.

FWIW, I'd say the ones I've been to have been about evenly 
divided between "little or not prior activity" and "enough prior 
activity, some of it possibly in poor directions" that efforts 
to recalibrate their directions would be "late surprises" even 
at the time a BOF is held.

      john

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Mon, 16 Feb 2004 08:40:05 -0800

--On 15. februar 2004 21:45 -0800 Dave Crocker <[email-masked]> wrote:

> HTA> evidence of an active, open community/discussion is the first thing
> we ask  HTA> for.
>

Dave, at the risk of beating an old, tired drum once again:
Can you name some of these BOFs, so that we can check whether all of us 
have the same perception of whether there was prior activity?

> discussion is easy.  Make progress is not.

On the last two sentences, we fully agree :-)

But the point I was trying to make was a completely different one - that we 
make BOFs for many reasons. For some types of BOFs, it's appropriate to ask 
for prior activity on a mailing list. For others, it's not.
So we either have to invent new names for subgroups of what we currently 
call "BOFs", or we have to trust someone's judgment on when to require 
proof of prior activity.

Which one is the best use of resources?

                           Harald

---

From: Dave Crocker <[email-masked]>
Date: Mon, 16 Feb 2004 09:14:30 -0800

Folks,

Since John and Harald seem to be missing the points I've been trying to
make, I will assume that others are also missing it.  So, here is a
concise re-statement:

The IETF is good at facilitating working groups that have clear,
practical direction, with significant participation and an ability to
conduct productive discussion in a timely fashion.

The IETF has a poor track record of "fixing" broken working groups. In
particular, working groups that lack cohesive, productive discussion
rarely change.

Problematic working groups are a significant drain on IETF resources,
including time for management oversight, consumption of valuable
participant time, and consumption of scarce meeting time. Problematic
working groups also have an opportunity cost. They set expectations for
a solution that is, in fact, unlikely to be produced in a timely or
useful fashion.

It is to the IETF's general benefit to find ways to avoid this waste of
resources and tone of frustration and non-productivity.

One approach that will help is to require that working group formation
be predicated on a demonstrated ability to conduct productive discussion
over a mailing list and with enough participation to demonstrate
meaningful constituency.

This _adds_ to the list of IETF BOF and working group chartering
pre-requisites. It requires that groups expected to be productive and
timely first show that they can be. In fact, this requirement is already
listed as being optional, in RFC 2418 (IETF Working Group Guidelines and
Procedures), before holding a pre-chartering BOF.

So the only change would be to make it always required, before
chartering and before holding a pre-chartering BOF.

An alternative would be to continue to charter working groups that lack
a track record for timely productivity, and continue to task the working
group chairs and cognizant area director with fixing the working groups
that fail to become coherent and productive in a timely fashion.

The problem with this approach is that it is very expensive and it does
not work. It has not worked reliably -- and probably has not worked at
all -- for any of the life of the modern IETF. Rather, it dilutes
allocations of time and focus, and it reduces the community sense of
productivity. All three of these are, in fact, problematic in the
current IETF.

Over the life of the IETF, some area directors have employed the policy
of requiring a group to demonstrate the requisite capabilities, before
chartering as an IETF working group.  In fact, some area directors have
required this before permitting a BOF to be held.  A BOF session is
very short and BOFs drain from the total pool of congested IETF
meeting time.

The suggestion is to move this requirement from "sometimes" to "always",
just as we always require a coherent charter with constituency support,
before creating a working group.

This moves the responsibility for timely productivity entirely to the
nascent working group, rather than imposing any of that requirement on
IETF management.

The nascent group must, of course, interact with IETF management and
IETF participants. However there is a considerable difference between
the obligations and costs of dealing with chartered efforts, versus
other efforts. Hence, a nascent group's ability to recruit participation
and assistance is a significant "market" test of that group's effort.

d/

---

From: Dave Crocker <[email-masked]>
Date: Mon, 16 Feb 2004 09:48:00 -0800

Harald,

>> years have had little or no prior activity.

HTA> Dave, at the risk of beating an old, tired drum once again:
HTA> Can you name some of these BOFs, so that we can check whether all of us 
HTA> have the same perception of whether there was prior activity?

No.

It would distract from the discussion.

However your query highlights a basic difference in views about
discussing and resolving strategic issues in the IETF.

Either there is rough consensus that we have a problem with working
groups and BOFs that are unproductive, or we don't. If we don't, then
let's move on. I'll shut up. If we do have rough consensus about this,
then we do not need to cite statistics.

Let me repeat:  If there is rough consensus that BOF time is generally
well spent, then everyone should please ignore my suggestion.

On the other hand, if there is rough consensus that BOF time is too
often wasted or, at least, inefficient, then we can focus on the nature
of my suggestions rather than on bookeeping.

Similarly, there is a basic difference in views, between "empowering"
groups to self-form and demonstrate capabilities, versus tasking
volunteer management with "fixing" those groups that lack the
capabilities.

If there is rough consensus that IETF management fixes things well, then
again, ignore my suggestion and let's move on.

If there is rough consensus that problematic working groups do not get
fixed all that well and do constitute a problem, then let's talk about
solutions.

One solution is to expect IETF management to do a better job of fixing
things, or of generally "managing" things better.

Another is to look for ways to prevent the problems in the first place,
or at least to move the problems to be outside the IETF.

HTA> But the point I was trying to make was a completely different one - that we
HTA> make BOFs for many reasons. For some types of BOFs, it's appropriate to ask 
HTA> for prior activity on a mailing list.

That's nice.  My own focus has been on pre-chartering BOFs.

HTA>  or we have to trust someone's judgment on when to require
HTA> proof of prior activity.

HTA> Which one is the best use of resources?

That sounds like the status quo.  Forgive me, but I thought that that
was not working very well.  As in, it has not been working at all.  So
why will it work in the future>?

Rather than trivialize this as a labeling excercise, I suggest we look
at real problems and pursue real solutions.

"Trust someone's judgement" has not been working. I believe we have
community consensus on this point. That means we need to look for
changes to make.

If you are suggesting a change, I'm not understanding what it is.

d/

---

From: Thomas Narten <[email-masked]>
Date: Tue, 17 Feb 2004 12:11:18 -0500

> conduct productive discussion in a timely fashion.

Note: as an AD, I find that making the judgement call as to whether an
effort (e.g., BOF preparation and/or mailing list BOF preparation) is
actually a "productive discussion" is not always so easy, and that my
assessment is frequently at odds with that of those participating in
or driving the effort. There are multiple dimensions here. Ability to
articulate the root problem. Ability to see potential solutions that
solve the exact problem (and no more). Willinginess to keep the
solution simple. Willingness (or ability) to see implications of a
particular approach on other parts of the Internet which are not an
immediate focus of the WG. Etc.  And because there is a judgement call
(on which different people may not agree) on many of these points,
coming up with clear criteria that folks will generally agree on is
probably not so easy.

> resources and tone of frustration and non-productivity.

I suspect everyone would agree that finding a way to address this
would be good!

> over a mailing list and with enough participation to demonstrate

I think that in practice this is already being done. I'd be interested
in hearing of specific cases where it is not. But as I said earlier,
people's views of "productive" vary, so in the end there is often a
judgement call that has to be made.

As one example (since folk here probably aren't aware of it...) Some
folk asked for an anycast BOF for the upcoming IETF. The RTG and INT
ADs were involved. We basically said no, because we weren't convinced
that enough prep work had been done for a BOF, even though the
proponent(s) probably felt otherwise. We specifically wanted more
mailing list discussion, a more clear problem statement and IDs. But
they did have a mailng list, a proposed charter and pointed to a
number of IDs... 

> that fail to become coherent and productive in a timely fashion.

One thing the above doesn't help with is work areas where there are
clearly folk interested in working on the problem, the problem is
real and in need of a solution, but the effort needs help getting on
track. Is it better to work with the group within the IETF? Or to keep
them at arms length outside and hope the can figure things out for
themselves?

I think the answer depends on many factors, but when the problem is of
importance to the IETF, and there is enough activity going on that if
the IETF ignores it, it will have to deal with the consequences later
anyway, keeping them at arms length doesn't seem like the better
alternative.

> very short and BOFs drain from the total pool of congested IETF

Speaking personally, I've certainly hosted a number of BOFs that
didn't seem like they were that well run or had prepared
adequately. At the same time, those efforts usually did have a fair
amount of behind-the-scene steering/encouragement from the ADs (and
others) trying to make them better and good enough. But at the end of
the day the question is: is the community better served by letting the
BOF (with whatever shortcomings) go forward, or should it be delayed
for at least another cycle in attempt (hope?) to make things more
baked? Pushing off a BOF for another cycle has the downside of
delaying an effort and/or opening the AD up to criticism that they
just don't (for whatever reason) like the BOF (or an ID, or  a
particular solutions, or...).

> before creating a working group.

I certainly look for this before creating a WG. Among the factors I
look for:

 - clear problem statement? Well-scoped and understood? (and is it
   written down so folk are agreeing to the same words?)
 - Is the problem important?
 - Is there really a need for a standard? (I.e., multiple vendors will
   need to implement this, solves an end-user problem)
 - are there sufficient worker bees? (editors, chairs, etc)
 - are the sufficient reviewers (with necessary expertise)?
 - do the participants seem to be largely in agreement on what the WG
   needs to do and how to get there? (or will this group be hard to
   manage and keep on track)?
 
> nascent working group, rather than imposing any of that requirement on

Certainly, there is a time for this. But I think there are too many
differences in each BOF situation to make hard-and-fast rules that
MUST be adhered to.

Thomas

---

From: Dave Crocker <[email-masked]>
Date: Tue, 17 Feb 2004 09:43:31 -0800

Thomas,

I had a more detailed response written, and then decided the most

>> nascent working group, rather than imposing any of that requirement on
TN> Certainly, there is a time for this. But I think there are too many
TN> differences in each BOF situation to make hard-and-fast rules that
TN> MUST be adhered to.

Do we have a problem?  Is the pattern of BOFs typically productive? Is
the timely productivity of a working group distinguished by the up-front
preparatory work that is done?  Do working groups that start badly end
well?

Should we leave all of the details and procedure fuzzy, to be decided by
the subjective assessment of individual ADs, as they are now?

If we do not have a problem, then we certainly should not try to fix it.

If we do have a problem, then what are you suggesting for fixing it?

d/

---

From: Dave Crocker <[email-masked]>
Date: Wed, 18 Feb 2004 18:08:48 -0800

Thomas,

TN> immediate focus of the WG. Etc.  And because there is a judgement call
TN> (on which different people may not agree) on many of these points,
TN> coming up with clear criteria that folks will generally agree on is
TN> probably not so easy.

Your list is quite good.  And the subjectivity in applying the criteria
is exactly why my suggestion does not specify any of those details.

In other words, it proposes a rigid requirement for some activities that
will provide very useful -- and very visible -- input to the chartering
process.

However, it wisely does not alter the core nature of that process.
Notably, the judgement call remains as it is now. And again we should
note that the suggestion is for something that is already done, just not
often enough.

>> over a mailing list and with enough participation to demonstrate

TN> I think that in practice this is already being done.

As I said, if there is rough consensus that we do not have a problem
with BOFs (and, yes, working groups) being authorized without sufficient
preparatory work, then my suggestion should not carry the day.

TN> As one example (since folk here probably aren't aware of it...) Some

There are many examples of good decisions. The problem is with the
pattern for the IETF. If folks think the _pattern_ or preparatory work
is acceptable, then let's move on to a topic that folks feel _is_ a
problem.

>> that fail to become coherent and productive in a timely fashion.

TN> One thing the above doesn't help with is work areas where there are
TN> clearly folk interested in working on the problem, the problem is
TN> real and in need of a solution, but the effort needs help getting on
TN> track.

It's probably worth exploring the term "getting on track".

I believe that it usually refers to a lack of clear focus, typically
including a problem and solution statement that is too fuzzy.  Such
situations are not yet ready for a standards effort, because they do not
know what they want to standardize.  Maybe they need an IRTF effort.
Maybe they just need the usual array of organizing pre-standards effort
that _any_ standards effort requires.

My own sense of things is that we do a really terrible job at getting
fuzzy efforts on track after a group is chartered.

So let's stop trying.

TN> I think the answer depends on many factors, but when the problem is of
TN> importance to the IETF, and there is enough activity going on that if
TN> the IETF ignores it, it will have to deal with the consequences later
TN> anyway, keeping them at arms length doesn't seem like the better
TN> alternative.

Importance to the net is, of course... important. However a failure to
develop a clear working path is a show-stopper. The IETF has
demonstrated a pretty clear pattern of success only when folks who will
work on a problem have a clear, shared sense of the problem and the
approach towards solving it.

I believe that working groups that are
chartered without having this usually fare very poorly, and that they
take a long time and often take a lot of resources to achieve that poor
result.

Does that mean keep the status quo for dealing with BOFs or did you have
a suggestion for something that will make them more useful?

d/

---

From: Thomas Narten <[email-masked]>
Date: Fri, 20 Feb 2004 14:13:49 -0500

Dave,

> preparatory work, then my suggestion should not carry the day.

I don't think the problem with underperforming WGs can be attributed

> approach towards solving it.

I think this is one of the better predictors for success. When the
folks in the WG (and the WG Chair/AD/IESG) are all on the same page in
terms you describe above.

But it is not all that uncommon for an effort (whether BOF or even WG)
for there to be much more diversity in where folks heads are at. The
challenge then is finding a way to steer towards a shared
vision/direction. And in some sense, that is part of the what the WG
process is all about. I.e, finding (or developing) consensus among the
set of participants.

> a suggestion for something that will make them more useful?

My own view is that the BOF->WG process does have problems. Namely, as
I recall the original message of the thread mentioning , we have cases
where it just seems to take longer than it should from start to
finish. I don't have any obvious answers.

What I think might be useful though it see if we can understand the
problem better than "it's not working". I.e., it might make sense to
look at 5-10 BOF->WG efforts that happened in the last year or so, and
try to understand how well the system worked (or didn't work) in each
of those cases. There may be some general observations/lessons there
that can be used to improve future efforts.

Thomas

---

From: Dave Crocker <[email-masked]>
Date: Sat, 21 Feb 2004 22:42:02 +0900

Thomas,

>> approach towards solving it.
TN> I think this is one of the better predictors for success.

OK.  Let's work with that point of agreement:

What I mean with the term "preparatory work" is the really difficult
stuff of

      a) being very clear about what problem is being solved,

      b) so clear that there is a very good idea how it will get used,
      and

      c) a clear, strong vendor push, and

      d) a reasonably clear customer pull.

In other words, what is the problem? what is the line of attack on the
problem; is there good indication of interest by folks who will do the
work; and is there good indication of interest by folks who will use
the result?  (All of this should sound pretty familiar, since it is
already supposed to be the basis for approving a working group.)

TN> When the
TN> folks in the WG (and the WG Chair/AD/IESG) are all on the same page in
TN> terms you describe above.

right.

TN> But it is not all that uncommon for an effort (whether BOF or even WG)
TN> for there to be much more diversity in where folks heads are at. The
TN> challenge then is finding a way to steer towards a shared
TN> vision/direction.

right.

TN>  And in some sense, that is part of the what the WG
TN> process is all about. I.e, finding (or developing) consensus among the
TN> set of participants.

Sure.  But the question is what is reasonable for a chartered working
group to formulate consensus on, and what consensus must exist _before_
chartering?

The question is how much shared sense of things there is, at the
start, versus how much risk there is of getting enough shared sense,
to get work done, _after_ working group chartering. Some cliche like
"critical mass" applies here.

And, yes, different groups can have very different degrees of working
things out and different timeframes for doing it.

The question is what the IETF should spend its scarce resources on?

We have serious and consistent patterns of working groups being
neither timely nor productive.  Having a _really_ good chartering
process and _really_ demanding both a rock solid charter and a strong
indication of group focus _before_ chartering does not guarantee
timeliness or productivity.  But it sure helps.

>> a suggestion for something that will make them more useful?
TN> My own view is that the BOF->WG process does have problems. Namely, as
TN> I recall the original message of the thread mentioning , we have cases
TN> where it just seems to take longer than it should from start to
TN> finish. I don't have any obvious answers.
TN> What I think might be useful though it see if we can understand the
TN> problem better than "it's not working". I.e., it might make sense to

Thomas, we have been in a state of terminal crisis for at least 2
years.  How much longer shall we "study" it?

TN> try to understand how well the system worked (or didn't work) in each
TN> of those cases. There may be some general observations/lessons there
TN> that can be used to improve future efforts.

Well, I think it's fine to do such a study.  What is not fine is
waiting around for another two years, making no improvements to the
IETF other than training and moving around some administrative duties
Or should we, perhaps, break this pattern of paralysis?

So let's consider this more simply:

BOFs are, at best, inefficient. Working groups do, in fact, typically
start with little clue about their work. Working groups often do, in
fact, thrash for a long time. And all of this does, in fact, drain the
life out of the IETF.

So the suggestion is to raise the bar for entry of new BOFs and
working groups. This permits us to focus our limited resources and
energy far better, to the groups that come to the IETF with their act
together (or, at least, _more_ together).

When your study shows that we can lower the bar, then we can lower it.

d/