[gnso-wpm-dt] WPM: Summary of 16 Feb 2010 Session (Step 6-In Progress)

["Ken Bour" <ken.bour@xxxxxxxxxxx>]

WPM Team Members:

 

I apologize once again for a longish email summary, but I attempted to
capture the decisions made during our teleconference today, 16 Feb 2010 and,
also, to flesh out some of the discussions that remained unresolved as our
time expired.  Ultimately, we are going to need a record of our
deliberations and I find that this writing process helps me congeal my own
thinking.  In preparing each of these reports, short and long, I frequently
research earlier summaries to recall what was said/done and, in many
instances, I end up consolidating material in an effort to sustain forward
progress.  Although this approach produces more written material to digest;
hopefully, you will agree that there is "method in the madness."  

 

Following my normal procedure, I will create a 2nd email to include the
Agenda Topics and Action Items for our next session scheduled for 23 Feb
2010 (1700 UTC).  Since there may be an opportunity to make progress via the
email list on some/all of the questions contained below, I will wait for
several days before compiling the separate email of agenda topics.  

 

1)      Project Criteria Definitions:  Definitions for Value and Difficulty
approved as drafted in Ken's 14 Feb email.  

2)      Step 6 Analysis/Questions:

a)      Rank order ties will be permitted.  If Council needs to decide (for
any reason) between one or more tied projects, then it will rate each
project's "Difficulty" as defined by the WPM team employing a process
similar to the one used for rating Value.  

b)      Frequency:  a formal prioritization rating session (all relevant
projects) will be conducted 3 times per year at ICANN meetings.  The team
noted that a face-to-face working session may facilitate using the Delphi
group ratings approach.  Specific methodology issues/questions are taken up
in (f) below.  [KB Note:  Chuck and I had an email exchange on the list
regarding the "political morale" question, which included the possibility of
creating a new exemption category (or two).  I had that email ready for
discussion today, but we did not have sufficient time.  May I suggest that
others register your thoughts in the intervening week, if you have an
opportunity?  If we do not resolve that question via the list, I will add it
to the Action Items for next time.] 

c)      New Projects:  when a new project arises, the Chair will assess
whether it can be addressed effectively by the Council (see d below) without
requiring that an official Value rating be determined.  If a specific
rating/ranking is deemed necessary to proceed with the Council's management
responsibilities, then the Chair will call for the new project to be
formally rated before further action is taken; otherwise, that project will
be officially prioritized at the next scheduled session (e.g. next ICANN
meeting). 

d)     New Project Questions:  the following questions were slightly
modified per today's discussion and are repeated to give the team another
look at them.  

o   Should this new project be undertaken, that is, have resources assigned
to accomplish a particular objective? 

o   What resource types, skills, and quantities are needed to adequately
staff this project? 

o   Are there sufficient resources (Staff and Community) available without
causing adverse impacts to other project work in progress?  If not, should
any other project work be stopped or postponed? 

o   Should this new project have a date/deadline imposed, thus establishing
urgency?  If it is determined to be urgent, can any real consequences be
identified that will cause the date to be perceived and treated as critical?


e)      Project Status Changes:  as noted in b above, updates will be
officially recognized and captured 3 times per year as reflected in the
Value ratings assigned.  Status changes will also be considered when new
projects are evaluated (see c-d above).  If, at any other time, conditions
warrant a formal reevaluation, the Chair may make such a recommendation to
the Council.  [KB Question:  does this mean that the Chair can call for
another prioritization exercise in between scheduled sessions?  If not, I am
not sure how these "status changes" would be captured or reflected.  The
team has not created any recording process, yet, for status changes other
than the actual Value ratings.]  

f)       Methodology:  several alternatives were briefly discussed as
today's call came to a conclusion.  I will attempt to summarize 4 Options
below including pros/cons as I perceive them.  Please feel free to
add/change/delete any of the advantages or disadvantages.  The options are
not presented in any order of importance or significance other than how they
occurred to me as I began sketching them.  

Option 1:  

Each Council member rates all projects individually in advance (e.g.
spreadsheet template); then a single Council Delphi session is held with all
members participating.  Process would include looking for mathematical
consensus, e.g. Std Dev<1.0 or Range<=2, ahead of the group session.  With
20 raters, I think it may be statistically unlikely, but it could happen.
Staff facilitates an Adobe Connect session using the polling feature and
blind voting.  The polling process ends when all individual scores occupy no
more than 3 adjacent ratings or Range<=2 (e.g. 4, 5, 6 or 2, 3, 4); the
median becomes the final rating.  

Pros:  

.         After one single iteration, all projects are rated and, thereby,
ranked (ties permitted).  

.         Prioritization occurs through group deliberation with all
viewpoints expressed and heard by everyone.

.         If successful, this process should result in a high level of
acceptance by the Council and the community.  

Cons:  

.         Time:  the WPM's experience with 5 participants was an average of
7 minutes discussion/voting per project (in both Y and X sessions).  Fifteen
(15) projects would take a minimum of 105 minutes not counting setup,
explanations, process discussions, etc.  

[KB Questions/Comments: 

1)      We haven't tested, so we must ask:  will 7 minutes per project hold
up with 20 potential participants?  Instead of 4-5 individuals expressing an
opinion, what if 10-15 Councilors elect to speak individually?  Should there
be time limits imposed (e.g. green/yellow/red)?  If the team believes that 7
minutes is a reasonable benchmark, then a 2 hour session would be barely
adequate - the first time through the process.  Note that at 10 minutes
average per project, the effort would require closer to 3 hours (180
minutes) including 20 minutes to start and 10 to wrap-up.  

2)      With 20 participants, how likely is it that a Range<=2 would occur
on the 1st polling vs. requiring multiple iterations?  Should a limit of 3
polling iterations be instituted in the event that, after the 3rd attempt, a
Range<=2 has not been achieved?  I assume that we would still take the
median score at that point.  

3)      Irrespective of the number of participants, I also wonder whether a
face-to-face session might actually prolong the discussions more so than
participation remotely via telephone. 

4)      Restating Wolf-Ulrich's concept, would it accelerate the Council's
process if the WPM provided an initial rating as input?] 

 

Option 2: 

Extending another of Wolf-Ulrich's ideas in a slightly different direction,
Ken offered an approach with these feature elements:  the Council asks each
SG to appoint one Council member to be a "delegate" to a standing GNSO
Project Prioritization Committee.  The non-voting NCA would be a 5th
delegate and represent all other non-SG (e.g. ALAC).  With Staff assistance
(TBD), each delegate would be responsible for developing a composite project
rating by soliciting/aggregating input from his/her SG.  Delegates would
then attend a single rating session (stay with 3x per year?) in which they
would use the group Delphi approach facilitated via Adobe Connect (same
procedures/rules as Option 1).  One delegate would be elected "Committee
Chair" by the team and that position would conduct/manage rating sessions,
direct Staff's involvement, and report the results to the Council.  

Pros:

.         Soliciting and consolidating Constituency/SG input is a common
Councilor function which would be applied to this task.

.         SG prioritizations are handled in the "background" and the group
session involves only 5 individuals - similar to the WPM's testing
experience.  The process would still take 2 hours, in all likelihood, but
would involve only 5 individuals vs. 20.  

.         Delegates would gain proficiency and efficiency both in collecting
SG input and working with colleagues on group ratings.  

.         After a single group session of the Committee, all projects are
rated and ranked. 

Cons:

.         Process might not enjoy the same acceptance level as Option 1,
especially if delegates are perceived to have yielded too much ground in a
quest to produce consensus.  

.         Delegates might tend to fight for their ratings more vigorously
than would be the case if individual Councilors rated independently and
without win/lose pressure.  

 

Option 3:

The Council is sub-divided into small groups:  RySG - 3; RrSG - 3; CSG - 6;
NCSG - 6; NCA's - 3) and each one uses the WPM procedures/rules outlined in
Option 1 (above) to produce a consensus rating for all 15 projects.  The
median from the 5 individual group results becomes the Council rating for
each project.  

Pros:

.         Smaller groups of individuals from like-minded communities may
achieve consensus more easily and efficiently since perceptions and goals
tend to be congruent. 

Cons:

.         There may be very different rating outcomes between the 5 groups,
which will be "averaged away" in calculating the medians. 

.         If the groups do produce very different rating results, the
acceptance level of the final prioritization may suffer. 

 

[KB Note:  this option could also be reworked with small groups being
selected more or less randomly so that each one has a mix of contracted and
non-contracted parties.  If the team thinks I should scope out that option,
I would be pleased to do so.]  

 

Option 4:

 

Each Council member rates all 15 projects independently (e.g. spreadsheet
template; no group discussions) and the results are averaged or the medians
computed.  

 

Pros:

.         Most efficient option and requires least amount of time.  

Cons:

.         Statistical centering is very likely and, therefore, there may
also be a much higher incidence of ties.  For a simple illustration, two
ratings of 1 and 7 produce a 4 both for mean and median, but that result
doesn't capture the reality of what really happened in the prioritization
process.  The team witnessed this same phenomenon when its individual
ratings (only 6 raters) were averaged (mean and median).  As we observed and
calculated, only 4 of the 15 projects achieved "natural" consensus
(Range<=2) in the Value ratings until after the group discussions took
place.  There was considerable clustering of the ratings; in fact, 11/15 of
the Y means were between the range of 3.7 and 5.7 and 10/15 of the Y medians
were between 3.5 and 5.0.  

.         As a result of averaging, acceptance of the ranking may suffer if
the results show little overall variance with many ties.  

3)      Process Outcomes:  Not discussed - move to 23 Feb agenda.

4)      Project Management Toolset Recommendation:  Not discussed - move to
23 Feb session.

5)   Independent ICANN Project Ratings?   Wolf-Ulrich wondered if
ICANN/Staff develop a separate prioritization list attendant to or
derivative from an operational or budget planning process.  If so, how
should it be dovetailed with this effort?  Ken will follow up Rob. 

 

Prepared by:  Ken Bour