RE: [gnso-wpm-dt] WPM: Summary of 16 Feb 2010 Session (Step 6-In Progress)

["Gomes, Chuck" <cgomes@xxxxxxxxxxxx>]

Thanks Ken.  Very helpful again.  Please see my comments below.
 
Chuck


________________________________

        From: owner-gnso-wpm-dt@xxxxxxxxx
[mailto:owner-gnso-wpm-dt@xxxxxxxxx] On Behalf Of Ken Bour
        Sent: Tuesday, February 16, 2010 5:50 PM
        To: gnso-wpm-dt@xxxxxxxxx
        Subject: [gnso-wpm-dt] WPM: Summary of 16 Feb 2010 Session (Step
6-In Progress)
        
        

        WPM Team Members:

         

        I apologize once again for a longish email summary, but I
attempted to capture the decisions made during our teleconference today,
16 Feb 2010 and, also, to flesh out some of the discussions that
remained unresolved as our time expired.  Ultimately, we are going to
need a record of our deliberations and I find that this writing process
helps me congeal my own thinking.  In preparing each of these reports,
short and long, I frequently research earlier summaries to recall what
was said/done and, in many instances, I end up consolidating material in
an effort to sustain forward progress.  Although this approach produces
more written material to digest; hopefully, you will agree that there is
"method in the madness."  

         

        Following my normal procedure, I will create a 2nd email to
include the Agenda Topics and Action Items for our next session
scheduled for 23 Feb 2010 (1700 UTC).  Since there may be an opportunity
to make progress via the email list on some/all of the questions
contained below, I will wait for several days before compiling the
separate email of agenda topics.  

         

        1)      Project Criteria Definitions:  Definitions for Value and
Difficulty approved as drafted in Ken's 14 Feb email.  

        2)      Step 6 Analysis/Questions:

        a)      Rank order ties will be permitted.  If Council needs to
decide (for any reason) between one or more tied projects, then it will
rate each project's "Difficulty" as defined by the WPM team employing a
process similar to the one used for rating Value.  [Gomes, Chuck]  Not
sure we will need a formal process for assessing difficulty but it is
probably okay. 

        b)      Frequency:  a formal prioritization rating session (all
relevant projects) will be conducted 3 times per year at ICANN meetings.
The team noted that a face-to-face working session may facilitate using
the Delphi group ratings approach.  Specific methodology
issues/questions are taken up in (f) below.  [KB Note:  Chuck and I had
an email exchange on the list regarding the "political morale" question,
which included the possibility of creating a new exemption category (or
two).  I had that email ready for discussion today, but we did not have
sufficient time.  May I suggest that others register your thoughts in
the intervening week, if you have an opportunity?  If we do not resolve
that question via the list, I will add it to the Action Items for next
time.] [Gomes, Chuck]  Not sure we finalized the idea of using in-person
meetings because of the time it might take.  We probably should flesh
that out further. It may involve some prework before the final in-person
work.

        c)      New Projects:  when a new project arises, the Chair will
assess whether it can be addressed effectively by the Council (see d
below) without requiring that an official Value rating be determined.
If a specific rating/ranking is deemed necessary to proceed with the
Council's management responsibilities, then the Chair will call for the
new project to be formally rated before further action is taken;
otherwise, that project will be officially prioritized at the next
scheduled session (e.g. next ICANN meeting). 

        d)     New Project Questions:  the following questions were
slightly modified per today's discussion and are repeated to give the
team another look at them.  

        o   Should this new project be undertaken, that is, have
resources assigned to accomplish a particular objective? 

        o   What resource types, skills, and quantities are needed to
adequately staff this project? 

        o   Are there sufficient resources (Staff and Community)
available without causing adverse impacts to other project work in
progress?  If not, should any other project work be stopped or
postponed? 

        o   Should this new project have a date/deadline imposed, thus
establishing urgency?  If it is determined to be urgent, can any real
consequences be identified that will cause the date to be perceived and
treated as critical? 

        e)      Project Status Changes:  as noted in b above, updates
will be officially recognized and captured 3 times per year as reflected
in the Value ratings assigned.  Status changes will also be considered
when new projects are evaluated (see c-d above).  If, at any other time,
conditions warrant a formal reevaluation, the Chair may make such a
recommendation to the Council.  [KB Question:  does this mean that the
Chair can call for another prioritization exercise in between scheduled
sessions?  If not, I am not sure how these "status changes" would be
captured or reflected.  The team has not created any recording process,
yet, for status changes other than the actual Value ratings.]  [Gomes,
Chuck]  I think it does mean that the "Chair can call for another
prioritization exercise in between scheduled sessions" but I am open to
other views.  Regardless, we should be clear.

        f)       Methodology:  several alternatives were briefly
discussed as today's call came to a conclusion.  I will attempt to
summarize 4 Options below including pros/cons as I perceive them.
Please feel free to add/change/delete any of the advantages or
disadvantages.  The options are not presented in any order of importance
or significance other than how they occurred to me as I began sketching
them.  

        Option 1:  [Gomes, Chuck]  At this point I like this one the
best by I am open to more input. 

        Each Council member rates all projects individually in advance
(e.g. spreadsheet template); then a single Council Delphi session is
held with all members participating.  Process would include looking for
mathematical consensus, e.g. Std Dev<1.0 or Range<=2, ahead of the group
session.  With 20 raters, I think it may be statistically unlikely, but
it could happen.  Staff facilitates an Adobe Connect session using the
polling feature and blind voting.  The polling process ends when all
individual scores occupy no more than 3 adjacent ratings or Range<=2
(e.g. 4, 5, 6 or 2, 3, 4); the median becomes the final rating.  

        Pros:  

        *         After one single iteration, all projects are rated
and, thereby, ranked (ties permitted).  

        *         Prioritization occurs through group deliberation with
all viewpoints expressed and heard by everyone.

        *         If successful, this process should result in a high
level of acceptance by the Council and the community.  

        Cons:  

        *         Time:  the WPM's experience with 5 participants was an
average of 7 minutes discussion/voting per project (in both Y and X
sessions).  Fifteen (15) projects would take a minimum of 105 minutes
not counting setup, explanations, process discussions, etc.  [Gomes,
Chuck]  This gives me a chance to plug my suggestion that we reduce the
# of projects by eliminating those that we decide to do regardless of
priority (e.g., IRTP-B, Geo). 

        [KB Questions/Comments: 

        1)      We haven't tested, so we must ask:  will 7 minutes per
project hold up with 20 potential participants?  Instead of 4-5
individuals expressing an opinion, what if 10-15 Councilors elect to
speak individually?  Should there be time limits imposed (e.g.
green/yellow/red)?  If the team believes that 7 minutes is a reasonable
benchmark, then a 2 hour session would be barely adequate - the first
time through the process.  Note that at 10 minutes average per project,
the effort would require closer to 3 hours (180 minutes) including 20
minutes to start and 10 to wrap-up.  [Gomes, Chuck]  Some time limits
will probably be needed but we should also allow a longer period of time
the first time through, even if we have to break it up into two
sessions. 

        2)      With 20 participants, how likely is it that a Range<=2
would occur on the 1st polling vs. requiring multiple iterations?
Should a limit of 3 polling iterations be instituted in the event that,
after the 3rd attempt, a Range<=2 has not been achieved?  I assume that
we would still take the median score at that point.  [Gomes, Chuck]
Sounds like a good idea to me to limit it to 3 interations. 

        3)      Irrespective of the number of participants, I also
wonder whether a face-to-face session might actually prolong the
discussions more so than participation remotely via telephone. [Gomes,
Chuck]  Could be. 

        4)      Restating Wolf-Ulrich's concept, would it accelerate the
Council's process if the WPM provided an initial rating as input?]
[Gomes, Chuck]  I have reservations about this but might be convinced
otherwise.  I personally think it would be good if each participant did
their initial rating without influence.  But, maybe we should provide an
opportunity for those less familiar with certain projects to ask
questions before starting. 

         

        Option 2: 

        Extending another of Wolf-Ulrich's ideas in a slightly different
direction, Ken offered an approach with these feature elements:  the
Council asks each SG to appoint one Council member to be a "delegate" to
a standing GNSO Project Prioritization Committee.  The non-voting NCA
would be a 5th delegate and represent all other non-SG (e.g. ALAC).
With Staff assistance (TBD), each delegate would be responsible for
developing a composite project rating by soliciting/aggregating input
from his/her SG.  Delegates would then attend a single rating session
(stay with 3x per year?) in which they would use the group Delphi
approach facilitated via Adobe Connect (same procedures/rules as Option
1).  One delegate would be elected "Committee Chair" by the team and
that position would conduct/manage rating sessions, direct Staff's
involvement, and report the results to the Council.  

        Pros:

        *         Soliciting and consolidating Constituency/SG input is
a common Councilor function which would be applied to this task.

        *         SG prioritizations are handled in the "background" and
the group session involves only 5 individuals - similar to the WPM's
testing experience.  The process would still take 2 hours, in all
likelihood, but would involve only 5 individuals vs. 20.  

        *         Delegates would gain proficiency and efficiency both
in collecting SG input and working with colleagues on group ratings.  

        *         After a single group session of the Committee, all
projects are rated and ranked. 

        Cons:

        *         Process might not enjoy the same acceptance level as
Option 1, especially if delegates are perceived to have yielded too much
ground in a quest to produce consensus.  

        *         Delegates might tend to fight for their ratings more
vigorously than would be the case if individual Councilors rated
independently and without win/lose pressure.  [Gomes, Chuck]  I think
this could be the biggest drawback to this approach. 

         

        Option 3:

        The Council is sub-divided into small groups:  RySG - 3; RrSG -
3; CSG - 6; NCSG - 6; NCA's - 3) and each one uses the WPM
procedures/rules outlined in Option 1 (above) to produce a consensus
rating for all 15 projects.  The median from the 5 individual group
results becomes the Council rating for each project.  

        Pros:

        *         Smaller groups of individuals from like-minded
communities may achieve consensus more easily and efficiently since
perceptions and goals tend to be congruent. 

        Cons:

        *         There may be very different rating outcomes between
the 5 groups, which will be "averaged away" in calculating the medians. 

        *         If the groups do produce very different rating
results, the acceptance level of the final prioritization may suffer. 

         

        [KB Note:  this option could also be reworked with small groups
being selected more or less randomly so that each one has a mix of
contracted and non-contracted parties.  If the team thinks I should
scope out that option, I would be pleased to do so.]  

         

        Option 4:

         

        Each Council member rates all 15 projects independently (e.g.
spreadsheet template; no group discussions) and the results are averaged
or the medians computed.  

         

        Pros:

        *         Most efficient option and requires least amount of
time.  

        Cons:

        *         Statistical centering is very likely and, therefore,
there may also be a much higher incidence of ties.  For a simple
illustration, two ratings of 1 and 7 produce a 4 both for mean and
median, but that result doesn't capture the reality of what really
happened in the prioritization process.  The team witnessed this same
phenomenon when its individual ratings (only 6 raters) were averaged
(mean and median).  As we observed and calculated, only 4 of the 15
projects achieved "natural" consensus (Range<=2) in the Value ratings
until after the group discussions took place.  There was considerable
clustering of the ratings; in fact, 11/15 of the Y means were between
the range of 3.7 and 5.7 and 10/15 of the Y medians were between 3.5 and
5.0.  

        *         As a result of averaging, acceptance of the ranking
may suffer if the results show little overall variance with many ties.  

        3)      Process Outcomes:  Not discussed - move to 23 Feb
agenda.

        4)      Project Management Toolset Recommendation:  Not
discussed - move to 23 Feb session.

        5)   Independent ICANN Project Ratings?   Wolf-Ulrich wondered
if ICANN/Staff develop a separate prioritization list attendant to or
derivative from an operational or budget planning process.  If so, how
should it be dovetailed with this effort?  Ken will follow up Rob. 

         

        Prepared by:  Ken Bour