RE: [gnso-wpm-dt] WPM-DT: Step 5 (In Progress)

["Gomes, Chuck" <cgomes@xxxxxxxxxxxx>]

Thanks Ken.  To get discussion started I inserted a few comments below.
 
Chuck


________________________________

        From: owner-gnso-wpm-dt@xxxxxxxxx
[mailto:owner-gnso-wpm-dt@xxxxxxxxx] On Behalf Of Ken Bour
        Sent: Monday, December 28, 2009 3:44 PM
        To: gnso-wpm-dt@xxxxxxxxx
        Subject: [gnso-wpm-dt] WPM-DT: Step 5 (In Progress)
        
        

        WPM-DT Members:

         

        As promised, this second memo today will outline a few issues
that were brought up in today's discussion as well as discussing some of
what we might consider in our next session.    

         

        Outstanding Issues:

         

        1)      Value/Benefit (Y Axis) Definition-Issue #1:   this is a
holdover from the 21 Dec session during which a question was raised as
to whether/how to modify the definition to ensure that the narrower GNSO
is represented vs. only the global Internet community and ICANN
stakeholders.  [Gomes, Chuck]  I still support including the GNSO focus
in addition to the globabl community focus. 

        2)      Resource Consumption (X Axis) Definition-Issue #1:  we
noted that there were variances in interpretation by participants as to
whether a project's Resource Consumption should be exhaustive, i.e. from
the beginning of its instantiation (sunk cost) through anticipated
closure or limited to the current period forward.  [Gomes, Chuck]  As
stated in our meeting today, I think that measuring resource consumption
from the present point in time makes the most sense because we will be
prioritizing our work going forward.    

        3)      Resource Consumption (X Axis) Definition-Issue #2:
another question surfaced as to whether the Resource Consumption
estimate should include only that portion of the project that relates to
developing a GNSO recommendation (e.g. Final Report) or also include an
assessment of the project's full implementation.   To bring everyone up
to speed, the example that was used was the Communications and
Coordination Work Team (CCT) and its GNSO website recommendations.
Given the Work Team's near completion of its final Technology report,
one view is that there is little work left to do; thus, a relatively low
X axis rating would result.   On the other hand, if the project is to
also consider the entire website development effort (whether performed
by Staff or outside consultants), that would produce a much higher
assessment of Resource Consumption.   The DT agreed that additional
discussion is needed to provide the Council clear guidance in terms of
where to draw the lines so that the ratings estimates are consistently
interpreted under a common rubric. [Gomes, Chuck]  We definitely need to
provide clear direction in this regard. I favor treating actual
implementation separately from the rating of resources needed and
therefore not including implementation resources in the estimate of X
values.  

        4)      Liz and Ken would like to introduce an additional
question/concern regarding the assessments of X and Y.   Staff
calculated in previous iterations that the correlation statistics
between X and Y ratings, in many instances, were high (> 50%) - a result
that continued with the DELPHI technique.   While a high correlation
between these variables is not necessarily indicative of a problem (i.e.
highly valued projects might indeed have high resource consumptions),
they should be considered independent variables.  Ken notes that, during
today's discussion of the X axis, comments were made a few times about
the "priority" of a project even when the team was explicitly evaluating
its resource consumption.   For some individuals, it may be challenging
to cleanly separate the issues between value and consumption.   One
potential remedy to this natural cross-correlation bias, would be to ask
DIFFERENT parties to rate X and Y independently.   For example, one
possibility might be to ask the Council to rate Value/Benefit and have
Staff rate Resource Consumption - especially since Staff has deeper
knowledge of and more extensive involvement with the various project
teams and also understands the amount/extent of Staff resources that are
being applied. [Gomes, Chuck]  This is an interesting idea worthy of
further discussion.  We could also consider having Staff and member(s)
of the applicable work team rate the resources needed.  We might want to
use the term "Resources Needed" instead of "Resource Consumption" to
avoid the implication that it includes resources already consumed,
assuming we agree on that approach. 

         

        Next Session:

         

        1)      Next Meeting:  a Doodle poll will be forthcoming to find
a time that the DT can meet on either 5 or 6 January 2010.   

        2)      After today's work [Note: for technical results, see
companion email; Subject: Step 3a (Completed)], the team agreed to
proceed to Step 5, "Team assessment of the construct and
process/methodology and recommendations."   However, during the next
meeting, in addition to the process evaluation topics (see below), the
team also wishes to consider whether to create a Step 3b in which it
would try another rating permutation (e.g. small groups of 2-3).   If
the team elects to perform a Step 3b, we could repeat Steps 4 and 5 for
that iteration before moving to Step 6 "Focus on HOW the team might
utilize the data in terms of developing a prioritization -- the ultimate
goal of this effort."  

        3)      We should also formalize the earlier list discussion
which would add a Step 7: "Red Team Analysis of Model, Methodology, and
Procedures"  

        4)      The above step also suggests another one or Step 8:
"Finalize WPM-DT Model and Recommendations to GNSO Council"   

         

        Considerations on Small Group DELPHI:

         

        Following are some thoughts that the team might begin to weigh
prior to next week's session: 

         

        *         After having rated all 15 projects both individually
and in one large group (of 5), what new information might be learned by
breaking into smaller groups of 2-3? [Gomes, Chuck]  I am started to
wonder more about this.  There was a lot of value in our group
discussion for each of the rating and the most helpful part of that came
from participants who were more knowledgeable of the applicable
projects. In a group of 2 or 3, the chances of having someone who is
knowledgeable on every project will be significantly reduced.   

        *         What, if any, criteria, should be applied in
constructing small groups?   For example:  (a) experience/knowledge
(pairing senior Councilor with new member); (b) heterogeneous or
homogeneous by Constituency and/or SG; (c) contracted party vs.
non-contracted; ... others?  [Gomes, Chuck]  In my opinion, having
members with experience/knowledge is critical for small groups; the
smaller the groups, the more difficult it will be to have
experience/knowledge across all projects.

        *         If small groups are constructed, should they be
facilitated by Staff using an Adobe Connect room or should individuals
get together independently to discuss and agree upon ratings to be
submitted (e.g. using form similar to the one provided earlier for
individual ratings)? [Gomes, Chuck]  The Adobe Connect tool worked very
well for us so far.  Would it work as well in smaller groups?  We should
at least test it if we decide to go that direction. 

         

        Questions for DT Interim Evaluation:

         

        The following approach is a suggestion for how we might proceed
in conducting next meeting's assessment discussion:[Gomes, Chuck]  Looks
good to me. 

         

        1)      Briefly recap the work completed thus far:

        o   Project list (15) and requirements for inclusion as "active"

        o   Two-dimensional model and definitions for X and Y

        o   Ratings scale 1-7 from Far Below Average to Far Above
Average

        o   Individual ratings using rating scale with final chart
plotting means or medians

        o   Large group DELPHI session using rating scale; facilitated
using Adobe Connect room with polling feature

        2)      Any changes recommended to work already completed, i.e.,
what worked, what didn't work? 

        3)      How is the DT doing against its objectives thus far?  

         

        Final Thought:   

         

        I have recently begun experimenting with Mind Mapping software
(using NovaMind4) and, mostly for my own purposes, I created a map of
this team's process so that I could keep track not only of the steps (I
did change sequencing a bit), but the various issues, questions,
documents, etc., that we have been accumulating via the list.   It is
still a work-in-progress, but if you would like to see a copy of the
map, please click on this link:
http://www.novamind.com/connect/nm_documents/723.   It should not
require any login or other requirements to view.   I also attached a
static JPEG image in case you have any trouble with the link.  

         

        Ken Bour