Re: [gnso-wpm-dt] WPM-DT: Step 5 (In Progress)
Hi all, Sorry I missed the last call, and thanks Ken for making sure we have such detailed accounts of what's going on. That is especially useful when you miss a call ;) Reading through the group's latest work, I am getting increasingly worried about the complexity of the models being proposed. Keeping in mind that what we are trying to do is develop a model that can they be used by the Council as a whole, do you really think that the complex techniques and mathematical models that we are now working with in this group are usable by people who, by definition, will only have limited time to devote to rating projects and whom will have had none of the benefits of having seen the models evolve (and therefore of understanding the theory behind them) like this group's members do ? As we move towards concrete implementation of a proposed solution that we can then offer the Council, I think it's important to stay mindful of how complex the stuff being done here has become. And of the need to keep our final "product" as usable as possible. Thanks, Stéphane Le 28 déc. 2009 à 21:44, Ken Bour a écrit : > WPM-DT Members: > > As promised, this second memo today will outline a few issues that were > brought up in today’s discussion as well as discussing some of what we might > consider in our next session. > > Outstanding Issues: > > 1) Value/Benefit (Y Axis) Definition-Issue #1: this is a holdover from > the 21 Dec session during which a question was raised as to whether/how to > modify the definition to ensure that the narrower GNSO is represented vs. > only the global Internet community and ICANN stakeholders. > 2) Resource Consumption (X Axis) Definition-Issue #1: we noted that > there were variances in interpretation by participants as to whether a > project’s Resource Consumption should be exhaustive, i.e. from the beginning > of its instantiation (sunk cost) through anticipated closure or limited to > the current period forward. > 3) Resource Consumption (X Axis) Definition-Issue #2: another question > surfaced as to whether the Resource Consumption estimate should include only > that portion of the project that relates to developing a GNSO recommendation > (e.g. Final Report) or also include an assessment of the project’s full > implementation. To bring everyone up to speed, the example that was used > was the Communications and Coordination Work Team (CCT) and its GNSO website > recommendations. Given the Work Team’s near completion of its final > Technology report, one view is that there is little work left to do; thus, a > relatively low X axis rating would result. On the other hand, if the > project is to also consider the entire website development effort (whether > performed by Staff or outside consultants), that would produce a much higher > assessment of Resource Consumption. The DT agreed that additional > discussion is needed to provide the Council clear guidance in terms of where > to draw the lines so that the ratings estimates are consistently interpreted > under a common rubric. > 4) Liz and Ken would like to introduce an additional question/concern > regarding the assessments of X and Y. Staff calculated in previous > iterations that the correlation statistics between X and Y ratings, in many > instances, were high (> 50%) – a result that continued with the DELPHI > technique. While a high correlation between these variables is not > necessarily indicative of a problem (i.e. highly valued projects might indeed > have high resource consumptions), they should be considered independent > variables. Ken notes that, during today’s discussion of the X axis, comments > were made a few times about the “priority” of a project even when the team > was explicitly evaluating its resource consumption. For some individuals, > it may be challenging to cleanly separate the issues between value and > consumption. One potential remedy to this natural cross-correlation bias, > would be to ask DIFFERENT parties to rate X and Y independently. For > example, one possibility might be to ask the Council to rate Value/Benefit > and have Staff rate Resource Consumption – especially since Staff has deeper > knowledge of and more extensive involvement with the various project teams > and also understands the amount/extent of Staff resources that are being > applied. > > Next Session: > > 1) Next Meeting: a Doodle poll will be forthcoming to find a time that > the DT can meet on either 5 or 6 January 2010. > 2) After today’s work [Note: for technical results, see companion email; > Subject: Step 3a (Completed)], the team agreed to proceed to Step 5, “Team > assessment of the construct and process/methodology and recommendations.” > However, during the next meeting, in addition to the process evaluation > topics (see below), the team also wishes to consider whether to create a Step > 3b in which it would try another rating permutation (e.g. small groups of > 2-3). If the team elects to perform a Step 3b, we could repeat Steps 4 and > 5 for that iteration before moving to Step 6 “Focus on HOW the team might > utilize the data in terms of developing a prioritization -- the ultimate goal > of this effort.” > 3) We should also formalize the earlier list discussion which would add > a Step 7: “Red Team Analysis of Model, Methodology, and Procedures” > 4) The above step also suggests another one or Step 8: “Finalize WPM-DT > Model and Recommendations to GNSO Council” > > Considerations on Small Group DELPHI: > > Following are some thoughts that the team might begin to weigh prior to next > week’s session: > > · After having rated all 15 projects both individually and in one > large group (of 5), what new information might be learned by breaking into > smaller groups of 2-3? > · What, if any, criteria, should be applied in constructing small > groups? For example: (a) experience/knowledge (pairing senior Councilor > with new member); (b) heterogeneous or homogeneous by Constituency and/or SG; > (c) contracted party vs. non-contracted; … others? > · If small groups are constructed, should they be facilitated by > Staff using an Adobe Connect room or should individuals get together > independently to discuss and agree upon ratings to be submitted (e.g. using > form similar to the one provided earlier for individual ratings)? > > Questions for DT Interim Evaluation: > > The following approach is a suggestion for how we might proceed in conducting > next meeting’s assessment discussion: > > 1) Briefly recap the work completed thus far: > o Project list (15) and requirements for inclusion as “active” > o Two-dimensional model and definitions for X and Y > o Ratings scale 1-7 from Far Below Average to Far Above Average > o Individual ratings using rating scale with final chart plotting means or > medians > o Large group DELPHI session using rating scale; facilitated using Adobe > Connect room with polling feature > 2) Any changes recommended to work already completed, i.e., what worked, > what didn’t work? > 3) How is the DT doing against its objectives thus far? > > Final Thought: > > I have recently begun experimenting with Mind Mapping software (using > NovaMind4) and, mostly for my own purposes, I created a map of this team’s > process so that I could keep track not only of the steps (I did change > sequencing a bit), but the various issues, questions, documents, etc., that > we have been accumulating via the list. It is still a work-in-progress, but > if you would like to see a copy of the map, please click on this link: > http://www.novamind.com/connect/nm_documents/723. It should not require any > login or other requirements to view. I also attached a static JPEG image in > case you have any trouble with the link. > > Ken Bour > > <GNSO Work Priortization.jpeg> Attachment:
smime.p7s
|