[gnso-wpm-dt] WPM Preliminary Status: Step 2

["Ken Bour" <ken.bour@xxxxxxxxxxx>]

Jaime:

 

My primary concern has been the time element and finalizing ratings for 15
projects among 20+ participants in an average of 7 minutes each!   Any
project that could be eliminated from discussion by reason of statistical
commonality would get us that much closer to completion in the limited time
available; however, as you point out, those projects with low Std Dev should
converge pretty quickly, hopefully in just one polling cycle.  

 

Just to be clear, I wasn?t suggesting making any change for this upcoming
effort, only ruminating about a potential process improvement for subsequent
sessions.   I recommend that we evaluate the entire process, systematically,
once this first effort is finished.   There might be several improvements to
consider? 

 

As facilitator for the session in Brussels, I was thinking that I might
actually start the group discussion with the more centrally convergent
projects (low Std Dev) as a way to break the ice, start with a few ?easier?
cases, gain some immediate success, and build positive momentum.   Does that
sound like a productive approach?   If it works, we would generate quick
results and store away some precious minutes for the more challenging
projects.   

 

Ken

 

From: Jaime Wagner - CGI [mailto:jaime@xxxxxx] 
Sent: Wednesday, June 09, 2010 3:25 PM
To: 'Ken Bour'; 'Stéphane Van Gelder'
Cc: gnso-wpm-dt@xxxxxxxxx
Subject: RES: [gnso-wpm-dt] WPM Preliminary Status: Step 2

 

Ken,

 

I think we should keep the systematic we devised based on range and Delphi
session. When the standard deviation is low we will probably converge in
only one session, provided the extremes are not stubborn.

 

Jaime Wagner
CGI (Comitê Gestor da Internet no Brasil)
Representante dos Provedores de Acesso e Conteúdo
jaime <mailto:jaime@xxxxxxxxxxxxxxxxxx> @cgi.br <mailto:jaime@xxxxxx>
(51)8126-0916
j.w@xxxxxxxxxxx

 

De: owner-gnso-wpm-dt@xxxxxxxxx [mailto:owner-gnso-wpm-dt@xxxxxxxxx] Em nome
de Ken Bour
Enviada em: quarta-feira, 9 de junho de 2010 10:19
Para: 'Stéphane Van Gelder'
Cc: gnso-wpm-dt@xxxxxxxxx
Assunto: [gnso-wpm-dt] WPM Preliminary Status: Step 2

 

Stephane:

 

I think you might have misunderstood my comment.   Every project?s Range
statistic is > 2, which means that, when I subtract the highest rating from
the lowest rating, that difference is larger than 2 for every project.   On
the surface, that statistic indicates a wide spread among certain Councilors
as to their perception of the each project?s relative Value.   It also means
that there are no projects that can be automatically removed from the
Brussels discussion based upon the individual ratings step, which required a
Range of 2 or less.  

 

One of the possible improvement steps, going forward, may be to change the
way that central tendency is measured after the individual ratings round.
When we only had 5 testers, the Range looked like the best indicator; it is
easy to calculate and understand.   Now that I am seeing over a dozen
values, the Standard Deviation might be a more useful statistic for
determining agreement since the population size is so much larger.   For
example, out of 12 ratings for one particular project, we have the following
ratings:   5, 3, 4, 3, 6, 4, 4, 4, 5, 4, 4, 4.   Even though the Range is 3
(6-3), the Mode, the Median, and the Mean are all equal to 4 and the
Standard Deviation is less than 1.0 (actually .8), which indicates a very
tight spread not only statistically, but visually as well.   For our current
exercise, we will not be excluding this project from discussion; but, in the
future, it could be a candidate for pre-determining agreement (in this case:
Rating=4) on the basis of its low Standard Deviation.  

 

Here is one more example that is interesting.  The 12 ratings thus far are:
6, 5, 7, 6, 5, 5, 7, 7, 4, 7, 7, 7.   The Range=3, the Mode=7, Median=7,
Mean=6, and Std Dev=1.04.   I think a strong case could be made for
accepting 7 as the group Value Rating although, if the rule were written
such that Std Dev had to be < 1.00, it would fail the test.  

 

I expect to have a more fully thought-out recommendation once this part of
the process has completed and I have more time to analyze the results.  

 

Lastly, I just want to be clear there are tons of individual project ratings
that are ?1? and, for that matter, ?7?.   From what I can discern in
examining each Councilor?s spreadsheet, it doesn?t appear that anyone
misunderstood the directions.   Any value from 1-7 could be selected for any
cell and, while some Councilors used the entire range and others did not, I
don?t think there is anything to be concluded other than that is how they
perceived Value across the range of Eligible Projects.   

 

Ken

 

From: Stéphane Van Gelder [mailto:stephane.vangelder@xxxxxxxxx] 
Sent: Wednesday, June 09, 2010 6:30 AM
To: Ken Bour
Cc: gnso-wpm-dt@xxxxxxxxx
Subject: Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2

 

Thanks Ken for that update. I wonder if the reason we are not getting any
projects with a lower score than 2 is that Councillors are note sufficiently
aware that they can strike projects all together should they wish to when
they rate them?

 

Stéphane

Le 8 juin 2010 à 21:08, Ken Bour a écrit :

 

WPM-DT Members:

 

I thought you might appreciate receiving a brief status report concerning
Step 2-Individual Councilor Ratings?

 

As of this afternoon, 8 June, I have received 12 Councilor ratings
spreadsheets.   The deadline, as you may know, has been extended to 9 June
(tomorrow).   Happily, other than a few names/dates being left off (I am
saving the emails and renaming the attachments so that I can positively ID
each one), the data aggregation process is going as planned and tested.   No
one, thus far, has failed to provide a 1-7 rating for each of the Eligible
Projects.   

 

You may be interested, if not surprised, to learn that not a single project
can be excluded from discussion after the individual rating step.   Every
project?s Range is already > 2 and, of course, it cannot get any tighter as
more results are received.   Of the 15 Eligible Projects:   

·         11 or 73% have a Range >= 5

·         7 or 46% have a Range = 6 (max)

 

I have developed a consolidation spreadsheet, which is automatically
color-coded to reveal the top/bottom ratings and the most prevalent answer
(or Mode).   Fortunately, several projects have pretty stable
Mode/Median/Mean results meaning that, while we might have a couple of 7?s
and 1?s (thus Range=6), most participants rated the project similarly.   In
a few cases, the Mode, Median, and Mean are the identical value indicating
strong central tendency (so far)!   In those instances, at least
theoretically, it should be possible to influence the small number of
outliers to move closer to the group?s most common rating.   Even if that is
not possible, after discussion, it will be somewhat comforting to know that
there was reasonably strong agreement statistically. 

 

For Brussels, I estimate that we will have about 105 minutes net (if we can
hold preliminaries to 15), which leaves an average of 7 minutes per project
for discussion and polling.  

 

I am currently drafting a letter that I plan to send out early next week
(14th or 15th) addressing as many preliminaries as possible so that the
Brussels meeting (on Saturday morning) can be quickly focused on the group
ratings discussions.   This letter will cover such topics as:   Councilor
Preparation, Meeting Setup, Guiding Principles, and Process Flow (briefly).
I will be encouraging participants to arrive a few minutes early so that we
can speed up the routine process of settling in?

 

If WPM-DT members would like to preview the letter before it goes out,
please let me know.   Although I recognize that you are all very busy, I
would appreciate another set of eyes on this next communication... 

 

Regards,

 

Ken Bour