[gnso-wpm-dt] WPM Preliminary Status: Step 2

["Ken Bour" <ken.bour@xxxxxxxxxxx>]

Stéphane:

 

I will try?  

 

You wrote:  ?I wonder if the reason we are not getting any projects with a
lower score than 2 is that Councilors are not sufficiently aware that they
can strike projects all together should they wish to when they rate them??

 

Shorter response:  

1)      Councilors are rating projects lower than 2 in Value; lots of ?1s?
have been registered.  

2)      My original comment about ?>2? dealt with the Range statistic, not
the individual Councilor project ratings.  

 

I apologize for using your inquiry to springboard into a broader discussion
of central tendency measures, but I wanted to convey those thoughts to the
WPM-DT while they were fresh on my mind.   Sorry, if my explanation was
overly confusing? 

 

Ken 

 

From: Stéphane Van Gelder [mailto:stephane.vangelder@xxxxxxxxx] 
Sent: Wednesday, June 09, 2010 3:56 PM
To: Ken Bour
Cc: gnso-wpm-dt@xxxxxxxxx
Subject: Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2

 

Hi Ken,

 

Well if I was confused before, I am now completely flummoxed after trying to
read your latest email :)

 

If there's any way you can explain it to me in a couple of sentences, I
would be grateful. If not, please just ignore my previous comment.

 

Stéphane

 

Le 9 juin 2010 à 15:18, Ken Bour a écrit :





Stephane:

 

I think you might have misunderstood my comment.   Every project?s Range
statistic is > 2, which means that, when I subtract the highest rating from
the lowest rating, that difference is larger than 2 for every project.   On
the surface, that statistic indicates a wide spread among certain Councilors
as to their perception of the each project?s relative Value.   It also means
that there are no projects that can be automatically removed from the
Brussels discussion based upon the individual ratings step, which required a
Range of 2 or less. 

 

One of the possible improvement steps, going forward, may be to change the
way that central tendency is measured after the individual ratings round.
When we only had 5 testers, the Range looked like the best indicator; it is
easy to calculate and understand.   Now that I am seeing over a dozen
values, the Standard Deviation might be a more useful statistic for
determining agreement since the population size is so much larger.   For
example, out of 12 ratings for one particular project, we have the following
ratings:   5, 3, 4, 3, 6, 4, 4, 4, 5, 4, 4, 4.   Even though the Range is 3
(6-3), the Mode, the Median, and the Mean are all equal to 4 and the
Standard Deviation is less than 1.0 (actually .8), which indicates a very
tight spread not only statistically, but visually as well.   For our current
exercise, we will not be excluding this project from discussion; but, in the
future, it could be a candidate for pre-determining agreement (in this case:
Rating=4) on the basis of its low Standard Deviation. 

 

Here is one more example that is interesting.  The 12 ratings thus far are:
6, 5, 7, 6, 5, 5, 7, 7, 4, 7, 7, 7.   The Range=3, the Mode=7, Median=7,
Mean=6, and Std Dev=1.04.   I think a strong case could be made for
accepting 7 as the group Value Rating although, if the rule were written
such that Std Dev had to be < 1.00, it would fail the test. 

 

I expect to have a more fully thought-out recommendation once this part of
the process has completed and I have more time to analyze the results. 

 

Lastly, I just want to be clear there are tons of individual project ratings
that are ?1? and, for that matter, ?7?.   From what I can discern in
examining each Councilor?s spreadsheet, it doesn?t appear that anyone
misunderstood the directions.   Any value from 1-7 could be selected for any
cell and, while some Councilors used the entire range and others did not, I
don?t think there is anything to be concluded other than that is how they
perceived Value across the range of Eligible Projects.   

 

Ken

 

From: Stéphane Van Gelder [mailto:stephane.vangelder@xxxxxxxxx] 
Sent: Wednesday, June 09, 2010 6:30 AM
To: Ken Bour
Cc: gnso-wpm-dt@xxxxxxxxx
Subject: Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2

 

Thanks Ken for that update. I wonder if the reason we are not getting any
projects with a lower score than 2 is that Councillors are note sufficiently
aware that they can strike projects all together should they wish to when
they rate them?

 

Stéphane

Le 8 juin 2010 à 21:08, Ken Bour a écrit :






WPM-DT Members:

 

I thought you might appreciate receiving a brief status report concerning
Step 2-Individual Councilor Ratings?

 

As of this afternoon, 8 June, I have received 12 Councilor ratings
spreadsheets.   The deadline, as you may know, has been extended to 9 June
(tomorrow).   Happily, other than a few names/dates being left off (I am
saving the emails and renaming the attachments so that I can positively ID
each one), the data aggregation process is going as planned and tested.   No
one, thus far, has failed to provide a 1-7 rating for each of the Eligible
Projects.   

 

You may be interested, if not surprised, to learn that not a single project
can be excluded from discussion after the individual rating step.   Every
project?s Range is already > 2 and, of course, it cannot get any tighter as
more results are received.   Of the 15 Eligible Projects:   

·         11 or 73% have a Range >= 5

·         7 or 46% have a Range = 6 (max)

 

I have developed a consolidation spreadsheet, which is automatically
color-coded to reveal the top/bottom ratings and the most prevalent answer
(or Mode).   Fortunately, several projects have pretty stable
Mode/Median/Mean results meaning that, while we might have a couple of 7?s
and 1?s (thus Range=6), most participants rated the project similarly.   In
a few cases, the Mode, Median, and Mean are the identical value indicating
strong central tendency (so far)!   In those instances, at least
theoretically, it should be possible to influence the small number of
outliers to move closer to the group?s most common rating.   Even if that is
not possible, after discussion, it will be somewhat comforting to know that
there was reasonably strong agreement statistically. 

 

For Brussels, I estimate that we will have about 105 minutes net (if we can
hold preliminaries to 15), which leaves an average of 7 minutes per project
for discussion and polling.  

 

I am currently drafting a letter that I plan to send out early next week
(14th or 15th) addressing as many preliminaries as possible so that the
Brussels meeting (on Saturday morning) can be quickly focused on the group
ratings discussions.   This letter will cover such topics as:   Councilor
Preparation, Meeting Setup, Guiding Principles, and Process Flow (briefly).
I will be encouraging participants to arrive a few minutes early so that we
can speed up the routine process of settling in?

 

If WPM-DT members would like to preview the letter before it goes out,
please let me know.   Although I recognize that you are all very busy, I
would appreciate another set of eyes on this next communication... 

 

Regards,

 

Ken Bour