Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2

[Stéphane Van Gelder <stephane.vangelder@xxxxxxxxx>]

Hi Ken,

Well if I was confused before, I am now completely flummoxed after trying to 
read your latest email :)

If there's any way you can explain it to me in a couple of sentences, I would 
be grateful. If not, please just ignore my previous comment.

Stéphane

Le 9 juin 2010 à 15:18, Ken Bour a écrit :

> Stephane:
>  
> I think you might have misunderstood my comment.   Every project’s Range 
> statistic is > 2, which means that, when I subtract the highest rating from 
> the lowest rating, that difference is larger than 2 for every project.   On 
> the surface, that statistic indicates a wide spread among certain Councilors 
> as to their perception of the each project’s relative Value.   It also means 
> that there are no projects that can be automatically removed from the 
> Brussels discussion based upon the individual ratings step, which required a 
> Range of 2 or less. 
>  
> One of the possible improvement steps, going forward, may be to change the 
> way that central tendency is measured after the individual ratings round.   
> When we only had 5 testers, the Range looked like the best indicator; it is 
> easy to calculate and understand.   Now that I am seeing over a dozen values, 
> the Standard Deviation might be a more useful statistic for determining 
> agreement since the population size is so much larger.   For example, out of 
> 12 ratings for one particular project, we have the following ratings:   5, 3, 
> 4, 3, 6, 4, 4, 4, 5, 4, 4, 4.   Even though the Range is 3 (6-3), the Mode, 
> the Median, and the Mean are all equal to 4 and the Standard Deviation is 
> less than 1.0 (actually .8), which indicates a very tight spread not only 
> statistically, but visually as well.   For our current exercise, we will not 
> be excluding this project from discussion; but, in the future, it could be a 
> candidate for pre-determining agreement (in this case: Rating=4) on the basis 
> of its low Standard Deviation. 
>  
> Here is one more example that is interesting.  The 12 ratings thus far are:  
> 6, 5, 7, 6, 5, 5, 7, 7, 4, 7, 7, 7.   The Range=3, the Mode=7, Median=7, 
> Mean=6, and Std Dev=1.04.   I think a strong case could be made for accepting 
> 7 as the group Value Rating although, if the rule were written such that Std 
> Dev had to be < 1.00, it would fail the test. 
>  
> I expect to have a more fully thought-out recommendation once this part of 
> the process has completed and I have more time to analyze the results. 
>  
> Lastly, I just want to be clear there are tons of individual project ratings 
> that are “1” and, for that matter, “7”.   From what I can discern in 
> examining each Councilor’s spreadsheet, it doesn’t appear that anyone 
> misunderstood the directions.   Any value from 1-7 could be selected for any 
> cell and, while some Councilors used the entire range and others did not, I 
> don’t think there is anything to be concluded other than that is how they 
> perceived Value across the range of Eligible Projects.   
>  
> Ken
>  
> From: Stéphane Van Gelder [mailto:stephane.vangelder@xxxxxxxxx] 
> Sent: Wednesday, June 09, 2010 6:30 AM
> To: Ken Bour
> Cc: gnso-wpm-dt@xxxxxxxxx
> Subject: Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2
>  
> Thanks Ken for that update. I wonder if the reason we are not getting any 
> projects with a lower score than 2 is that Councillors are note sufficiently 
> aware that they can strike projects all together should they wish to when 
> they rate them?
>  
> Stéphane
> 
> Le 8 juin 2010 à 21:08, Ken Bour a écrit :
> 
> 
> WPM-DT Members:
>  
> I thought you might appreciate receiving a brief status report concerning 
> Step 2-Individual Councilor Ratings…
>  
> As of this afternoon, 8 June, I have received 12 Councilor ratings 
> spreadsheets.   The deadline, as you may know, has been extended to 9 June 
> (tomorrow).   Happily, other than a few names/dates being left off (I am 
> saving the emails and renaming the attachments so that I can positively ID 
> each one), the data aggregation process is going as planned and tested.   No 
> one, thus far, has failed to provide a 1-7 rating for each of the Eligible 
> Projects.   
>  
> You may be interested, if not surprised, to learn that not a single project 
> can be excluded from discussion after the individual rating step.   Every 
> project’s Range is already > 2 and, of course, it cannot get any tighter as 
> more results are received.   Of the 15 Eligible Projects:   
> ·         11 or 73% have a Range >= 5
> ·         7 or 46% have a Range = 6 (max)
>  
> I have developed a consolidation spreadsheet, which is automatically 
> color-coded to reveal the top/bottom ratings and the most prevalent answer 
> (or Mode).   Fortunately, several projects have pretty stable 
> Mode/Median/Mean results meaning that, while we might have a couple of 7’s 
> and 1’s (thus Range=6), most participants rated the project similarly.   In a 
> few cases, the Mode, Median, and Mean are the identical value indicating 
> strong central tendency (so far)!   In those instances, at least 
> theoretically, it should be possible to influence the small number of 
> outliers to move closer to the group’s most common rating.   Even if that is 
> not possible, after discussion, it will be somewhat comforting to know that 
> there was reasonably strong agreement statistically. 
>  
> For Brussels, I estimate that we will have about 105 minutes net (if we can 
> hold preliminaries to 15), which leaves an average of 7 minutes per project 
> for discussion and polling.  
>  
> I am currently drafting a letter that I plan to send out early next week 
> (14th or 15th) addressing as many preliminaries as possible so that the 
> Brussels meeting (on Saturday morning) can be quickly focused on the group 
> ratings discussions.   This letter will cover such topics as:   Councilor 
> Preparation, Meeting Setup, Guiding Principles, and Process Flow (briefly).   
> I will be encouraging participants to arrive a few minutes early so that we 
> can speed up the routine process of settling in…
>  
> If WPM-DT members would like to preview the letter before it goes out, please 
> let me know.   Although I recognize that you are all very busy, I would 
> appreciate another set of eyes on this next communication... 
>  
> Regards,
>  
> Ken Bour
>  
>