RE: [gnso-wpm-dt] WPM Preliminary Status: Step 2

["Gomes, Chuck" <cgomes@xxxxxxxxxxxx>]

Good questions Stéphane, but please note that there will not be time in today's 
agenda to discuss the prioritization project.  We will just have a Status 
update.  That discussion will start happening in Brussels.  Of course, we can 
discuss this on the list.

 

Chuck

 

From: owner-gnso-wpm-dt@xxxxxxxxx [mailto:owner-gnso-wpm-dt@xxxxxxxxx] On 
Behalf Of Stéphane Van Gelder
Sent: Thursday, June 10, 2010 4:48 AM
To: Ken Bour
Cc: gnso-wpm-dt@xxxxxxxxx; council@xxxxxxxxxxxxxx Council
Subject: Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2

 

Hi Ken,

 

Thanks for providing such a clear and concise response.

 

I would venture to suggest that the fundamental question we should now be 
asking ourselves is whether the system, as currently proposed, would allow the 
GNSO Council to decide that projects should be stricken from the to-do list 
because they are rated too low in priority?

 

A secondary question may be, should the GNSO Council determine that there is a 
finite number of projects that can be undertaken at the same time, considering 
the available resources (# of Councillors, # of support Staff, etc.)?

 

I am copying the Council because this may be a point we want to discuss during 
our teleconference tonight.

 

I guess what I am saying is that whilst prioritization is a good first step, to 
be truly efficient, the GNSO may want to decide that it can only work on 15 
projects in one given time period (e.g. a year) and that more projects can only 
be added when one of the 15 is finished (the "15" is arbitrary here, I only use 
it to illustrate my idea, I am not suggesting that should be the number)? I 
freely admit to being inspired by Staff's batching idea for new gTLDs in this 
regard ;)

 

Stéphane

Le 9 juin 2010 à 22:26, Ken Bour a écrit :





Stéphane:

 

I will try... 

 

You wrote:  "I wonder if the reason we are not getting any projects with a 
lower score than 2 is that Councilors are not sufficiently aware that they can 
strike projects all together should they wish to when they rate them?"

 

Shorter response: 

1)      Councilors are rating projects lower than 2 in Value; lots of "1s" have 
been registered. 

2)      My original comment about ">2" dealt with the Range statistic, not the 
individual Councilor project ratings. 

 

I apologize for using your inquiry to springboard into a broader discussion of 
central tendency measures, but I wanted to convey those thoughts to the WPM-DT 
while they were fresh on my mind.   Sorry, if my explanation was overly 
confusing...

 

Ken

 

From: Stéphane Van Gelder [mailto:stephane.vangelder@xxxxxxxxx] 
Sent: Wednesday, June 09, 2010 3:56 PM
To: Ken Bour
Cc: gnso-wpm-dt@xxxxxxxxx
Subject: Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2

 

Hi Ken,

 

Well if I was confused before, I am now completely flummoxed after trying to 
read your latest email :)

 

If there's any way you can explain it to me in a couple of sentences, I would 
be grateful. If not, please just ignore my previous comment.

 

Stéphane

 

Le 9 juin 2010 à 15:18, Ken Bour a écrit :






Stephane:

 

I think you might have misunderstood my comment.   Every project's Range 
statistic is > 2, which means that, when I subtract the highest rating from the 
lowest rating, that difference is larger than 2 for every project.   On the 
surface, that statistic indicates a wide spread among certain Councilors as to 
their perception of the each project's relative Value.   It also means that 
there are no projects that can be automatically removed from the Brussels 
discussion based upon the individual ratings step, which required a Range of 2 
or less. 

 

One of the possible improvement steps, going forward, may be to change the way 
that central tendency is measured after the individual ratings round.   When we 
only had 5 testers, the Range looked like the best indicator; it is easy to 
calculate and understand.   Now that I am seeing over a dozen values, the 
Standard Deviation might be a more useful statistic for determining agreement 
since the population size is so much larger.   For example, out of 12 ratings 
for one particular project, we have the following ratings:   5, 3, 4, 3, 6, 4, 
4, 4, 5, 4, 4, 4.   Even though the Range is 3 (6-3), the Mode, the Median, and 
the Mean are all equal to 4 and the Standard Deviation is less than 1.0 
(actually .8), which indicates a very tight spread not only statistically, but 
visually as well.   For our current exercise, we will not be excluding this 
project from discussion; but, in the future, it could be a candidate for 
pre-determining agreement (in this case: Rating=4) on the basis of its low 
Standard Deviation. 

 

Here is one more example that is interesting.  The 12 ratings thus far are:  6, 
5, 7, 6, 5, 5, 7, 7, 4, 7, 7, 7.   The Range=3, the Mode=7, Median=7, Mean=6, 
and Std Dev=1.04.   I think a strong case could be made for accepting 7 as the 
group Value Rating although, if the rule were written such that Std Dev had to 
be < 1.00, it would fail the test. 

 

I expect to have a more fully thought-out recommendation once this part of the 
process has completed and I have more time to analyze the results. 

 

Lastly, I just want to be clear there are tons of individual project ratings 
that are "1" and, for that matter, "7".   From what I can discern in examining 
each Councilor's spreadsheet, it doesn't appear that anyone misunderstood the 
directions.   Any value from 1-7 could be selected for any cell and, while some 
Councilors used the entire range and others did not, I don't think there is 
anything to be concluded other than that is how they perceived Value across the 
range of Eligible Projects.   

 

Ken

 

From: Stéphane Van Gelder [mailto:stephane.vangelder@xxxxxxxxx] 
Sent: Wednesday, June 09, 2010 6:30 AM
To: Ken Bour
Cc: gnso-wpm-dt@xxxxxxxxx
Subject: Re: [gnso-wpm-dt] WPM Preliminary Status: Step 2

 

Thanks Ken for that update. I wonder if the reason we are not getting any 
projects with a lower score than 2 is that Councillors are note sufficiently 
aware that they can strike projects all together should they wish to when they 
rate them?

 

Stéphane

Le 8 juin 2010 à 21:08, Ken Bour a écrit :







WPM-DT Members:

 

I thought you might appreciate receiving a brief status report concerning Step 
2-Individual Councilor Ratings...

 

As of this afternoon, 8 June, I have received 12 Councilor ratings 
spreadsheets.   The deadline, as you may know, has been extended to 9 June 
(tomorrow).   Happily, other than a few names/dates being left off (I am saving 
the emails and renaming the attachments so that I can positively ID each one), 
the data aggregation process is going as planned and tested.   No one, thus 
far, has failed to provide a 1-7 rating for each of the Eligible Projects.   

 

You may be interested, if not surprised, to learn that not a single project can 
be excluded from discussion after the individual rating step.   Every project's 
Range is already > 2 and, of course, it cannot get any tighter as more results 
are received.   Of the 15 Eligible Projects:   

·         11 or 73% have a Range >= 5

·         7 or 46% have a Range = 6 (max)

 

I have developed a consolidation spreadsheet, which is automatically 
color-coded to reveal the top/bottom ratings and the most prevalent answer (or 
Mode).   Fortunately, several projects have pretty stable Mode/Median/Mean 
results meaning that, while we might have a couple of 7's and 1's (thus 
Range=6), most participants rated the project similarly.   In a few cases, the 
Mode, Median, and Mean are the identical value indicating strong central 
tendency (so far)!   In those instances, at least theoretically, it should be 
possible to influence the small number of outliers to move closer to the 
group's most common rating.   Even if that is not possible, after discussion, 
it will be somewhat comforting to know that there was reasonably strong 
agreement statistically. 

 

For Brussels, I estimate that we will have about 105 minutes net (if we can 
hold preliminaries to 15), which leaves an average of 7 minutes per project for 
discussion and polling.  

 

I am currently drafting a letter that I plan to send out early next week (14th 
or 15th) addressing as many preliminaries as possible so that the Brussels 
meeting (on Saturday morning) can be quickly focused on the group ratings 
discussions.   This letter will cover such topics as:   Councilor Preparation, 
Meeting Setup, Guiding Principles, and Process Flow (briefly).   I will be 
encouraging participants to arrive a few minutes early so that we can speed up 
the routine process of settling in...

 

If WPM-DT members would like to preview the letter before it goes out, please 
let me know.   Although I recognize that you are all very busy, I would 
appreciate another set of eyes on this next communication... 

 

Regards,

 

Ken Bour