[gnso-wpm-dt] WPM-DT: Step 3a (Rating Test #1 - In Progress)

["Ken Bour" <ken.bour@xxxxxxxxxxx>]

Stéphane:

 

Thank you for these test ratings.  I am glad that the mechanics worked
properly.

 

Your understanding of the scaling as applied to both dimensions is correct.
We can make that point clearer to future raters and appreciate the feedback.
The procedures, rating template, and instructions are all part of the
testing process and fair game for comment and improvement.

 

Regarding the X axis, a question was asked during the early planning stages
whether any reliable hard data could be made available, but we concluded
that it would be incomplete at best and, possibly, misleading.   Only
attendance records are kept for work teams, which represents a partial cost,
but would not be indicative of the total effort expended.  Also, for this
prioritization exercise, we have been trying to steer away from having hard
dollar cost be the primary driver for X, preferring, instead, more of a
?soft? cost.  As you will recall, the team had removed the word ?cost? from
the definition completely.  The term ?budget? was introduced based upon your
recent input.  Thinking about raters to come, since hard cost/budget data
does not exist for these projects, should we consider returning to a
definition that is even less quantitative?

 

I ran a statistical correlation on your X, Y values and it came out to 45%
compared to < 1% when they were selected by a random number generator.  It
implies that, for any project, if you picked a high (or low) rating for
Value/Benefit, you were almost 50% as likely to pick a correspondingly high
(or low) value for Resource Consumption.  Like you, I?m not entirely sure
what the connection may be, but it is certain worth continuing deliberation.
I will also run correlations on all other test ratings, individually, to see
if a pattern emerges.

 

One thought is that there probably is some natural correlation between value
and cost, that is, the more expensive things are and the harder we have to
work for them, the higher they are prized.  This relationship may become
more pronounced once projects and activities have started vs. before they
actually begin.  Expressions like, ? Don?t throw good money after bad?
derive from the human proclivity to salvage sunk costs and confuse
effort/activity with outcomes.  In our case, we are prioritizing ACTIVE
projects several of which are more than half-completed.  We thus have more
information about them than we might have at their instantiation and initial
prioritization.  Could any of the above be factors at work?

 

Ken

 

From: Stéphane Van Gelder [mailto:stephane.vangelder@xxxxxxxxx] 
Sent: Monday, December 14, 2009 9:40 AM
To: Ken Bour
Cc: gnso-wpm-dt@xxxxxxxxx
Subject: Re: [gnso-wpm-dt] WPM-DT: Step 3a (Rating Test #1 - In Progress)

 

Thanks Ken,

 

Please find attached my contribution.

 

A few comments:

 

- I did not deem it necessary or even desirable to take the time to go back
to the RrSG in order to get feedback on rating. I consider this a test and
the points awarded only reflect my own personal judgment or experience.

- For the X axis, I consider the higher the number of points, the less
desirable the project (as it is consuming more resources). This is reversed
for the Y axis (the more points awarded, the more a project is worthy of the
GNSO's attention). If this is consistent with everyone else's understanding,
it may be worthwhile making this very clear once the definitive rating
instructions are sent to the Council.

- I found the X axis very difficult to rate. It is impossible for me to have
a clear idea of the amount of budgetary resources a project requires without
having some kind of figure in front of me from staff. Would it be worthwhile
thinking about putting such a figure next to each project description listed
in the word document that came with Ken's email?

- I was surprised when rating to find that projects that tended to be of
lower value (according to me) also tended to require less resources (less
man hours spent on them, less expensive). I think there's something in that,
still trying to work out what it is ;)

 

Thanks,

 

Stéphane