[gnso-wpm-dt] WPM-DT: Step 3a (In Progress) -- Analysis of 6 Test Ratings and Prep for DELPHI Session Tomorrow

["Ken Bour" <ken.bour@xxxxxxxxxxx>]

WPM-DT Members:

 

Attached is an Excel spreadsheet containing the Test Results and analysis
based upon the 5 DT members and 1 Staff person who provided rating files as
of today.   Incidentally, the sixth rater is Liz Gasster.   Following
Jaime's suggestion, I asked her if she would like to toss in another set of
ratings and she agreed.   Marika also provided a set of ratings; however,
they were not included in this tabulation since she cannot participate in
our call tomorrow.  

 

I have developed a mathematical process for identifying commonality in the
results although, for tomorrow's session, I will not be able to include any
additional ratings other than what has been aggregated as of this evening.
The hard part was to noodle through the options to see if I could come up
with anything that would work - it fell into place rather nicely using two
common statistics:  Range and Standard Deviation (StdDev).   

 

There are a few things I'd like to report thus far and all are contained in
the attached spreadsheet.   Incidentally, the spreadsheet has two tabs:
most of the detail and analysis are in the Ratings tab and the Graphs/Charts
(#4 and #5 below) are in the Summary tab.  

 

1)      FYI - the 6 correlation statistics between X and Y for our test
ratings are:  

Chuck             11%

Jaime              25%

Stephane        45%

Wolf               51%

Olga               52%

Liz                  87%

 

The above stat may be something we'll end up discussing tomorrow during the
DELPHI discussions.   Once again, the higher the correlation statistic, the
more likely it was that a high rating for X was matched by a high rating for
Y.   In other words, for 3-4 of you, the more valuable a project was
perceived to be, the higher was its perceived resource consumption.
Alternatively, the more a project was perceived to consume lots of
energy/resource/time, the more value you assigned to it.   

 

2)      For the X axis, there were only 4 projects whose ratings had a tight
Range (Diff between High and Low = 1 or 2) and a StdDev < 1.0.   For those
elements, I am recommending that we accept the MEDIAN value (shown below) as
our DELPHI answer.    They are highlighted in GREEN (Range=1) and ORANGE
(Range=2).   All the others had too much deviation/variance and should be
discussed by the entire group.  

 

        
X VALUES

                        

SEQ NO

SVG

WUK

CG

JW

OC

LG

        DELPHI


1

7

5

1

6

2

7

                

2

3

4

3

3

4

3

        3.0


3

3

2

1

4

1

2

                

4

4

2

2

4

1

1

                

5

5

4

3

5

4

5

        4.5


6

5

5

5

5

4

5

        5.0


7

6

4

2

5

3

3

                

8

5

5

5

6

5

7

        5.0


9

5

2

3

5

4

5

                

10

5

2

3

5

3

5

                

11

5

3

4

5

5

7

                

12

5

3

2

6

3

4

                

13

5

4

2

4

3

5

                

14

6

5

3

6

4

7

                

15

2

3

3

3

3

6

                

 

3)      For the Y axis, there are 5 projects whose ratings produced a StdDev
< 1.0 and they are highlighted as described above.

 


SEQ NO

SVG

WUK

CG

JW

OC

LG

        DELPHI


1

7

6

6

6

5

6

        6.0


2

4

6

3

6

3

2

                

3

2

5

1

4

1

1

                

4

5

2

1

4

3

1

                

5

5

4

4

4

3

6

                

6

5

3

1

7

2

6

                

7

4

6

5

7

4

3

                

8

6

7

7

6

6

6

        6.0


9

6

4

7

6

6

5

                

10

6

4

5

5

4

5

        5.0


11

6

4

4

5

5

5

        5.0


12

6

3

5

6

4

5

                

13

4

3

4

3

3

5

        3.5


14

4

6

5

7

5

7

                

15

5

4

5

7

4

4

                

 

 

4)      Using RAW Means (Averages), the following would be the Work
Prioritization Chart given these six raters.   Notice that there are no
projects in Q4 (high cost, low value).   The averaging process, as
predicted, really does cause the numbers to cluster although, in this case,
not so much at the Average (4, 4), but in the range 3-5 on X and 4-6 on Y.
Nine or 60% of the projects are located in that tight area.  Incidentally, I
used GREEN letters above Y=4 and ORANGE below Y=4.   Ties went to GREEN.  

 



 

5)      If I take the same ratings and use MEDIAN values (middle result)
instead of MEANS, we get the following picture.  You can see that the data
spread is wider and less bunched compared to the above chart.    

 



 

 

I am still waiting for the information about our Adobe Connect room; but, as
soon as I have it, you will be alerted.   I think I have figured out how to
upload the data that we will need tomorrow (same as above and attached).   

 

As for process, I suggest that we start with the first project for which
there is not already a DELPHI answer (see above) and begin with the Y or X
dimension - then work our way through them one at a time.   After each round
of discussion, I believe that I can do an internal POLL inside Adobe asking
everyone to vote.   I'm not sure how it will work and whether I will be able
to see how each of you answered, but we'll figure it out as we go along.
Even if we use the Adobe Chat feature, we should be able to capture our
individual ratings decisions as we progress.  As we reach a DELPHI
solution/answer, I will post it into my master spreadsheet and then produce
new analyses and charts after we are finished with all of the projects.


 

I will suggest this again tomorrow, but we should try our very best to work
quickly and efficiently.   We only skipped 9 elements (or 30%) by virtue of
natural commonality in the existing ratings, which means that there will be
21 project/dimension combinations to discuss and, even if we move briskly,
that is still a lot of ground to cover in one hour.    That comes down to
about 1 decision every 2 minutes by the time we subtract the 5-10 minutes on
the front and back of every conference call needed for other business.  

 

At our next meeting after tomorrow's session (TBD), we can either continue
if we are not finished or use that time to step back, evaluate what we've
done, decide what we liked, disliked, and then ask ourselves if there is
anything else we would like to test.  

 

I hope this information is helpful.   I look forward to our session
tomorrow.

 

Ken Bour

Attachment: GNSO Work Prioritization-TEST RESULTS (KBv1).xls
Description: MS-Excel spreadsheet