RE: [gnso-wpm-dt] WPM-DT: Step 3a (In Progress) -- Analysis of 6 Test Ratings and Prep for DELPHI Session Tomorrow

["Gomes, Chuck" <cgomes@xxxxxxxxxxxx>]

Thanks Ken.
 
Chuck


________________________________

        From: owner-gnso-wpm-dt@xxxxxxxxx
[mailto:owner-gnso-wpm-dt@xxxxxxxxx] On Behalf Of Ken Bour
        Sent: Wednesday, December 16, 2009 6:45 PM
        To: gnso-wpm-dt@xxxxxxxxx
        Subject: [gnso-wpm-dt] WPM-DT: Step 3a (In Progress) -- Analysis
of 6 Test Ratings and Prep for DELPHI Session Tomorrow
        
        

        WPM-DT Members:

         

        Attached is an Excel spreadsheet containing the Test Results and
analysis based upon the 5 DT members and 1 Staff person who provided
rating files as of today.   Incidentally, the sixth rater is Liz
Gasster.   Following Jaime's suggestion, I asked her if she would like
to toss in another set of ratings and she agreed.   Marika also provided
a set of ratings; however, they were not included in this tabulation
since she cannot participate in our call tomorrow.  

         

        I have developed a mathematical process for identifying
commonality in the results although, for tomorrow's session, I will not
be able to include any additional ratings other than what has been
aggregated as of this evening.   The hard part was to noodle through the
options to see if I could come up with anything that would work - it
fell into place rather nicely using two common statistics:  Range and
Standard Deviation (StdDev).   

         

        There are a few things I'd like to report thus far and all are
contained in the attached spreadsheet.   Incidentally, the spreadsheet
has two tabs:   most of the detail and analysis are in the Ratings tab
and the Graphs/Charts (#4 and #5 below) are in the Summary tab.  

         

        1)      FYI - the 6 correlation statistics between X and Y for
our test ratings are:  

        Chuck             11%

        Jaime              25%

        Stephane        45%

        Wolf               51%

        Olga               52%

        Liz                  87%

         

        The above stat may be something we'll end up discussing tomorrow
during the DELPHI discussions.   Once again, the higher the correlation
statistic, the more likely it was that a high rating for X was matched
by a high rating for Y.   In other words, for 3-4 of you, the more
valuable a project was perceived to be, the higher was its perceived
resource consumption.   Alternatively, the more a project was perceived
to consume lots of energy/resource/time, the more value you assigned to
it.   

         

        2)      For the X axis, there were only 4 projects whose ratings
had a tight Range (Diff between High and Low = 1 or 2) and a StdDev <
1.0.   For those elements, I am recommending that we accept the MEDIAN
value (shown below) as our DELPHI answer.    They are highlighted in
GREEN (Range=1) and ORANGE (Range=2).   All the others had too much
deviation/variance and should be discussed by the entire group.  

         

        X VALUES

                        
SEQ NO

SVG

WUK

CG

JW

OC

LG

        DELPHI

1

7

5

1

6

2

7

                
2

3

4

3

3

4

3

        3.0

3

3

2

1

4

1

2

                
4

4

2

2

4

1

1

                
5

5

4

3

5

4

5

        4.5

6

5

5

5

5

4

5

        5.0

7

6

4

2

5

3

3

                
8

5

5

5

6

5

7

        5.0

9

5

2

3

5

4

5

                
10

5

2

3

5

3

5

                
11

5

3

4

5

5

7

                
12

5

3

2

6

3

4

                
13

5

4

2

4

3

5

                
14

6

5

3

6

4

7

                
15

2

3

3

3

3

6

                

         

        3)      For the Y axis, there are 5 projects whose ratings
produced a StdDev < 1.0 and they are highlighted as described above.

         

SEQ NO

SVG

WUK

CG

JW

OC

LG

        DELPHI

1

7

6

6

6

5

6

        6.0

2

4

6

3

6

3

2

                
3

2

5

1

4

1

1

                
4

5

2

1

4

3

1

                
5

5

4

4

4

3

6

                
6

5

3

1

7

2

6

                
7

4

6

5

7

4

3

                
8

6

7

7

6

6

6

        6.0

9

6

4

7

6

6

5

                
10

6

4

5

5

4

5

        5.0

11

6

4

4

5

5

5

        5.0

12

6

3

5

6

4

5

                
13

4

3

4

3

3

5

        3.5

14

4

6

5

7

5

7

                
15

5

4

5

7

4

4

                

         

         

        4)      Using RAW Means (Averages), the following would be the
Work Prioritization Chart given these six raters.   Notice that there
are no projects in Q4 (high cost, low value).   The averaging process,
as predicted, really does cause the numbers to cluster although, in this
case, not so much at the Average (4, 4), but in the range 3-5 on X and
4-6 on Y.   Nine or 60% of the projects are located in that tight area.
Incidentally, I used GREEN letters above Y=4 and ORANGE below Y=4.
Ties went to GREEN.  

         

         

         

        5)      If I take the same ratings and use MEDIAN values (middle
result) instead of MEANS, we get the following picture.  You can see
that the data spread is wider and less bunched compared to the above
chart.    

         

         

         

         

        I am still waiting for the information about our Adobe Connect
room; but, as soon as I have it, you will be alerted.   I think I have
figured out how to upload the data that we will need tomorrow (same as
above and attached).   

         

        As for process, I suggest that we start with the first project
for which there is not already a DELPHI answer (see above) and begin
with the Y or X dimension - then work our way through them one at a
time.   After each round of discussion, I believe that I can do an
internal POLL inside Adobe asking everyone to vote.   I'm not sure how
it will work and whether I will be able to see how each of you answered,
but we'll figure it out as we go along.   Even if we use the Adobe Chat
feature, we should be able to capture our individual ratings decisions
as we progress.  As we reach a DELPHI solution/answer, I will post it
into my master spreadsheet and then produce new analyses and charts
after we are finished with all of the projects.      

         

        I will suggest this again tomorrow, but we should try our very
best to work quickly and efficiently.   We only skipped 9 elements (or
30%) by virtue of natural commonality in the existing ratings, which
means that there will be 21 project/dimension combinations to discuss
and, even if we move briskly, that is still a lot of ground to cover in
one hour.    That comes down to about 1 decision every 2 minutes by the
time we subtract the 5-10 minutes on the front and back of every
conference call needed for other business.  

         

        At our next meeting after tomorrow's session (TBD), we can
either continue if we are not finished or use that time to step back,
evaluate what we've done, decide what we liked, disliked, and then ask
ourselves if there is anything else we would like to test.  

         

        I hope this information is helpful.   I look forward to our
session tomorrow.

         

        Ken Bour