Re: [ncdnhc-discuss] The NCDNHC's .org report is numerically inconsistent. Corrected figures do not change a lot.

[Dany Vandromme <vandrome@xxxxxxxxxx>]

Thomas Roessler wrote:
> 
> First of all, I'd like to congratulate the NCDNHC team for the
> great amount of work spent on its report for the .org bid
> evaluation.
> 
> However, the numerical material provided in the report is
> inconsistent.
> 
> The most obvious problem occurs in the table on page 49, where
-
There are less than 49 pages to this report
-
> responsiveness scores are simply sorted in decreasing order.  Here's
> a corrected version of that table:
> 
> +-------------+----------------+---------+-----------------+---------+
> | name        | responsiveness | support | differentiation | total   |
> +-------------+----------------+---------+-----------------+---------+
> | unity       |          27.25 |       9 |            20.5 | 24.5575 |
> | isoc        |          21.75 |       9 |            14.5 | 20.6725 |
> | ims/isc     |             14 |       7 |              15 |   16.78 |
> | gnr         |          26.75 |       3 |              14 | 15.8225 |
> | uia         |          16.75 |       5 |             7.5 | 12.5225 |
> | neustar     |          12.75 |       3 |              15 | 12.4425 |
> | dotorg      |           20.5 |       1 |               9 |  10.135 |
> | registerorg |          11.75 |       0 |              16 |  9.5725 |
> | .org        |              5 |       5 |               5 |    8.35 |
> | switch      |              8 |       0 |              10 |    6.16 |
> | organic     |              0 |       0 |            11.5 |     4.6 |
> +-------------+----------------+---------+-----------------+---------+
> 
> (total = 0.27 * responsiveness + support + 0.4 * differentiation)
> (Note that I didn't bother to reduce the numbers in the total column
> to the appropriate number of significant digits.)
> 
> The data sources I used for this table are on pages 4, 14, and 43 of
> the NCDNHC report.
> 
> Note that my results match the ones on page 27 of the report - the
> remaining differences may be due to rounding errors in the weighting
> factors.
-
Agree
At the end, Milton and I did conclude with a slightly different approach
to make the figures synthesis of the 3 criteria. That's why we offered
the two in the report.
-
> 
> When I asked Alexander Svensson to independently verify my concerns,
> he came up with another problem: The table on page 14
> (responsiveness and governance rankings) is inconsistent in itself.
> GNR's score should be 27.75 (instead of 26.75, thereby placing GNR
> on rank 1, ahead of unity with 27.25), while ISOC's score should be
> 23.25 (instead of 21.75; no ranking changes caused).
> 
> This error also sheds a spotlight on a methdological problem in the
> final evaluation of the NCDNHC's results: By merely averaging
> ranks, small differences in the originating scores (possibly caused
> by minor errors - the mistake in GNR's score corresponds to an error
> of about 3.5%!) are exaggerated in the end result.
-
Agree
-
> In this
> particular case, for instance, the corrected "responsiveness" rating
> would place GNR on the same rank as Neustar in the average ranking
> evaluation on page 26 of the report.
-
The second method for synthesis is not really impacted by the two
changes pointed out above:
-----------------------------------------------------------------------------
                 | Corrected grades    |  Grades as in rthe Report      
   |
-----------------------------------------------------------------------------
Unity            | 24.47               |                24.47           
   |
ISOC             | 21.00               |                20.47           
   |
IMS/ISC          | 16.73               |                16.73           
   |
GNR              | 16.00               |                15.73           
   |
UIA              | 12.47               |                12.47           
   |
Neustar          | 12.40               |                12.40           
   |
DotOrg Foundation| 10.07               |                10.07           
   |
Register Org     |  9.53               |                 9.53           
   |
.Org Foundation  |  8.33               |                 8.33           
   |
Switch           |  6.13               |                 6.13           
   |
Organic Names    |  4.60               |                 4.60           
   |
----------------------------------------------------------------------------- 

Regards
DV
-
> 
> I'll leave it to the NCDNHC team, ICANN staff, and the applicants
> themselves to check and verify the rest of the material provided.
> --
> Thomas Roessler                        <roessler@does-not-exist.org>
> _______________________________________________
> Discuss mailing list
> Discuss@icann-ncc.org
> http://www.icann-ncc.org/mailman/listinfo/discuss

-- 
-------------------------------------------------------------------
Dany VANDROMME                    |  Directeur du GIP RENATER

                Reseau National de Telecommunications
         pour la Technologie, l'Enseignement et la Recherche

                                  |  ENSAM
Tel   :  +33 (0)1 53 94 20 30     |  151 Boulevard de l'Hopital
Fax   :  +33 (0)1 53 94 20 31     |  75013 Paris
E-mail: Dany.Vandromme@renater.fr |  FRANCE
--------------------------------------------------------------------