gTLD Marketplace Health Index Proposal: work needed
Below are notes regarding the gTLD Marketplace Health Index Proposal. Some of the proposed metrics (KPIs) -- and the assumptions that underlie them -- need more examination. I offer the following as constructive comments: The proposal states that "ICANN must be able to efficiently and cost-effectively collect and analyze data underlying these KPI." I observe that some of the proposed KPIs are crafted around data that is currently available. That makes them cost-effective to collect, but it does not mean that the data is always fit for the designated purpose. In some cases ICANN may need to develop new sources of data. I.(b) Average number of registrars offering each gTLD: The proposal assumption is "A larger number might indicate greater technical, operational, legal, etc., accessibility of gTLDs by registrars." This assumption is flawed, because many gTLDs are not intended for wide accessibility. An example are the .BRAND TLDs, many of which are deliberately accessible via one registrar only. The KPI does not take into account the diversity of the market and the incentives of the varied registries and registrars. I.(f) Innovation and introduction of new services: RSEP data is not indicative of innovation or new services. The overwhelming number of RSEP requests are for non-innovative and old services, like release of two-character domains, standard IDN sets, and country & territory names. I.(g) gTLD renewal rates: The proposal assumption is "A greater ratio of renewals to deletions of second-level gTLDs might reflect greater actual use or intent to use domains and a greater perception of those domains' intrinsic value by registrants." Actual use is different from intent to use. Neither is demonstrated by renewal rate, and intent to use cannot be gauged reliably. Actual usage can be measured by more accurate and objective means, including spidering sites to see if they have non-parking content on them. II.(c) Relative incidence of ICANN breach notices issued to registries and registrars. The proposal assumption is that "A smaller number of breach notices could indicate fewer noncompliant registries and registrars, and therefore, a healthier and more trustworthy marketplace." This KPI needs to be re-thought for the following reasons: One, the Compliance Department decides when to send breach notices. So ICANN controls this KPI and the metrics. Two, the number of breach notices does not tell us anything about how many registries or registrars are actually out of compliance. ICANN does not send out a breach notice when a registrar or registry is reported to be -- or is actually even found to be -- out of compliance. A formal, public breach letter goes out only after ICANN makes multiple private communications to the contracted party and the contracted party fails to fix the problem after several attempts. II.(d) Quantity and relative incidence of complaints regarding inaccurate, invalid, or suspect Whois records. The assumption is that "A smaller number of Whois accuracy complaints could indicate greater reputation and trust of gTLDs." Use of the term "complaints" indicates that ICANN wishes to use the data from its Whois Inaccuracy Complaint Form. The number of inaccuracy complaints has no demonstrable correlation with the public's perception. Clearly there are many, many times more invalid WHOIS records than are reported to ICANN. Also, the language used here is confusing -- what is the difference between "inaccurate" and "invalid," and what is considered "suspect"? Instead, statistics should be provided by the WHOIS Accuracy Reporting System (ARS) that ICANN has been building (http://whois.icann.org/en/whoisars). ARS is designed to objectively measure the incidence of inaccuracy across and within TLDs, and is based on clear definitions of what is "inaccurate." III. (b) Total number of unique phishing reports, as measured by Anti-Phishing Working Group reports. I am the editor of the APWG's quarterly trends reports, and am the co-author of the APWG's semi-annual Global Phishing Survey Reports. The APWG's quarterly trends reports do not distinguish between phishing on gTLD domains versus ccTLD domains and therefore are not appropriate for the proposed use. The Global Phishing Survey Reports provide break-downs by TLD, and so the stats can be separated into gTLD and ccTLD. More crucially, the Global Phishing Survey Reports distinguish between domain names registered by phishers ("malicious domains") and domains that were broken into on compromised servers. Domain names registered by phishers are more relevant to the trust issue. Domains that were broken into on compromised servers is really a measure of hosting (IT) security and so are not as relevant to stability is the gTLD space. I also note that phishing is only one kind of security or abuse issue, and it is one that consumes relatively few domain names compared to other types of problems. Terminology also needs some examination. For example, note that III(a) and III(b) are measures of security, not of "stability". III(c) is a measure of technical stability, and is an important metric, but the Health Index is proposed to measure "non-technical stability." The health of the marketplace can also be measured by: . the total number of registrars and the number accredited and de-accredited . the number of registry failures, the number of registry contracts that change hands, and/or the number of registries sent to Emergency Transition. Sincerely yours, --Greg Aaron President, Illumintel Inc.