ACT Comments on the Draft ATRT Proposed Recommendations - A search for Measurement

[Jonathan Zuck <jzuck@xxxxxxxxxxxxx>]

Quis custodiet ipsos custodes?

This famous query by Juvenal (loosely "who watches the watchers?") is nowhere 
more applicable than to ICANN and the ATRT process. When ACT submitted comments 
to the ATRT back in February, we suggested the review process should be 
preceded by the establishment of metrics with which to measure the success by 
ICANN processes and initiatives and explore areas for improvement. Absent that, 
a review team could hardly hold ICANN accountable to objectives that were not 
set. It comes as no surprise that the review process was as contentious as it 
was. Specious misdirection like "we're not a regulator, we're a non-profit" are 
the understandable result of trying to review an organization in the abstract 
instead of the concrete. Making up criteria and then judging on the basis of 
that criteria was inevitably fraught with hazard. While predictable, the 
results of this process are disheartening.

Perhaps just as disheartening is the ATRT itself making some of the same 
missteps as ICANN. The ATRT recommendations suggest that ICANN deal with public 
comment in a more structured fashion, the appropriate implication being that 
providing a forum for speech is not the same as listening.  Unfortunately, it 
is difficult to tell from the ATRT recommendations and transcripts which 
comments were considered and how and the justifications of setting those 
suggestions aside. ACT believes that the recommendations from the ATRT on 
public comments are excellent but would suggest future review teams lead by 
example.

Setting process aside for the time being, however, I would like to reiterate 
that ICANN will never, I stress never, be truly accountable or transparent 
without established, public performance metrics for its various initiatives and 
departments. In certain instances, the ATRT report suggests that ICANN is 
looking into metrics in a couple areas but this is a serious issue that needs 
to be treated more seriously than that. While it makes sense that ICANN staff 
proposes actual target values for various metrics, it seems completely 
appropriate that the ATRT suggest a framework of measurable objectives.

A good start might be the ATRT recommendations themselves. An initial metric 
might be a timeline for the implementation of the recommendations. Absent that, 
we are likely to find ourselves once again confronted with effort as a proxy 
for results. During the next review process for Accountability and 
Transparency, we will read how "ICANN began..." or "ICANN launched a study of" 
rather than actually having a system for tracking public comments or GAC 
recommendations.

In February, ACT made the following suggestions for potential metrics:

(a)          continually assessing and improving Board governance, including 
ongoing evaluation of Board performance, Board selection process, the extent to 
which Board composition meets ICANN's present and future needs, and the 
consideration of an appeal mechanism for Board decisions;

1.            Board candidate pool (quantity)
2.            Board candidate diversity (regions represented)
3.            Is the Board following appeal mechanisms?

(b)          assessing the role and effectiveness of the GAC and its 
interaction with the Board and making recommendations for improvement to ensure 
effective consideration by ICANN of GAC input on the public policy aspects of 
the technical coordination of the DNS;
1.            % of ccTLDs signing Framework Agreements
2.            % of GAC Members signing a "Declaration of Support" for the ICANN 
model for DNS management.
3.            % of non-GAC members signing a Declaration of Support for the 
ICANN model

(c)           continually assessing and improving the processes by which ICANN 
receives public input (including adequate explanation of decisions taken and 
the rationale thereof);

1.            Categorization and Analysis of Comments
2.            Documentation of Questions Asked and Community Response
3.            Documentation of Comment Responses, ensuring each issue raised is 
addressed and resolved one way or another.

(d)          continually assessing the extent to which ICANN's decisions are 
embraced, supported and accepted by the public and the Internet community; and
1.            Is policy development community-driven or Staff-driven?  ( e.g 
Expression of Interest; Rr-Ry Integration
2.            % of comments disregarded
3.            Surveys to determine baselines and deltas in confidence

(e)          Assessing the policy development process to facilitate enhanced 
cross community deliberations, and effective and timely policy development.
1.            % of SOs filing comments
2.            % of SOs joining Working groups
3.            Diversity and representation within SO's
4.            Mechanisms for outside participation perhaps driven by SO's 
rather than general comment periods.

One area that seems well explored by the ATRT is the area of public comments 
for that I commend them. The structured handling of comments is something that 
should have been considered long ago. Hoping some of these other ideas might 
see the light of day.

Further ACT would stress that while the focus of this particular review was on 
the Accountability and Transparency of ICANN, it should not limit itself to a 
review of existing mechanisms of transparency. The very essence of transparency 
is understanding what the organization intends to do and a requirement for 
accountability is being able to determine if they have succeeded. Accordingly, 
it's appropriate for the ATRT to request the ICANN staff develop a SYSTEM of 
metrics across all of its initiatives, thereby clarifying objectives and record 
of success. For example, a metric with respect to registrar agreement 
compliance is critical for transparency and, ultimately, accountability.

Metrics should be second nature to the current CEO and time after time, both 
Mr. Beckstrom and Mr. Dengate Thrush have agreed and said "What gets measured 
gets done." With nothing getting measured...there's certainly very little to 
review. The ATRT have been giving a sacred duty at a critical time in ICANN's 
history and future. Quis custodiet ipsos custodes?

Jonathan Zuck
President
Association for Competitive Technology
3 December 2010