Comment on Initial Report on Data & Metrics for Policy Making

[Amr Elsadr <aelsadr@xxxxxxxxxxx>]

Hi,

First, I would like to thank the members of the working group and ICANN staff 
supporting them for the work they have done to meet the requirements set forth 
in the working group charter, as well as the opportunity to have comments 
submitted and considered prior to publication of the working group final report 
and recommendations.

In principle, I fully support the notion of the GNSO improving its policy 
development process by using empirical data while considering the intentions 
and implications of policies being developed. Furthermore, and as the working 
group has noted, metrics and quantitative analysis of data can be very useful 
in helping to determine the extent to which a previously developed policy is 
meeting its desired goal, or not.

Having said that, I do have the following comments on some of the content and 
recommendations of the initial report:

The suggested language of section 4.5 of the GNSO operating procedures 
detailing the “Working Group Metrics Request Form” only indicates the 
procedures for requesting data/metrics. There is no indication in the proposed 
changes to the operating procedures or the metrics request tree that prior to 
aggregation of data, there is any requirement for the chartering organization 
(GNSO Council) to approve the request. Considering the potential cost of both 
time and funds (at the issue scoping phase or during the PDP working group 
phase), it may be worthwhile to consider whether or not the chartering 
organization should play a role in determining the extent to which the “Issue 
to be solved” in the working group metrics request form warrants such 
delays/costs. Clarification on a process to approve a submitted working group 
metrics request form should ideally be included in the DMPM working group’s 
final report.

In section 5.3.5.1, the third bullet observes that “For instance, the charter 
template could require that WGs identify a set of baseline data that should be 
captured to allow for the community to determine if a set of recommendations 
was effective or not”. It is important to note here that collection of data and 
analysing it quantitatively cannot be the sole determinant of whether or not a 
set of policy recommendations is effective. Qualitative research methods plays 
an important role in informing a discussion during policy development, and may 
very well also play a role in measuring its success post-implementation. 
Although addressing the means by which qualitative research methods may be 
useful to the GNSO in policy development is not strictly within the scope of 
this working group, citing the advantages of using quantitative analysis should 
not be expressed as an absolute determinant of the success or failure of a 
policy recommendation. Qualitative methods of research often uncover compelling 
considerations to be taken account of that may not hold any statistical 
significance in quantitative analysis. The redline text in section 9 of the PDP 
manual suggested by the DMPM working group takes this into account nicely. This 
should also be reflected in the relevant parts of the working group final 
report and recommendations.

Finally, at no point in the DMPM working group’s initial report, or even in the 
Metrics Request Decision Tree is critical appraisal of the collected data 
mentioned. Any quantitative analysis of data should be subject to transparent 
methods of assessment prior to putting it to use for the purpose of 
evidence-based policy development. This could be done during the public comment 
period for a preliminary issues report, or perhaps during the PDP working group 
deliberations. An example where this may be constructive is determining the 
appropriateness of methods used for data collection. Was the data collected 
using an established reliable system? Are the data elements/samples 
geographically/temporally representative of the study subject, which may be 
impacted by a policy being developed? Was the selection of study subjects (or 
controls if applicable) biased resulting in an inability to generalize the 
results? These are simple examples of questions that need to be answered before 
determining the extent to which data/metrics are usable in a PDP working group.

Thanks.

Amr