Report of Public Comments

 

Title:

Data & Metrics for Policy Making

Publication Date:

06 October 2015

Prepared By:

Steve Chan

Comment Period:

Comment Open Date:

29 July 2015

Comment Close Date:

7 September 2015

Staff Contact:

Steve Chan

Email:

Policy-staff@icann.org

Section I:  General Overview and Next Steps

This public comment proceeding sought to obtain community input on the Initial Report from the GNSO's Working Group (WG) on possible recommendations for the use of Data and Metrics for Policy Making.  The WG's intent is to develop solutions that will improve the way in which consensus policies are developed and help ensure that the most critical registrant, registry, and registrar issues are addressed by evolving the culture to better informed, fact-based policy development and decision-making.

 

 

Section II:  Contributors

At the time this report was prepared, a total of [number] (n) community submissions had been posted to the Forum.  The contributors, both individuals and organizations/groups, are listed below in chronological order by posting date with initials noted.  To the extent that quotations are used in the foregoing narrative (Section III), such citations will reference the contributor’s initials.

Organizations and Groups:

Name

Submitted by

Initials

Registry Stakeholder Group

Paul Diaz

RySG

Google

Stephanie Duchesneau

GOOG

Business Constituency

Steve DelBianco

BC

ISPCP

Christian Dawson

ISPCP

RrSG

Michele Neylon

RrSG

At-Large

Heidi Ullrich

ALAC

NCSG

Rafik Dammak

NCSG

 

Individuals:

Name

Affiliation (if provided)

Initials

Maarten Botterman

Public Interest Registry

MB

Arthur Zonnenberg

Hostnet

AZ

Amr Elsadr

 

AE

Section III:  Summary of Comments

General Disclaimer :  This section is intended to broadly and comprehensively summarize the comments submitted to this Forum, but not to address every specific position stated by each contributor.  Staff recommends that readers interested in specific aspects of any of the summarized comments, or the full context of others, refer directly to the specific contributions at the link referenced above (View Comments Submitted). 

 

 

 

 

 

Section IV:  Analysis of Comments

General Disclaimer :  This section is intended to provide an analysis and evaluation of the comments received along with explanations regarding the basis for any recommendations provided within the analysis.

 

Refer to the Public Comment Review tool tables below for analysis of the comments and actions taken by the group, if any.

 

 

 

 



Public Comment Categories – Table of Contents

General Comments

Charter & Final Report Templates

Pilot Effort

Metrics Request Form

Early Outreach


Public Comment Review Tool – DMPM WG – Initial Report

Updated 30 September 2015

General Comments

#

Comment

Contributor

WG Response / Action Taken

Section Summary:

All comments were in “Agreement” with the WG’s recommendations.  Four offered up “New Ideas” for a total of seven actions taken by the WG when considering the WG Final Report.

 

  1.  

Fully support the needs and overall aims of the work. Just noting that data as such are important to be able to do fact based policy development, and that it is crucial to define which data you need.

 

Next to “specific purpose” (i.e. what is the issue at hand) I also see strong merit in “approach” (i.e. how are we going about to make sure the policy making process is well informed). I have not yet understood how the WG sees this.

 

From my experience with methodological approaches (during my work for RAND Corp and GNKS) towards facts based policy development I can warmly recommend to consider the ex ante impact study framework that several public interest organisations around the world adopted as part of their better regulation planning, such as UNDP and the European Commission.

 

Whereas ICANN is *not* a government type of organization, it is an organization that aims to work in the public interest, and it has committed itself to “better policy making” as an ongoing process. Hence lessons can be learned from best practice frameworks such as the two mentioned above.

 

What does this mean for data metrics?

 

The approach (ex ante impact assessment) determines certain systematic steps to be taken to ensure a fully informed decision can be taken. This includes systematic collection of data for specific purposes in the steps.  In particular, a systematic collection of data that help determine to define <1> the impact on different stakeholdergroups and <2> the impact of specific future developments that are highly relevant yet not within the “control” of ICANN, would, in my humble opinion, help develop to enhance ICANNs ability to get to facts based policy decisions.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00000.html

MB

Agreement   New Idea  

WG Response:

The WG discussed the concept of ex ante impact study and determined that it was one of the key concepts driving its work.  However, the WG noted that its recommendation on reviewing the implementation of policy only considered positive implications and not the possibility of a negative outcome requiring an adjustment to the implemented policy

 

Action Taken: [BC1]

Review final report language to determine if both positive and negative implications are mentioned and revise Recs #4 & #5 reflecting the possible outcome of both.

 

NOT COMPLETED:   Added positive and negative outcome statement to WG observations in section 5.3.5.2.

 

  1.  

I fully support the goal of this work. I think in particular high level performance data gathered for the IRTP will provide an essential pilot effort to show the positive value the DMPM WG can have on policy.

In additional response to the charter questions as made by the WG, find my comments below.

 

Charter Question A:

Data describing a problem can be the start of a policy, in addition to being gathered in response to policies.

 

Charter Question B:

Beyond contractual compliance, it is advisable to address and accept input on whether sufficient data and metrics exist for measuring existing policies

 

Charter Question C:

I agree on the necessity of prioritizing and right-sizing solutions.

 

Charter Questions D, E and F.

Note that a lot of interesting data can already be gathered on a registry level alone, bypassing the need for registrar consent or cost.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00001.html

AZ

Agreement

WG Response:

Noted and the WG thanks the commenter for the input also noting that the WG has deliberated on these exact points.

 

Action Taken:

None

  1.  

The gTLD Registries Stakeholder Group (RySG) supports the proposed recommendations for the use of Data and Metrics for Policy Making. We commend the Working Group's efforts, and appreciate its inclusion of previous RySG input in this Initial Report.

 

See full comment:  http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00002.html

RySG

Agreement

WG Response:

WG Appreciates the comment of support and thanks the group for their response.

 

Action Taken:

None

  1.  

Google Inc. appreciates the opportunity to comment on the Initial Report on Data & Metrics for Policy Making (“Initial Report”). Google agrees that the Generic Names Supporting Organization (GNSO) Policy Development Process (PDP) should make use of available data to improve policy outcomes. The use of data can complement the multistakeholder process and will help focus community efforts on the issues and initiatives with the broadest and most significant potential effects. The use of data and metrics may also improve the effectiveness of the PDP by helping the community set clear goals and benchmarks as a part of the PDP itself. Data and metrics will allow ICANN and the community to assess whether PDP outcomes support their objectives, and to use these findings to better guide future efforts. Taken collectively, the use of data and metrics to refine and improve the PDP will benefit all stakeholders.

While supporting the general effort of the working group, we propose the following recommendations to improve the use of data and metrics as part of the PDP:
 

[See suggested comments through sections of this document]

 

See full comment:  http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00003.html

GOOG

Agreement

WG Response:

WG Appreciates the comment of support and thanks the commenter for their response.

 

Action Taken:

None

  1.  

Provide for the collection of qualitative data, in addition to quantitative data, to the extent that it is not already captured in the multistakeholder process.

The Initial Report focuses almost exclusively on quantitative data over qualitative data. While the nature of the multistakeholder process itself ensures that the PDP takes into account some qualitative experience, there may be relevant voices and experiences that are not reflected in the PDP. Alternatively, the composition of a PDP working group may not be adequately representative of the range of parties affected by the issue at study. In these cases, structured use of qualitative data as a complement to quantitative research may significantly improve the policy process.

The need for qualitative data is particularly acute where policies touch on registrants or Internet users. Given the size and diversity of these categories, it is impossible to garner an accurate view of registrants’ or users’ perspectives solely through the ICANN working group model. In this model, participation by registrants and users is highly idiosyncratic, and often not adequately representative of the Internet user community as a whole. As such, policies that directly affect registrants or users could benefit from surveys or direct observation of more representative samples of users to understand how they engage with the Domain Name System (DNS) along relevant indicators.

 

See full comment:  http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00003.html

GOOG

Agreement

WG Response:

The group noted that the WG’s charter is to determine ways in which the community can use data, mostly quantitative, as much of the current policy deliberations are focused around qualitative data.  The WG is reviewing how to incorporate data and metrics that are measurable in the policy process and the WG’s initial reaction is that this suggestion might be out of scope.  The group also discussed the difficulty of defining, collecting, interpreting qualitative data into measurable mechanism.  The WG noted that much of the current policy process is focused around qualitative input, from the early stages of the policy process via the Issue Report, Public Comments, early outreach from WGs, and deliberations by WGs and the Council.  The concepts of surveys was also discussed in their structure and audience with desires to reach well outside the ICANN community noting the difficulties of communicating their availability, but perhaps that these issues could be mitigated by the use of enterprise services such as Big Pulse available internationally.  In the end for surveys, well distilled qualitative data can be become quantitative as for example CCT Review Metrics.  Ultimately, WGs should define their problem quantitatively so that solutions are defined quantitatively thus becoming a better defined communication tool for qualitative reactions to problem definition and proposed solutions.

 

Action Taken: [BC2]

No changes to recommendations, but review report for changes to our justifications/observations to reflect improved communication for WGs who might make qualitative inputs.

 

NOT COMPLETED:   TBD Reviewed document and no changes were made.  WG reiterated current use of qualitative data within the current policy development process .

 

  1.  

Leverage existing data sources to the fullest extent possible.

Working groups should be encouraged to use easily accessible, pre­existing data sources where available. For example, working groups could rely first on data that is publicly available, as well as data that is already being collected and processed by existing providers that observe and analyze DNS trends, WHOIS information or other relevant data. This approach avoids the costs and delays associated with appointing a new third party provider to collect and handle relevant information. To facilitate this process, ICANN staff could include in the issues report all publicly available or readily accessible data sources related to a PDP and how such sources would be used, if applicable. The Key Metric Considerations sections for future working group charters proposed in Annex A could be revised to reflect these available data sources.

 

See full comment:  http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00003.html

GOOG

Agreement   New Idea  

WG Response:

The WG agrees with the comment and had deliberated on the concept of publicly available data being considered first for any type of metrics and data request.  The group reconfirmed that the use of a 3 rd party aggregator would only be utilized in cases where data is not necessarily publicly available or that is sensitive data that may contain anti-trust implications.

 

Action Taken: [BC3]

Make adjustment to Annex A in Charter Template to include publicly available data sources as a sort of hints & tips section for future drafting teams of charters.

 

COMPLETED:   Created Hints & Tips form in Annex C to be linked to in operating documents so that it can expand with updates without requiring the standard approval process.

  1.  

Assign full­time staff to manage data collection and analysis.

Past efforts by ICANN to collect data for the purposes of reporting or policy making have relied heavily on the use of independent third party providers. Similarly, the initial report makes repeated reference to the use of third parties to collect and process such data. To the extent that the use of data and metrics as described in the initial report becomes a regular and significant part of the PDP, ICANN should appoint or hire full­time staff with the requisite skills to manage the collection and analysis of such data, and/or maintain ongoing relationships with third party firms capable of filling gaps in the organization’s capacity to manage such data. Ad hoc use of independent third party providers can create inefficiencies, including time expended to manage a Request for Proposals and engage a provider, as well as costs in retaining that provider. ICANN should weigh whether efficiency and value could be maximized by retaining staff with the requisite skills to manage such processes or establishing ongoig retainers with capable third party firms.

See full comment:  http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00003.html

GOOG

Agreement

WG Response:

The WG agrees in principle about efficiencies and value being an important requirement for managing data and metrics requests either through staff or 3 rd party providers and this will likely be an outcome to consider from the pilot effort of recommendation #1.  The WG reconfirmed the pilot effort’s scope that it is meant to be targeted, lower cost types of requests where less formal RFPs would be required to fulfill future requests.  It was also confirmed again that 3 rd party aggregators will be required regardless where sensitive data or requirements of anonymization are needed.  Levels of trust for ICANN handling such data are minimal at best.  The WG discussed the possibility to explore staff expansion to handle analysis, but the requirement remains where confidentiality requirements exist.  Certainly, established relationships with data providers and/or companies on retainer are important.  The WG noted that this occurs already in some parts of staff, for example the AoC review of CCT.  However, they noted that staff are not dedicated to such a function.

 

Action Taken: [BC4]

WG to add suggestions that dedicated staff, established relationships with data providers and/or retainers should be considered as part of the recommendation #1 pilot effort.

 

COMPLETED:   Updated Recommendation #1 Details to suggest (not recommend) dedicated staff and relationships with providers.

 

  1.  

Consistently apply procedures for the collection and and analysis of data.

We appreciate the Initial Report’s high­level acknowledgement of the limitations on requesting data from registries and registrars and support the principles set forth for the process of requesting data from the contracted parties. Given their roles in the DNS, a significant fraction of PDP­relevant data will inevitably reside with the contracted parties. However, we are somewhat concerned by the Initial Report’s differential treatment of registry and registrar data with that of outside parties. The principles outlined, including clear purpose, confidentiality, anonymization, aggregation, and removal of PII, should be generally applicable to any data collection process and not be specific to the contracted parties. Further, just as ICANN and working groups cannot require outside parties to surrender data, the same must be true of registry and registrar data.

While adherence to these principles is likely necessary to mitigate contracted parties’ concerns in surrendering data, it may not be sufficient. Where issues persist, the outcome of this non­PDP working group cannot be deemed to create new requirements for contracted parties to surrender data as part of the policy process outside of the limited circumstances provided for in the Registry Agreement and Registrar Accreditation Agreement. Accordingly, working groups must also consider the potential selection biases associated with data that is voluntarily provided by contracted parties.

See full comment:  http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00003.html

GOOG

Agreement

WG Response:

The WG is in agreement with the comment.

 

Action Taken: [BC5]

Add statement to Contracted Parties principles that it should be applicable to any data collection process.

 

COMPLETED:   Updated principles bullet to make applicable to all data requests in Section 5.3.4.1.

 

  1.  

Clarify any changes to PDP to the Work Product Templates referenced in the Initial Report.

While the Initial Report makes repeated reference to the use of standardized templates set forth in the Draft Redline of the GNSO Operating Procedures, the redline itself does not include active links to the documents referenced. With the exception of the proposed updated Charter, which is also set forth in Exhibit A, it is not clear whether and how this documentation is to be updated. This documentation should be provided as part of a subsequent public comment process to allow the community to weigh in on any proposed changes to the documentation, which may have implications for the PDP.

See full comment:  http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00003.html

GOOG

Agreement

WG Response:

The WG acknowledges the comment but does not have concern with the omission of the links to the templates as suggested in the red-line of the GNSO Operating Procedures.  While links to the templates were not included, the intent here is to fast-track implementation of the WG’s recommendations should the Council approve them thus avoiding an additional public comment period.  The templates will be shells for use by staff when initiating the respective work product and not impact any working group or drafting team.  Further, the recommendation suggested by the DMPM is listed in its report.  Staff confirmed that once the template are made available, a communication will be sent to the GNSO prior to formal publication of the next version of the GNSO Operating Procedures.

 

Action Taken:

None

 

  1.  

The BC strongly supports the use of data and metrics, among other tools, to aid in ICANN's policy making processes.  Specifically, the BC believes that the use of data and metrics as a complement to the multi-­‐stakeholder process will improve the effectiveness of the PDP by helping the community set clear goals and benchmarks at the outset, and will also help focus efforts on the initiatives with the most significant impact on the community.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00004.html

BC

Agreement

WG Response:

The WG appreciates the BC’s support of the DMPM proposed recommendations.

 

Action Taken:

None

  1.  

The BC believes that Working Groups should be encouraged to use data sources that are preexisting and/or publicly available when considering what data and metrics would support a PDP, as to minimize the costs and delays associated with appointing a new third party provider to collect and handle the information.  To the extent that the use of data and metrics as described in the initial report becomes a regular and major part of the PDP, ICANN should consider hiring additional staff with the requisite skills to manage the collection and analysis of such data.  Improved approaches to making policy may be found via other organizations with similar challenges whose work can serve as examples.  Also, the academic discipline of policy-­‐making continues to progress, and likely has useful information for addressing specific uses of data and metrics.  (One example: "Cambridge Conference: Policy-­‐Making in the Big Data Era", June 2015; http://www.publicpolicy.cam.ac.uk/data-­‐for-­‐policy-­‐conference)

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00004.html

BC

Agreement

WG Response:

The WG appreciates the BC’s support of the DMPM proposed recommendations.  The WG noted that this comment matched the prior Google comment and refers readers back to line item #7 on page 8 of this document.

 

Action Taken:

None

  1.  

The BC supports the collection of qualitative data, in addition to quantitative data, to the extent that it is not already captured in the multi-­‐stakeholder process.  Qualitative data is particularly important where policies touch on Internet users.  Given the size and diversity of this community, it may not be possible to garner an accurate and fully representative view of users’ perspectives solely through the ICANN Working Group model.  Accordingly, policies that directly affect users could significantly benefit from surveys or direct observation of more representative samples of users.  However, qualitative input should be used to inform and interpret data and metrics, but not to replace or outweigh them.

 

The BC appreciates ICANN's move toward research-­‐based policymaking, and envisions an outcome with an adaptive approach that will improve the quality of ICANN policies.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00004.html

BC

Agreement

WG Response:

The WG appreciates the BC’s support of the DMPM proposed recommendations.  The WG noted that this comment matched the prior Google comment and refers readers back to line item #5 on page 6 of this document.

 

Action Taken:

None

  1.  

The Internet Service Providers and Connectivity Providers operate Internet backbone networks and/or provide access to Internet and related services to End Users.  We are key players on the Internet, and have an essential role in its stability and development.  The Internet Service Providers and Connectivity Providers Constituency seeks to strongly support the efforts and work product contained within the Initial Report on Data & Metrics for Policy Making.

 

The ISPCP constituency will continue to comment in the subsequent dialogue phases.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00005.html

ISPCP

Agreement

WG Response:

The WG appreciates the ISPCP’s support of the DMPM proposed recommendations.

 

Action Taken:

None

  1.  

Registrars would like to thank ICANN staff and the working group volunteers for their work in on the Initial Report on Data & Metrics for Policy Making. The potential to improve and shape the ICANN policy development process outlined by this work are important and worthy of notice by the entire ICANN community.

 

The RrSG believes the basic premise of this WG would benefit the ICANN community by encouraging better informed, fact-based policy. We would like to see ICANN make a commitment to this initiative and consequently have a number of comments below:

 

Broadly, we would like to note and emphasize the WGs recognition that data will be most impactful in the processes prior to launching a PDP, and the initial stages there of. Given the finite resources of the community, it is absolutely crucial that we ensure that the problems the community attempts to resolve are both material and appropriately prioritized.

 

Following that, we would also like to highlight the importance of defining wherever possible, quantitative, measurable goals for undertaken PDPs. The community should be casting a wary, skeptical eye on issues raised that lack evidence or rely on anecdotes and the GNSO should be able to reject a PDP, or at least constrain a PDP, if there is no evidence to support that a problem exists.

 

We’d like to recommend the working group consider emphasizing the continuous improvement role that data driven decision making can provide. This should include the measuring of the impact of a policy change for a period of time post implementation, and possibly a process to revert policy if the desired impact is not achieved.

 

As noted in the initial report, an independent third-party will likely be able to collect, anonymize and aggregate data. Such a service provider, however, will require funding and the cost should not be borne by the contracted parties. If ICANN wishes to promote more fact-based decision making, it will need to commit to financially supporting service providers to collect and process data and/or facilitate the provisioning of data from other independent sources. Ensuring that an approved, secure, and neutral third-party is collecting and aggregating data from contracted parties will make adoption of these recommendations far more palatable and will help to allay some of the concerns outlined below.

 

Lastly, we wish to reiterate a number of reservations regarding the collection and sampling of data from registrars:

  • Given the diversity in the size of registrars, from one person operations to those with thousands, there will be difficulty in obtaining data from a broad enough array of registrars to ensure information is representative. Not all registrars will have the capacity or resources to provide data. We would encourage the working group to consider how to ensure that data collected has appropriate diversity and sampling.
  • There is a danger that data may be skewed by the large differences in registrar process implementation and business models. These differences will need to be carefully accounted for.
  • Registrars hold a considerable volume of personal and private information. This type of information should be excluded from any ICANN related data request.
  • The collection of data from Registrars may raise anti-trust concerns.
  • The length of time data collected will be kept for is also concern. It may grow stale and less relevant over time. Relatedly, the data should only be used for the purpose originally and explicitly intended, and should be disposed of when the relevant work is complete. This will help mitigate opportunistic data mining and requests for data without a legitimate policy concern.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00006.html

RrSG

Agreement   New Idea  

WG Response:

The WG appreciates the Registrar’s support of the DMPM proposed recommendations.  An idea was discussed about an “opt-out” option if there is a determination that a data request will contain confidential data.  The request decision tree has an approval mechanism whereby a request has the option of being refused when not contractually obligated to do so.  The WG also touched upon the quality of data, reliability and accuracy of qualitative data.  The diversity of Registrar models was also discussed in how some requests may not be applicable in certain situations.

 

Action Taken: [BC6]

WG to review the principles in section 5 for inclusion of the opt-out notion on data and metrics requests.  A further review of the requirements section of the Metrics Request Form will be completed.

 

COMPLETED:   Added two bullets to Section 5.3.4.1 principles.

  1.  

First, I would like to thank the members of the working group and ICANN staff supporting them for the work they have done to meet the requirements set forth in the working group charter, as well as the opportunity to have comments submitted and considered prior to publication of the working group final report and recommendations.

 

In principle, I fully support the notion of the GNSO improving its policy development process by using empirical data while considering the intentions and implications of policies being developed. Furthermore, and as the working group has noted, metrics and quantitative analysis of data can be very useful

in helping to determine the extent to which a previously developed policy is meeting its desired goal, or not.

 

Having said that, I do have the following comments on some of the content and recommendations of the initial report:

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00007.html

AE

Agreement

WG Response:

The WG appreciates the contributor’s general support of the DMPM proposed recommendations, noting some enhancements.

 

Action Taken:

None

  1.  

Finally, at no point in the DMPM working group’s initial report, or even in the Metrics Request Decision Tree is critical appraisal of the collected data mentioned. Any quantitative analysis of data should be subject to transparent methods of assessment prior to putting it to use for the purpose of evidence-based policy development. This could be done during the public comment period for a preliminary issues report, or perhaps during the PDP working group deliberations. An example where this may be constructive is determining the appropriateness of methods used for data collection. Was the data collected using an established reliable system? Are the data elements/samples geographically/temporally representative of the study subject, which may be impacted by a policy being developed? Was the selection of study subjects (or controls if applicable) biased resulting in an inability to generalize the results? These are simple examples of questions that need to be answered before determining the extent to which data/metrics are usable in a PDP working group.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00007.html

AE

Agreement   New Idea   Concerns  

WG Response:

Initial WG reaction to this is that these are questions that are more often applied to commissioned studies (i.e. the Whois data accuracy study or recent survey efforts launched by the CCT) and less about just getting access to raw data.  There's never going to be a solution to it in the sense that data is collected and there has to be some ability to allow the expertise and clear thinking of the working group to assess whether or not the data it has received is actually going to be particularly relevant or persuasive given the question they're trying to answer.  There's not always going to be a need for geographically or temporally representative data. If data/results come back with bias then it should be noted and/or rejected.  The disclosure can always be referenced at a later time. 

 

Action Taken: [BC7]

Create some assessments tips about what would constitute appropriate data in the metrics request decision tree and request form.  Add a decision tree element for any considerations about the data raised that needed to be addressed by a group. Make note that any quantitative analysis of data should be subject to transparent methods of assessment prior to putting it to use for the purpose of evidence-based policy development. Potential suggestions:

  • Was the data collected using an established reliable system?
  • Are the data elements/samples geographically/temporally representative of the study subject, which may be impacted by a policy being developed?
  • Was the selection of study subjects (or controls if applicable) biased resulting in an inability to generalize the results?

 

COMPLETED:   Added to Hints & Tips page in Annex C.  Updated Decision Tree in Annex B with added assessment logic.

 

  1.  

The ALAC appreciates the need for solutions that will improve the way in which consensus policies are developed, especially in relation to critical registrant, registry and registrar issues. It is desirable that these issues are addressed within an open and transparent working culture as well as an environment where data is collected in a confidential and anonymous manner.  Subsequently, this will encourage better-informed, fact-based policy development and decision-making.

 

It is important that all parties involved in GNSO decision-making recognise the benefit and value of relevant baseline data and metrics to the Policy Development Process, especially at the initial stages of scoping, understanding and describing a problem or issue. In order to ensure engagement of all parties in the new process, the ALAC supports the possible need to employ an independent third party in order to address any concerns relating to the collection, anonymization and aggregation of data. Fact-based deliberations and decision-making will enable the appropriate prioritization of critical issues based on tangible evidence rather than "gut feeling" or anecdotal examples.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00008.html

ALAC

Agreement

WG Response:

The WG thanks the ALAC for their confirmation of the group’s recommendations.

 

Action Taken:

None

  1.  

I am sending this on behalf of NCSG to endorse the comment sent by Amr Elsadr http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00007.html . can you please add the endorsement for the comment summary.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00008.html

NCSG

Agreement

WG Response:

The WG thanks the NCSG for their confirmation of the group’s recommendations and support of AE’s comments as noted in prior sections of the PCRT.

 

Action Taken:

None

 

 


Charter & Final Report Templates

#

Comment

Contributor

WG Response / Action Taken

Section Summary:

All comments were in “Agreement” with the WG’s recommendations.  One offered up a “New Idea” for a total of one action taken by the WG when considering the WG Final Report.

 

  1.  

Clearly define objectives at the start of the PDP process by stating deficiencies or desired improvements in the chartering stage.

Google strongly supports the recommendation to include Key Metric Considerations at the chartering stage of the PDP. Because a PDP may have multiple interrelated objectives and phases, we recommend that the the template provided in Annex A be augmented to clearly and specifically define the issue or prospective improvement associated with a metric. By way of example, for the recent Inter­Registrar Transfer Policy (IRTP) Part C recommendations, a reduction in the number of emergency reversals could have been used as a Key Metric tied to the goal of reversing domain name hijackings. Tying prospective improvements to tangible metrics will improve the definition of PDP objectives and associated success indicators and better target PDP­related work.

In addition to including criteria that define the success of the policy effort, the working group may wish to consider updating the template in Annex A to distinguish between Key Metrics that relate to measuring the effect of the policy and data elements that the working group itself may find useful during the policy development process.

 

See full comment:  http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00003.html

GOOG

Agreement   New Idea

WG Response:

The WG thanks GOOG for their confirmation of the group’s recommendation.  Much of this was discussed in WG deliberations noting a difference between metrics that are used to analyze an issue evolving to recommendations versus metrics that should be used when determining success.

 

Action Taken: [BC8]

Add as a suggestion or a hints and tips to set some sort of initial benchmark, noting early stage risk due to difficulty in predicting/forecasting targets that sometimes are in the abstract until seen in practice.

 

COMPLETED:   Updated Charter Template in Annex A with baseline and benchmark content. It will also be applied to Final Report template upon approval of the WG’s recommendations.

 

  1.  

Provide expanded recommendations for the use of data following the conclusion of a PDP.

We strongly support of Recommendation #4, which proposes to “include an additional recommendation that measures whether the policy change produced the intended effect.”

However, while guidelines for how data and metrics will be requested during the early phases of a PDP are captured in relative detail within the Initial Report and supported by available documentation, little information is provided about the processes for using data and metrics to support policy making following the conclusion of a PDP.

We recommend that, at minimum, studies be performed for the indicators set forth in a Charter’s Key Metric Considerations to allow assessment of whether a policy process achieved its intended aims and, where deficiencies exist, attempt to isolate the gaps or flaws in the PDP process that produced them. Data garnered from such studies could be used to improve future PDP work both substantively, by identifying issues requiring future work, and procedurally, by forcing a critical look at elements of the PDP that may have produced undesirable, inadequate, or unintended outcomes.

See full comment:  http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00003.html

GOOG

Agreement

WG Response:

The WG thanks GOOG for their confirmation of the group’s recommendation. Continuous improvement during the pilot effort and beyond is implied into this process. Refined distinction as noted in the prior comment should satisfy this distinction.

 

Action Taken:

None – Refer to Action in prior comment.

  1.  

The BC supports the recommendation to include Key Metric Considerations at the chartering stage of the PDP.  As a PDP may have numerous objectives and phases, we recommend that the chart provided in Annex A be modified to clearly and specifically define the issue or prospective improvement associated with a particular data point.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00004.html

BC

Agreement  

WG Response:

The WG thanks the BC for their confirmation of the group’s recommendation.

 

Action Taken:

None

  1.  

The ISPCP supports the creation of WG work product templates including updates to the GNSO’s Operating Procedures and Working Group Guidelines (WGG), as well as an update to the Charter template to assist drafting teams in determining key metrics to be used to meet the goals of resolving issues.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00005.html

ISPCP

Agreement

WG Response:

The WG thanks the ISPCP for their confirmation of the group’s recommendation.

 

Action Taken:

None

  1.  

In section 5.3.5.1, the third bullet observes that “For instance, the charter template could require that WGs identify a set of baseline data that should be captured to allow for the community to determine if a set of recommendations was effective or not”. It is important to note here that collection of data and analysing it quantitatively cannot be the sole determinant of whether or not a set of policy recommendations is effective. Qualitative research methods plays an important role in informing a discussion during policy development, and may very well also play a role in measuring its success post-implementation.

Although addressing the means by which qualitative research methods may be useful to the GNSO in policy development is not strictly within the scope of this working group, citing the advantages of using quantitative analysis should not be expressed as an absolute determinant of the success or failure of a policy recommendation. Qualitative methods of research often uncover compelling considerations to be taken account of that may not hold any statistical significance in quantitative analysis. The redline text in section 9 of the PDP manual suggested by the DMPM working group takes this into account nicely. This should also be reflected in the relevant parts of the working group final report and recommendations.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00007.html

AE

Agreement

WG Response:

The WG thanks AE for his confirmation of the group’s recommendation.  The WG deliberated extensively the differences of qualitative vs quantitative making note that much of the current PDP is qualitative based nor is this WG suggesting that it be done away with.  This notion was mentioned in a prior submission and action taken there.

 

Action Taken:

None

  1.  

The ALAC supports the revision of the templates for the Issue Report, Charter and Final Report to update earlier WG guidelines and also the development of a decision tree. These changes will help determine the best avenue to request additional data and metrics. The development of a Policy Development Process Manual, which will include the "Metrics Request Tree and Form", will ensure consistency of both process and practice.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00008.html

ALAC

Agreement

WG Response:

The WG thanks ALAC for their confirmation of the group’s recommendation.  The WG discussed how these components of the WGGs can be refined as lessons are learned from the pilot study.

 

Action Taken:

None

 


Pilot Effort

#

Comment

Contributor

WG Response / Action Taken

Section Summary:

All comments were in “Agreement” with the WG’s recommendations.  No “New Ideas”  were present but aa total of one action taken by the WG when considering the WG Final Report.

 

  1.  

Rescope the proposed pilot study to ensure that it advances real community objectives, while minimizing associated costs.

While we support the general concept of a pilot study to observe the application of data to ongoing policy processes, further scoping work is required to ensure that the effort expended advances real GNSO community objectives, as well as to minimize cost. To these ends, we propose two specific modifications to the proposed pilot effort. First, to the extent possible, the pilot study should focus on ongoing efforts that can be supported by data that is publicly available or that is readily accessible at a low cost. This change to the pilot study would minimize costs and delays associated with appointing a third party provider and procuring data, while still providing a sound early assessment of how data could be used to support policy making. Second, we recommend that the pilot study focus on efforts that are occurring at the GNSO level, as opposed to requests that come from a particular Supporting Organization or Advisory Committee, to ensure that additional resources leveraged in data collection and analysis are supporting projects and initiatives for which there is general support within the GNSO.

See full comment:  http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00003.html

GOOG

Agreement

WG Response:

The WG thanks GOOG for their confirmation of the group’s recommendation.  The WG noted similarity from a prior comment and action. Forms of scrutiny should be a part of the request process as to avoid excessive costs and beyond request scope/objective.  The WG envisioned for this pilot effort was that a group, an advocate of getting a policy development process started on something, use of data would help to bolster or mitigate their claim.  The idea is to prevent going down a path for which there wasn't a significant need (in that need to be revealed by data or lack of need to be revealed by data).

 

Action Taken: [BC9]

See prior action to create a listing of publicly available and third party sources for data/metrics in addition to logic to scrutinize requests within the decision tree.

 

COMPLETED:   Created a hints & tips page in Annex C that included public data/metrics resources.

 

  1.  

We believe that this particular issue is vitally important to the future of ICANN, to determining accountability and to encourage responsible decision making. We support a process to better inform fact-based policy development and decision making through the sharing of metrics and data.  The ISPCP enthusiastically supports the initiation of a pilot effort whereby the GNSO community at the early stages of the policy process can submit tactical sized requests for data and metrics to assist in validation of issues or to better inform policy deliberations. We would see value in that pilot and be happy to engage in it and provide actionable feedback to it.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00005.html

ISPCP

Agreement

WG Response:

The WG thanks the ISPCP for their confirmation of the group’s recommendation.

 

Action Taken:

None

  1.  

The ALAC supports the introduction of a "pilot" where working groups will be able to submit proposals or ideas whereby the collection and assessment of fact-based data and metrics can become the basis for the initial identification and analysis of issues and/or problems. We also support the view that any funding required to implement the pilot should be considered an investment in the improvement of the policy process rather than a cost against budget.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00008.html

ALAC

Agreement

WG Response:

The WG thanks the ALAC for their confirmation of the group’s recommendation.

 

Action Taken:

None


Metrics Request Form

#

Comment

Contributor

WG Response / Action Taken

Section Summary:

One comment contained a “concern” resulting in one action taken by the WG when considering the Final Report.

 

  1.  

The suggested language of section 4.5 of the GNSO operating procedures detailing the “Working Group Metrics Request Form” only indicates the procedures for requesting data/metrics. There is no indication in the proposed changes to the operating procedures or the metrics request tree that prior to aggregation of data, there is any requirement for the chartering organization (GNSO Council) to approve the request. Considering the potential cost of both time and funds (at the issue scoping phase or during the PDP working group phase), it may be worthwhile to consider whether or not the chartering organization should play a role in determining the extent to which the “Issue to be solved” in the working group metrics request form warrants such delays/costs. Clarification on a process to approve a submitted working group metrics request form should ideally be included in the DMPM working group’s final report.

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00007.html

AE

Concerns  

WG Response:

There is a risk with a subjective refusal to collect data to objectively identify whether there is a problem. A real-life situation could be that the requester makes a hypothetical that the data will show X and then if it does the GNSO Council could maybe decide whether or not that that outcome was worth the cost or something to that effect. Was that problem impactful enough?  Lessons learned out of that pilot process and the outcomes should be reviewed.

 

Action Taken: [BC10]

Highlight the decision tree make clear that requests will require scrutiny by GNSO Council and/or staff once it's submitted.

 

COMPLETED:   Added scrutiny logic to Hints & Tips page in Annex C and updated Decision Tree in Annex B with added assessment logic.

 

 


Early Outreach

#

Comment

Contributor

WG Response / Action Taken

Section Summary:

A single comment was in “Agreement” with the WG’s recommendation that resulted in one action taken by the WG when considering the WG Final Report.

 

  1.  

Establishing a framework for distributing information through early outreach to other SOs, ACs and related organisations will facilitate broader qualitative input and support a culture of collaboration between our organisations. This will not only contribute to continuous improvement being fully integrated into the Policy Development Process but also encourage the potential of an Open Data culture across ICANN

 

See full comment: http://forum.icann.org/lists/comments-data-metrics-29jul15/msg00008.html

ALAC

Agreement

WG Response:

The WG thanks the ALAC for their confirmation of the group’s recommendation.  The WG noted some holders of data that are a little concerned about the terminology of “open data” across ICANN. Caution should be used for the terms we use since a lot of the data is considered to be commercially important. Caution against making a recommendation for open data because that will make some of the people from whom we would be requesting data nervous.  The anonymization of data doesn’t necessarily remove the commercial utility of data. It removes the privacy implications but, if people are making valuable data available even in an anonymized form can still have commercial value.

 

Action Taken:

Review final report for use of “Open” data. [BC11]

 

COMPLETED:   Searched document for use or reference of Open Data and none was found.

 

 

 

 


[BC1] Action Required

[BC2] Action Required

[BC3] Action Required

[BC4] Action Required

[BC5] Action Required

[BC6] Action Required

[BC7] Action Required

[BC8] Action Required

[BC9] Action Required

[BC10] Action Required

[BC11] Action Required.