AT-LARGE GATEWAY

At-Large Website

ALAC

RALOs

At-Large Regional Policy Engagement Program (ARPEP)

At-Large Governance

At-Large Reports

Policy Comments & Advice

Working Groups

ICANN Meetings

At-Large Summit (ATLAS III)

At-Large Review Implementation Plan Development

ALAC and RALO Elections, Selections, and Appointments

At-Large Priority Activities - 2021

Comment Close
Date		Statement
Name 

Status

Assignee(s)

Call for
Comments Open		Call for
Comments
Close 		Vote OpenVote CloseDate of SubmissionStaff Contact and EmailStatement Number
 Initial Report on Data & Metrics for Policy MakingADOPTED 13Y, 0N, 0AMaureen Hilyard    23:59 UTC Steve Chan steve.chan@icann.orgAL-ALAC-ST-0915-02-00-EN

For information about this Public Comment, please click here 

Brief Overview

Purpose: This public comment proceeding seeks to obtain community input on the Initial Report from the GNSO's Working Group (WG) on possible recommendations for the use of Data and Metrics for Policy Making.  The WG's intent is to develop solutions that will improve the way in which consensus policies are developed and help ensure that the most critical registrant, registry, and registrar issues are addressed by evolving the culture to better informed, fact-based policy development and decision-making.

Next Steps: After close of the public comment period, the WG will review the submitted comments and update the report as agreed upon, thus producing the Final Report for submission to the GNSO Council.

Section I: Description, Explanation, and Purpose

The Data & Metrics for Policy Making Working Group (WG) is exploring opportunities to review standard methodologies of reporting and metrics that could better inform fact-based policy development and decision making; including how the community can collaborate with Contracted Parties and other service providers in the sharing of metrics and data. 

The WG's charter contains eight specific questions for the WG, derived from the Final Issue Report that was published in March 2013. The Charter questions cover issues including:

  • The question “which comes first, policy-process or definitive data describing the problem?” along with suggestions as to how data can be gathered when it has not yet been included in the reporting process.
  • How processes can be continuously improved, simplified and made more consistent for people wishing to either report a problem or learn about their options when their problem falls outside ICANN policy;
  • Principles that enhance metrics and data available to better inform the GNSO policy development process;
  • Improved understanding of the limits of ICANN policies regarding data measurement and tracking and other options to pursue if an issue is not covered by policies that gather data
  • Mechanisms whereby GNSO working groups can request information (both internal toICANN or external, including GNSO contracted parties) which support fact-based policy-making;
  • Mechanisms to ensure appropriate safeguards with regard to the confidentiality of certain types of information;
  • A framework for distributing information to the GNSO policy-making community with the intent of both informing those groups and providing the ongoing basis for identifying and correcting problem-reporting and data-collection problems;
  • Any changes needed to incorporate the processes described above into the ongoing Policy Development Process.

Opportunities discovered by the group include:

  • Improve the way in which Consensus Policies are developed and help ensure that the most critical registrant, registry, and registrar issues are addressed
  • Evolve culture to better informed, fact-based policy development and decision making
  • Base deliberations and decisions on tangible evidence as opposed to “gut feeling” or anecdotal evidence
  • Instill notion of continuous improvement to the policy process and effectiveness consensus policy implementations

Key Definitions:

  • Data: Individual facts, a set of values, statistics, or items of information
  • Metrics: A set of measurements that help quantify results, which allows for better determination of the level of success against a set of goals

The WG’s preliminary recommendations include:

  • Initiation of a pilot effort whereby the GNSO community at the early stages of the policy process (Issue Identification or Working Groups) can submit tactical sized requests for data and metrics to assist in validation of issues or to better inform policy deliberations
  • Creation of WG work product templates (Charter, Final Report) including updates to theGNSO’s Operating Procedures and Working Group Guidelines (WGG)
  • Update to the Charter template to assist drafting teams in determining key metrics to be used to meet the goals of resolving issues
  • Updates to the Final Report template with a standard but modifiable recommendation forcing a review of the policies implemented at a future date to determine if it met the stated intent
  • Creation of a Metrics Request Decision Tree and Request Form to formalize the request process; including inclusion into the WGG.

Following analysis of all public comments received, the WG will finalize its recommendations and prepare a Final Report for delivery to the GNSO Council, for its review and action.

Section II: Background

The 2010 Registration Abuse Policies Working Group (RAPWG) identified the Meta Issue: Uniformity of Reporting which it described as "need for more uniformity in the mechanisms to initiate, track, and analyze policy-violation reports."  The RAPWG recommended in its Final Report that "the GNSO and the larger ICANN community in general, create and support uniform problem-reporting and report-tracking processes."

The GNSO Council recommended the creation of an Issue Report to further research metrics and reporting needs in hopes to improve the policy development process.  The report created by ICANN staff outlined accomplishments regarding reporting and metrics by the Contractual Compliance function and it also reviewed other reporting sources that may be of relevance.  TheGNSO Council subsequently adopted the recommendation to form this non-PDP Working Group tasked with exploring opportunities for developing reporting and metrics processes and/or appropriate standardized methodologies that could better inform fact-based policy development and decision making.

The WG has developed preliminary recommendations for a number of the charter questions it was tasked to address, which are described in its Initial Report [PDF, 395 KB].  Those recommendations that contain suggested edits to the GNSO's Operating Procedures are also included in a redline version [PDF, 768 KB] should the WG's recommendations be adopted by the GNSO Council.

Section III: Relevant Resources

Section IV: Additional Information

Note that a few of the WG’s recommendations suggest updates to the GNSO’s Operating Guidelines; specifically Annex 1 "GNSO Working Group Guidelines" and Annex 2 "Policy Development Process Manual”. The recommendations within the Initial Report include the proposed changes or creation of new material. A redline version of GNSO’s Operating Guidelines are included to highlight the possible updates to this document should the GNSOCouncil adopt the WG’s recommendations.

Section V: Reports

Staff Contact

Steve Chan
steve.chan@icann.org

 

FINAL VERSION TO BE SUBMITTED IF RATIFIED

Please download the PDF document below. 



FINAL DRAFT VERSION TO BE VOTED UPON BY THE ALAC

ALAC Statement on the Initial Report on Data & Metrics for Policy Making

The ALAC welcomes the opportunity to provide community input into the Initial Report from the GNSO's Working Group with regards to possible recommendations for the use of Data and Metrics for Policy Making.

The ALAC appreciates the need for solutions that will improve the way in which consensus policies are developed, especially in relation to critical registrant, registry and registrar issues. It is desirable that these issues are addressed within an open and transparent working culture as well as an environment where data is collected in a confidential and anonymous manner. Subsequently, this will encourage better-informed, fact-based policy development and decision-making.

It is important that all parties involved in GNSO decision-making recognise the benefit and value of relevant baseline data and metrics to the Policy Development Process, especially at the initial stages of scoping, understanding and describing a problem or issue. In order to ensure engagement of all parties in the new process, the ALAC supports the possible need to employ an independent third party in order to address any concerns relating to the collection, anonymization and aggregation of data. Fact-based deliberations and decision-making will enable the appropriate prioritization of critical issues based on tangible evidence rather than "gut feeling" or anecdotal examples.

The ALAC supports the introduction of a "pilot" where working groups will be able to submit proposals or ideas whereby the collection and assessment of fact-based data and metrics can become the basis for the initial identification and analysis of issues and/or problems. We also support the view that any funding required to implement the pilot should be considered an investment in the improvement of the policy process rather than a cost against budget.

The ALAC supports the revision of the templates for the Issue Report, Charter and Final Report to update earlier WG guidelines and also the development of a decision tree. These changes will help determine the best avenue to request additional data and metrics. The development of a Policy Development Process Manual, which will include the "Metrics Request Tree and Form", will ensure consistency of both process and practice.

Establishing a framework for distributing information through early outreach to other SOs, ACs and related organisations will facilitate broader qualitative input and support a culture of collaboration between our organisations. This will not only contribute to continuous improvement being fully integrated into the Policy Development Process but also encourage the potential of an Open Data culture across ICANN.

 

FIRST DRAFT SUBMITTED

The ALAC welcomes being given the opportunity to provide community input into the Initial Report from the GNSO's Working Group with regards to possible recommendations for the use of Data and Metrics for Policy Making.

The ALAC appreciates the need for solutions that will improve the way in which consensus policies are developed, especially in relation to critical registrant, registry and registrar issues. It is desirable that these are addressed within a working culture that is open and transparent; collects data in an environment that ensures confidentiality and anonymity; and subsequently encourages better informed, fact-based policy development and decision-making.

All parties involved in GNSO decision-making must recognise the benefit and value of relevant baseline data and metrics to the policy development process, especially at the initial stages of scoping, understanding and describing a problem or issue. Because of the importance of engaging all parties into the new process,the ALAC supports the possible need to employ an independent third party in order to address any concerns relating to the collection, anonymization and aggregation of data.  Fact-based deliberations and decision-making will enable critical issues to now be prioritised appropriately based on tangible evidence rather than "gut feeling" or anecdotal examples.

The ALAC supports the introduction of a "pilot"  where working groups will be able to submit proposals or ideas whereby the impact of fact-based data and metrics collection can become the basis for the initial identification and analysis of issues and/or problems.  We also support the view that any funding required to implement the pilot should be considered an investment in the improvement of the policy process rather than a cost against budget.

The ALAC supports the revision of the templates for the Issue Report, Charter and Final Report to update earlier WG guidelines and also the development of a decision tree to help to determine the best avenue to request additional data and metrics. The development of a Policy Development Process Manual, which will include the "Metrics Request Tree and Form", will ensure consistency of both process and practice.

Establishing a framework for distributing information through early outreach to other SOs and ACs  and other related organisations, will facilitate broader qualitative input as well as support a culture of collaboration between our organisations. This will not only contribute to continuous improvement being fully integrated into the policy development process but also encourages the potential of an Open Data culture across ICANN.

  • No labels

13 Comments

  1. Maureen Hilyard

    INTRODUCTION

    Following on from the Issue Report produced by ICANN Staff regarding metrics and reporting by the Compliance Function and other sources, the GNSO Council formed a non-PDP Working Group to further explore opportunities for developing reporting and metrics processes and/or appropriate standardized methodologies that could better inform fact-based policy development and decision making.  

     From <https://community.icann.org/pages/viewpage.action?pageId=41888667>

    SUMMARY

    The Data and Metrics for Policy Making Working Group (DMPMWG) has provided a very comprehensive report with recommendations aimed at encouraging a more data and metrics driven culture when making decisions related to the generic name space and the ICANN community.

                  The WG addressed eight Charter questions and from their deliberations, they developed seven recommendations that require changes to their current WG structure, the Charter and their overall decision-making processes. These recommendations contribute to a seven phase policy development process requiring the following changes:

    1.       Issue identification - a Data and Metrics Pilot Request Project

    2.       Issue Scoping – data collection; and a Metrics Request Decision Tree and Metrics Request template for formal WG requests.

    3.       Initiation - an update to the current Charter Template.

    4.       Working Group - outreach to organisations external to ICANN’s SOs and ACs during the public consultation process; the inclusion of metrics and an impact assessment; and the Pilot Request Project

    5.       Council Deliberations - unchanged

    6.       Board vote – unchanged

    7.     Implementation – a post-implementation assessment of the impact of the policy

    Some thoughts:

    ·       The collection of gathering baseline data to better inform and understand the issue/problem depending on its complexity, may simply require the services of staff. However, depending on how many complex issues the GNSO community propose – even for the pilot project - the cost of gathering the optimal level of data to implement the recommendations may require more resources (staff or funds) than is anticipated on the final page of the recommendations.

    ·       A recommendation has been made about expanding the outreach to external organisations during the public consultation process. Although the recommendation has been made to include a statement addressing this into the GNSO’s PDP Manual, should public consultation not be included as a WG deliverable in the Charter, alongside the metrics and impact assessment recommendations for the WG, especially as the consultation process and feedback is expected to be acknowledged in the final report?

    ·       I have a problem with Charter Question B where the recommendation was to pass any complaints that fall “outside of ICANN policy” to ICANN Contractual Compliance. Far be it from me to suggest that the gNSO ever receives any complaints, but what if a problem arises that is of sufficient “grey area” enough to offer the potential of another policy development request? What processes does the gNSO have in place to address this type of problem and to ensure that processes are “continuously improved, simplified and made more consistent”? Is this where the metrics request decision kicks in? 

  2. Maureen Hilyard

    COMMENTS FROM CHERYL AND ALAN AT THE ALT MEETING (12-08-2015)

    ALAN GREENBERG: Thank you Maureen. A couple of comments on the history of this. There have been discussions within PDPs, particularly by contracted parties who’ve been objecting to doing things, that at times we attempt to make policy based on hearsay and feelings in your stomach, and not on real data. There have also been statements made that in some cases the only real data is held by the contracted parties, who refuse to give it up. How convenient. When this original project started, there was a tone in it that they were supposed to be looking at things that we should be asking contracted parties to keep track of, so that we could have data in the policy making process. Along the way, the tone has changed somewhat. I haven’t looked at the current document, so I’m not quite sure where we are today, but I think that’s something to keep in mind. It’s fine to say we should only make policy based on real data, but we need access to the real data that we’re going to make the policy on. That’s something I think we have to look very carefully on when we look at this particular document. Anyone else with a comment on this? No? Cheryl, go ahead.

    CHERYL LANGDON-ORR: Just briefly - as you probably know, I’ve had more attendance than the Co Chairs in this particular Working Party - I’m quite comfortable with the tone and where we are. There’s a plan for not only a pilot, but more importantly to build into the normal scheme of events as a basis from the pilot, appropriate funding models, so there can be a third-party acquisition, as required, and anonymized acquisition, as required, of  data. What we did during this process was work quite successfully to… Ameliorate is one way of putting it, some of the frequently forwarded issues in terms of commercial and in-confidence materials. There really aren’t too many places in the shadows to hide, providing the data is required. We also don’t want to just do these things for the sheer joy of doing them. I’m quite happy with where we’re up to, and I do know Maureen and I should have had a conversation about this, and she’s just launching into her [unclear 00:13:40] and I’m just surfacing from it. Or maybe we’re both surfacing from it, so we might actually get together. I hear where you’re coming from Alan, because you and I suffered through a few of those interesting conversations. But I’m actually relatively, if not totally, comforted that this will work. But more importantly we can iron out any kinks during the funded pilot of five exercises of cases we’ll be doing. By the way, Alan, we did use some of your more memorable reference points on this as case studies during our work.

    ALAN GREENBERG: Are you talking about a contractual requirement that will force contracted parties to participate?

    CHERYL LANGDON-ORR: Not at this stage, but we’re talking about a development of a set of expectations whereby to not play ball - and we’ve got a couple of people particularly keen to play ball - they would be very much out of step if they didn’t. But they’d also have the ability to take third-party data on board, because it’s a funded model, and of course the funded model makes a huge difference.

  3. Maureen Hilyard

    Hi everyone — my first draft,,.. sorry this is late.. I'd be happy to make any amendments, etc. Please feel free to make suggestions.

     

    The ALAC welcomes being given the opportunity to provide community input into the Initial Report from the GNSO's Working Group with regards to possible recommendations for the use of Data and Metrics for Policy Making.

     The ALAC appreciates the need for solutions that will improve the way in which consensus policies are developed, especially in relation to critical registrant, registry and registrar issues. It is desirable that these are addressed within a working culture that is open and transparent;  collects data in an environment that ensures confidentiality and anonymity; and subsequently encourages better informed, fact-based policy development and decision-making.

     All parties involved in GNSO decision-making must recognise the benefit and value of relevant baseline data and metrics to the policy development process, especially at the initial stages of scoping, understanding and describing a problem or issue. Because of the importance of engaging all parties into the new process,the ALAC supports the possible need to employ an independent third party in order to address any concerns relating to the collection, anonymization and aggregation of data.  Fact-based deliberations and decision-making will enable critical issues to now be prioritised appropriately based on tangible evidence rather than "gut feeling" or anecdotal examples.

     The ALAC supports the introduction of a "pilot"  where working groups will be able to submit proposals or ideas whereby the impact of fact-based data and metrics collection can become the basis for the initial identification and analysis of issues and/or problems.  We also support the view that any funding required to implement the pilot should be considered an investment in the improvement of the policy process rather than a cost against budget.

     The ALAC supports the revision of the templates for the Issue Report, Charter and Final Report to update earlier WG guidelines and also the development of a decision tree to help to determine the best avenue to request additional data and metrics. The development of a Policy Development Process Manual, which will include the "Metrics Request Tree and Form", will ensure consistency of both process and practice.

     Establishing a framework for distributing information through early outreach to other SOs and ACs  and other related organisations, will facilitate broader qualitative input as well as support a  culture of collaboration between our organisations.  This will not only contribute to continuous improvement being fully integrated into the policy development process but also encourages the potential of an Open Data culture across ICANN.

     

  4. Cheryl Langdon-Orr

    Short and to the point, this Draft covers all the issues I would think of,  so has my support...  Thanks Maureen 

  5. Sebastien Bachollet

    I just and only read the proposed statement written by Maureen and it seems a good way to go.
    I just want to suggest that we add some reference to possibly turn ICANN (all it's components) in an OPEN DATA culture.
    Thanks Maureen 

    1. Ariel Liang

      Sebastien Bachollet and Maureen Hilyard, perhaps ICANN's KPI dashboard will interest you: https://www.icann.org/progress. This beta version was just launched this past Friday, August 28. 

    2. Carlton Samuels

      I strongly support Sebastien's proposal that the ALAC recommend that ICANN acculturate an Open Data culture

       

      -Carlton

      1. Dev Anand Teelucksingh

        Just to fully support Sebastien's comment which is in line with ATLAS II Recommendation 39 - "ICANN should encourage “open data” best practices that foster re-use of the information by any third party."

         

         

  6. Maureen Hilyard

    Thank you Sebastian and Cheryl. I agree that the changes that the GNSO makes to their processes should be noted as positive potentials for the development of improvements in the communication processes of other SOs and ACs and perhaps elsewhere in ICANN. I have adapted my original statement above to include your suggestion as well as made other slight amendments - underlined to indicate the changes.

    1. Ariel Liang

      Hi Maureen. I moved your latest version (I saw you modified it in the comment section) under 'First Draft Submitted' for easy reference. 

  7. Olivier Crepin-Leblond

    Jonathan Zuck has provided a full update and explanation of the recommendations on today's GNSO Council call.

    Slides: http://gnso.icann.org/en/correspondence/presentation-dmpm-03sep15-en.pdf

    Recommended for any ALAC member who wishes to catch up quickly on the contents of the report.

     

  8. Maureen Hilyard

    Jonathan's ppt is a great summary. 

  9. Holly Raiche

    Agree with Maureen's draft and also agree that the power point is a really good summary