General Observations

The Multistakeholder Strategy and Strategic Initiatives Department developed a survey which was sent to the GNSO Review Working Party, as well as current and former GNSO Council members.  The objective of the survey was to gather feedback after the Review work concluded.  The survey covers several areas – Independent Examiner, ICANN Organization support and Working Party work.  Feedback will inform future reviews and will be used to continue process improvements. 

Fourteen people responded to the survey.  Responses to each question of the survey were not mandatory.  Approximately 70% (ten) of the responses were from Working Party members while 30% (four) were from Council members or observers; 84% (twelve) of respondents felt they were either moderately active or very active in the GNSO Review (conducted between 2014-2016).

Cells in yellow highlight the answers with the highest number of responses.  All comments appear as submitted, without editing.

Questions Pertaining to Westlake Governance

Consolidated Responses

Respondents acknowledge that Westlake Governance was experienced in reviews and was knowledgeable about ICANN and the GNSO.  They showed an understanding of the assignment and flexibility in meeting the needs of the Working Party.  However, respondents indicate that Westlake Governance was not effective in engaging the GNSO community and did not perform their duties in line with the scope of work as directed by the Organizational Effectiveness Committee and approved by the Working Party.  Respondents do not recommend Westlake Governance be hired for future Reviews.  Below are the consolidated responses from the survey.

 

1  (least)

2

3

4

5 (most)

During the course of the Review, how knowledgeable was Westlake Governance on ICANN and the GNSO?

7.69%

46.15%

23.08%

23.08%

0.00%

How effective was Westlake Governance at tailoring the review methodologies (surveys, interviews, observations of meetings, etc.) to achieve the objectives provided by the Organizational Effectiveness Committee?

38.46%

23.08%

7.69%

23.08%

7.69%

How effective was Westlake Governance at engaging with the GNSO Community to gather feedback from a broad and diverse group of people?

15.38%

46.15%

7.69%

30.77%

0.00%

We know that there were community concerns about a few of the recommendations.  Notwithstanding those recommendations, how satisfied were you with the majority of the recommendations in Westlake Governance’s Final Report?

30.77%

23.08%

15.38%

15.38%

15.38%

 

Yes

No

Understand the assignment provided in the scope of work as directed by the Organizational Effectiveness Committee and approved by the Working Party?

53.85%

46.15%

Perform its duties in line with the scope of work as directed by the Organizational Effectiveness Committee and approved by the Working Party?

38.46%

61.54%

Routinely attend and participate in scheduled Working Party calls?

100.00%

0.00%

Provide sufficient explanation and justification for its findings and recommendations?

23.08%

76.92%

Show flexibility and responsiveness in meeting the Working Party’s requests and timelines?

53.85%

46.15%

Demonstrate experience in conducting broadly similar reviews?

38.46%

61.54%

Would you hire Westlake Governance again for future Reviews within ICANN?

30.77%

69.23%

The survey allowed respondents the opportunity to provide additional comments about Westlake.  Below are the comments received from respondents.  [All comments appear as submitted, without editing.]

Westlake did not understand the inner working, interests, characters and machinations that make the GNSO what it is at all. They insisted on dealing with the GNSO as a "normal and rational" organization, which it certainly is not. They where completely out of their depth at every stage. I did feel sorry for them.

using questionable methods such quoting as basis without checking  them against facts like stats, online public info etc independent examiner should avoid taking or advocating for positions but provides elements for thinkings and objective recommendations.

There was no initiative from Westlake to tackle structural reform despite Board calls to do so. There was no attempt to resolve conflicting messages between the Board and the OEC on this issue. The consultant took the easy option not the option that was best for ICANN.

Westlake was very responsive to the Working Party throughout most of the process.  It was only at the end where they made some recommendations that seemed to be unresponsive and that was minimal.

Interview sample was totally biased.  Response to feedback was rude.  Too many iCANN staff had input to interviews, whereas GNSO councillors did not.  Suggests staff manipulation, which might not be the fault of the contractor.  Survey was entirely inadequate and biased in my view and in the view of academic statisticians and experienced researchers in my constituency.  Our input was mostly ignored. I would fight very hard any attempt to hire them again.

One of the biggest areas of concern was that Westlake blindsided the Working Party with a last minute controversial recommendation.  It would be helpful to coordinate such comments in the future to avoid community backlash that then undermines all of the good work and generally accepted recommendations.

I did have a few concerns with Westlake Governance's performance as the independent examiner. First, I do not believe they did as well as they could to get a representative sample of the community in the 360 review and general interviews. I do note that they did make an effort to interview me, but I was not able to make the time to be interviewed, so I certainly do not hold them to blame for this. Still..., there were some topics (including the review of the NCSG and its constituencies) where they either did not interview an appropriately diverse number of informants. On occasion, they apparently decided to ignore feedback received by members of the NCSG, particularly the NUCU. This is easy to determine, because a great deal of admirable work was done by staff to accumulate feedback received by Westlake, especially during public ICANN meetings. Many of the responses provided by NCUC members seemed to have not found their way into considerations in Westlakes' final report. Some of these recommendations, such as the ones concerning constituencies in the GNSO, had a potential far reaching impact on the GNSO in general. Subsequently, one of those recommendations, which was only published in the final report and not directly discussed with the working party was not adopted by the working party, the GNSO Council or the ICANN Board. In fact, the unanimous recommendation from the working party members was to not adopt this recommendation. I believe this reflects poorly on Westlake.

Another concern I had with their methodology was in how they conducted their qualitative analysis of interviews. In my experience, when an organization as complex as ICANN is being studied and qualitatively analyzed, findings and conclusions need to be justified, not by simple observation and opinion..., but by grounding these findings in established theoretical frameworks. Westlake did not attempt to do this in any way, which to me, severely decreased the credibility of their rationale and conclusions.

Questions Pertaining to ICANN Organization

Respondents acknowledged that there were issues with the Review but were hesitant to blame ICANN Organization/staff.  They noted that individuals supporting the review seemed responsive and effective at ensuring the review was conducted in an independent and objective manner and that they properly communicated key developments to the community and Organizational Effectiveness Committee.  Overall, more than 75% of respondents were satisfied with support by ICANN  Organization.  Below are the consolidated responses from the survey.

 

1 (least)

2

3

4

5 (most)

How effective was ICANN Staff at ensuring the Review was conducted in an independent and objective manner?

7.69%

15.38%

15.38%

15.38%

46.15%

How effectively did ICANN Staff communicate key developments pertaining to the Review to the community and the OEC?

0.00%

0.00%

15.38%

53.85%

30.77%

How responsive was ICANN Staff to the concerns of the GNSO community?

7.69%

15.38%

30.77%

15.38%

30.77%

How satisfied were you with support and guidance provided by ICANN Staff during the GNSO Review?

0.00%

23.08%

15.38%

23.08%

38.46%

The survey allowed respondents the opportunity to provide additional comments about Westlake.  Below are the comments received from respondents.  [All comments appear as submitted, without editing.]

The working group should have stopped the review as soon as it became clear that Westlake was not capable of doing the job. I was not the responsibility of staff to do so.

review status updates by means of glossy brochures seem to be extravagant. Content is most important!

It was difficult to determine what was going on here.  I hesitate to blame staff for direction that may have been coming from elsewhere, but this was a very deeply flawed review.  Committee chair was apparently unaware the group had been hired to do the ALAC review previously, and that there had been great unhappiness with that process.  So why did we hire them again???  Also what did it cost??? many questions linger....but I am very reluctant to blame staff.

I think Staff to an incredible job of coordinating the review and responding to community concerns, including significantly extending the time needed to ensure the best possible outcome.

ICANN staff support was exemplary as far as I am concerned. I also note that the OEC was extremely patient and supportive of the GNSO community's concerns and perceived requirements in management of Westlake's recommendations. Unfortunately, I do not believe the OEC's predecessor (the SIC) did as well a job in taking concerns into consideration when the design of the study was done in the early stages of the review.

Questions Pertaining to the Working Party

Approximately 92% of the respondents agreed that the Working Party was effective at serving as a liaison between ICANN Organization, the Organizational Effectiveness Committee, and Westlake Governance.  The responses also note that the Working Party had a productive effect on the ability of the community to provide input into the Review and that subsequent reviews should employ the same working party model.  Below are the consolidated responses from the survey.

 

1 (least effective)

2

3

4

5 (most effective)

How effective was the GNSO Review Working Party at serving as a liaison between Staff, the Organizational Effectiveness Committee and the Independent Examiner

0.00%

7.69%

53.85%

15.38%

23.08%

 

Yes

No

Did the work of the Working Party have a productive effect on the ability of the community to provide input to the Review?

84.62%

 

15.38%

The construct of a Review Working Party was piloted with the GNSO Review.  Would you recommend that a Review Working Party be adopted for future organizational reviews (for example, At-Large, RSSAC, etc.)?

100%

0%

Were the recommendations on implementation by the Working Party helpful to the GNSO?

91.67%

8.33%

Additional Comments

What improvements, if any, would you suggest to make the construct of the Review Working Party more effective?  [All comments appear as submitted, without editing.]

with regard to 12 above. It should be the model, BUT the Working party needs to act in a professional manor. Part of the WP had only the goal to prevent critical recommendations from seeing the light of day. I was verbally atact by WP members several times during the process. It was a very bruising affair and even more bruising was to see that the tactic worked and the truth was deleted from the recommendations at an early stage.

A pre review meeting with the OEC,

Get broader representation.

Scope needs to be agreed by the Community, not dictated by ICANN Staff and its paid consultant.  More effort needs to be made to find truly independent consultants who understand the role of the community vs. the role of ICANN.

A discussion about potential trade-off between a coordination role and resposibility may be needed. The independent examiner shouldn't be given a chance to instrumentalize missing WP input as an argument for delay.

I think it worked well and provided for an outlet for comments that were not taken on board by the contractors.  Should not be necessary but they did an excellent job this time, and it was certainly necessary.

More Feed backs to community in simplified manner that constituencies members can understand the whole process

It would be helpful for the Working Party to have more input at the outset, particularly in working with the OEC and Staff to frame the methodologies and scope of the Review.  This became a contentious issue throughout the process and was often used as a reason to simply discount the work being done because other did not believe it was properly scoped or that the methodology was not correct.

Most of the working party members were not active, and I believe this was reflected in last minute concerns being raised by the community. I'm sure this wasn't helpful to staff or even the independent examiner. More attention from all stakeholders subject to he review is important in future reviews. I was a member of the working party for a short time, then asked to be replaced (because of an increasing workload both in and out of ICANN). Towards the end of the review process, I felt a need to re-engage, seeing that participation by working party members was exceedingly poor.

If you have any additional comments you would like to share about the GNSO Review Working Party, please provide your comments here.  [All comments appear as submitted, without editing.]

In the end, it was a big waste of time and great source of disillusionment and pain.

Jen did a great job leag the working party and working with staff and Westlake.

The scope was far too narrow, for no good reason, leaving us still with an ineffective GNSO (except for contracting parties) at least until next time...

Do it with current reviews please.

Please provide comments, if any, on how the Review was conducted (what worked well, suggestions for improvements, etc).  [All comments appear as submitted, without editing.]

A missed opportunity for deeper GNSO reform despite many calls to not miss the opportunity. A disappointment.

Unfortunately, the review got very short shrift from the community due to the predetermined, too narrow scope, and due to all the community work going on with new gTLDs and the IANA transition.  This was unfortunate and cannot be allowed to happen again next time.

Don't hire these guys again.  Provide the assessment criteria for contracting to the   Review committee in advance.

In the future, I would suggest that a working party for any review be established prior to the review beginning. It would be helpful to consult a group such as a working party at earlier stages, such as defining the terms of reference of the study to be conducted. Engagement with the GNSO Council, in the case of the GNSO Review, may not have been enough at that early stage.

  • No labels