Design Considerations

Focus of Self-Assessment: Operational Effectiveness

My current thinking is that the most important dimension to understand is how EFFECTIVE the WG was in its operations, behaviors, and protocols leading (or not) to the achievement of its mission. In particular, the WG Guidelines and Charter are important tools developed to help WGs become more effective; however, they are not ends in themselves. As a result, I do not think we should ask questions about how well organized or even useful these documents are to WG members. One reason is that a team of intelligent and committed ICANN volunteers (WG-WT) spent the better part of a year going over every paragraph to ensure that the end products were appropriately organized, thorough, clear, and useful. Apart from evaluating the broader set of tools/support that a WG is eligible to receive, we should depend upon WG members to single out any particular document(s) that are especially useful or in need of revision. A person who has participated in many WGs may never need to consult the Guidelines having learned through experience how to be a valued contributor. In my view, it is more important to know if the WG itself was effective in its forming, storming, norming, and producing stages. If any critical element of the WG process is perceived to be ineffective and the survey explanations/reasons are insufficient for proper diagnosis, the Chartering Organization can and probably should investigate further arranging follow-up interviews with WG members. 

Should Self-Assessments be completed by each WG Member individually or collectively by the entire team?

Individual Member (separately)

•  Possible to obtain opinions and viewpoints (anonymously) that might never be admitted or conceded by the entire group.
•  More efficient to ask each individual for impressions and perspectives than to ask a group to coalesce around a single representation. A team might consider such a self-assessment task to be more challenging and difficult (obtaining consensus) than its original mission.
•  It may be more challenging to gain an overall impression of the WG's experience by examining individual perspectives. 

Entire Team (collectively)

•  A group characterization, if achieved through consensus, may be the most balanced and informative because it will have been discussed openly by all members. 
•  If a WG was dominated by one or two individuals or labored under weak leadership, that type of information might be suppressed in a collective assessment unless individuals are permitted to submit minority reports. Similarly, if something negative occurred that could represent a learning opportunity or process improvement, a team may be reluctant to admit shortcomings, failures, or "air its laundry" in public. 

Demographic Information and Anonymity

I recommend that the following data fields be captured for each WG Member in order to ensure that (1) all participants' views have been registered, and (2) that no spurious (or duplicate) entries have been entered intentionally or accidentally. Respondents should be informed that anonymity will be safeguarded: 

Demographic Fields:

• Name
• Organization (Drop-Down List)
• Email Address
• Predominant Role on the WG
  • Leader (e.g., Chair, Co-Chair, Vice-Chair, or other elected position/office)
  • Contributing Member
  • Liaison
  • Observer
  • Advisor/Consultant
  • Support (e.g., secretary, technical, administrative)
  • Other (please describe)
• Engagement Experience (Years Working with ICANN; Drop-Down List)
  
  • Less than 1 year
  • 1 - 2 years
  • 2 - 4 years
  • 4 - 6 years
  • 6 - 8 years
  • More than 8 years
• Engagement Intensity (Drop-Down List; Average Hours/Month)
  
  • Less than 10 hours
  • 10 - 20 hours
  • 20 - 40 hours
  • 40 - 60 hours
  • 60 - 80 hours
  • More than 80 hours

Size, Complexity, and Length (Time)

There are no hard and fast rules concerning survey length; however, many experts recommend that online questionnaires and surveys be structured in such a way that they can be completed in 30 minutes or less by most respondents. Once a draft instrument has been designed, it will be important, of course, to test it to be sure that complexity and length are not burdensome. 

Methodology

One possibility is to utilize the ICANN Wiki survey template as the host for each WG to complete the self-assessment.

• Pros:
  
  • WG Members have login accounts and utilize the Wiki for collaboration and work activities
  • Has been used successfully for the GNSO Statement of Interest (SOI) survey
  • General user familiarity with tools, look/feel, and Wiki operations
• Cons:
  • It may be more difficult to protect anonymity on the Wiki platform due to the number of people who hold Admin authorizations and can view restricted data. 

Another option is to utilize an online tool such as QuestionPro or SurveyMonkey.

• Pros:
  
  • ICANN has accounts with both services
  • QuestionPro was used successfully for the Stakeholder Satisfaction Survey (2011) and the Globalization Survey (2012)
  • SurveyMonkey has been used successfully for a number of GNSO questionnaires
  • Anonymity of responses can be safeguarded under both services
• Cons:
  
  • QuestionPro requires some expertise to input/modify questions and download/analyze the results