Page History

It is suggested that this key ICANN board resolution is included:

In its resolution of 28 September 2013 the ICANN Board stated: “The expansion of the TLD space has increased the number and variety of stakeholders participating in GNSO policy making and a review needs to take place on schedule to examine whether the current model meets the needs of a new generation of stakeholders.

GNSO Structure is unlikely to accommodate the anticipated new stream of stakeholders resulting from the expansion of the TLD space. The GNSO Review will be an important vehicle for considering and addressing this issue. The unbalance that is already occurring needs to be addressed by the GNSO Review. ”

I would close by saying that I would be delighted to be interviewed, if Westlake wished to enlarge its sample size a bit.  I am a relatively new participant at ICANN (two years, 8 meetings) and a new member on council.  I don’t believe newcomers are well represented in the current sample, which is a pity, because we are less likely to have history which might colour our observations, and we are supposed to be a target group for recruiting and preventing burnout.

Personally I find it unfortuante that your draft report put out calumnious quotes that have no basic in fact.  I for one have been working very hard to help NPOC make its way in the NCSG and find the fact that this is what you picked to include so very unfortunate.  Now instead of dealing with the substance of the report the NCUC is going to have to deal with arguing against the lies.

It also seems that you went against your rule of not including quotes where the speaker could be identified.  While it may be hard to pin the specific speaker, the choice of one of two possible speakers is relatively easy.  What you have done is take a sometimes difficult situation that has been improving over the years, entered it as patisans and made it so much worse.  Now a competition of who can mud sling best is going to be much harder to contain.

It is sad that you have made the work of consensus building so much harder.

(FOLLOW-UP EMAIL):

I wish to withdraw part of my statement.  I assumed I knew the short list who uttered those statements.  I can't know and there is every chance that I am wrong.  I am glad I did not name names, but I must say, the problem remains with the quotes and the fact that many have already had the same assumption  about the speakers I had.

Apologies for the intemperance on that one point.

I do request that the incendiary quotes be removed from the report.

The study seems to have a constantly changing and imprecise design that meanders between various means of investigation without fully investing in any one.

As a qualitative study there is no perceptible strategy or control other than the relay of “observations” of Westlake staff and the selected use of anecdotes from unidentified parties. A clearly defined narrative approach may have proven useful here but there is no indication that was ever considered or acted upon. The quantitative aspects of the study lack any rigor or application of standard statistical sampling or analysis techniques. Samples are generally undefined and too small to generate the conclusions extrapolated from them. Adjustment of methodology mid-study (e.g. the Supplementary Working Group survey) raise questions of corrective measures polluting the findings (e.g. strategic sampling).

All aspects of sampling in this study are problematic.
The criteria used by Westlake for inclusion of subjects in various components of this study is unclear. There does appear to be some application of snowball sampling techniques in this work, which is itself a problem. Although useful when researching hard to identify or locate populations, snowball sampling is not a preferred technique when conducting research into easily identifiable, bounded (and divided) communities like the GNSO.
There is no indication that the sample used in the quantitative portion of this study was a result of anything other than self-selection. We are told that the quantitative portion of the study is based upon 152 surveys completed by 250 individuals who initially accessed the survey site. The Report states “this provided a wide and representative sample” (p. 8). We see no evidence of this. There is no indication as to general population size, response or rejection rates, or precise subgroup identification of those who responded. We have no indication that this survey provides results representative of the GNSO community as claimed.
Although we have no indication as to the statistical validity of the 360º survey with reference to the general community, we can demonstrate that the subgroup responses relating to Constituencies contained on pages 78 and 79 of this draft Report have no practical value.
For example, in the Table on pg. 78, the draft report shows that just 52% of the subgroup responding about NCUC believe that “the executive committee of the group is balanced and appropriately representative”. This is based upon a cohort of 27 respondents. There is no indication whether these 27 individuals are all NCUC members (which would be surprising, given that the 360º review was conducted while we were gearing up to elect five regional representatives to the Executive Committee, EC) or self-selected to answer questions related to the NCUC based on whatever level of familiarity they had with our elections and EC composition. But if all were NCUC members, a best case scenario for survey relevance, consider the statistical validity of this small sample:

Number of NCUC members: 404
NCUC respondents: 27
Desired confidence level: 95%
Agreement Percentage: 52%
Confidence Interval: 18.23
In this specific instance, if all 27 respondents attributed to the NCUC were NCUC members, something not in evidence, all that can be inferred from this question is that, with a 95% level of confidence, somewhere between 33.77% and 70.23% of NCUC members agree with the statement “the executive committee of the group is balanced and appropriately representative”.
Clearly, the subgroup sample sizes are too small to provide statistically valid information of any real value. Further data is needed, as above, to determine whether the same holds true for statistical data relating to questions answered by all survey respondents.
Parenthetically, the Table on pgs. 78-79 should not be introduced with the misleading phrase, “This table shows the extent to which,” because it does not. It shows the extent to which respondents made certain assertions, irrespective of their relationship to actual facts (some of which easily could have been checked, for all SG/Cs concerned).

The analysis is replete with generalities not adequately linked to facts. This is particularly problematic when so much of this study is based upon Westlake’s observations and selected anecdotes.

A high degree of approximation occurs throughout the study. Consider these examples: “There was a view that” (p. 82); “anecdotal but credible instances” (p. 7); “we received no comment…based on this, we conclude this is no longer a significant issue” (p. 95); “some survey respondents” (p. 81); “there is a perception among some” (p. 81); “there is an often expressed view” (p. 90); and “we encountered active hostility to new leaders from a few participants” (p. 90). Instead of some, how many? Instead of often, how often?

A professional study should have recommendations based on specific facts in evidence. Too often this study does not. Of particular concern are the Westlake Review Recommendations on page 92 of this Report. Several of these recommendations, such as those involving travel, are not supported by anything substantive previously discussed in the draft Report.

Quotations used in this study are completely stripped of context and because of this are of limited value.
It is certainly understandable that the identity of respondents are kept confidential. What is hard to understand is why other information, such as constituency or stakeholder group membership or other demographic information, is not provided. Knowing something about the speakers’ backgrounds adds context, value and enables one to better understand the comments themselves.

A professional study of this type should have as its base proper design, a reliance upon facts obtained in a credible and transparent manner that meet basic standards of empirical research. This study does not, as is acknowledged at times by its own authors. Until it does, any recommendations made by Westlake should be considered as emanating from a flawed and poorly constructed study.

One of the more unsettling aspects of the draft Report is its substitution of opinion or conjecture for data that is readily available. We note that the authors of this independent study were guided in this process by ICANN staff who “on several occasions have directed us to information that we might not otherwise have been aware of or otherwise been able to find” (p. 11). We certainly would have welcomed the same opportunity at an earlier time to help and guide the Westlake team in this manner.
We need to reiterate that the small sample size involved in the ‘numeric results’ of the 360º survey renders any result from the ‘Stakeholder Groups and Constituencies’ section of the survey invalid. In fact, by acting as if these numbers have statistical validity, the authors of this report are misleading the GNSO community. The confidence interval is simply too large and the sample not random enough for any inference to the larger NCUC population to be made on the basis of this study.