ALAC Metrics - Discussion
WHAT ARE METRICS AND WHY ARE WE WORKING ON METRICS?
Very basically, Metrics are measures which assess how well organisational members are performing - meeting targetted expectations and giving value.
The ALAC Rules of Procedure Section 9 is entitled "Performance, Metrics and Remediation"
This section states that the performance of the ALAC is measured by its ability to represent the interest of Internet users, and depends on strong participation from all ICANN regions.
Furthermore, the ALAC can function effectively only if all ALAC Members and Appointees meet the obligations of their positions.
Satisfactory performance is a complex concept including both objective and subjective issues and must factor the very significant personal contributions made by At-Large volunteers.
Moreover, the ALAC and At-Large receive significant funding from ICANN, both for travel and other activities, and the ALAC must be able to justify such expense.
There are three key things we have to do to get started on establishing appropriate metrics:
1.Establish what is expected of an ALAC member - key performance tasks
2. Identify what the outputs of these tasks should (specific and quantifiable)
3. Establish the criteria against which the results of these tasks can be rated (these are the measures)
FOR EXAMPLE
ALAC: Definition of the criteria of involvement and participation:
- Attendance:
1.1 Attend ALAC conference calls and any other virtual meeting
1.2 Attend face to face ALAC meetings held during an ICANN meeting
1.3 Attend monthly RALO meetings
1.4 Attend RALO and RALO-related meetings
1.5 Apologies given before a meeting when not available to attend
2. Participation and Contribution:
2.1 Participate and contribute to ALAC discussions relating to ICANN policy development through:
2.1.1 mailing lists
2.1.2 wiki pages
2.1.3 virtual meetings
2.1.4 face to face meetings
2.2 Take a lead role in ALAC-initiated working groups
2.3 Participate and contribute to ALAC-initiated working groups
2.4 Take a lead role (eg pen-holder) in working groups of other bodies within ICANN
2.5 Participate and contribute to the role of ALAC in working groups of other bodies within ICANN
2.6 Participate and contribute to RALO and RALO-related meetings
3. Voting:
3.1 Participate in the decisions and votes by the ALAC on ICANN policy issues
4. Reporting:
4.1 Report to the ALAC on ALAC -initiated working groups or other ICANN activities
4.2 Report to the ALAC on any liaison roles related to other bodies within ICANN
4.3 Report to the ALAC on any working group activities
4.4 Report to the RALO on ALAC decisions and involvements
ALAC: Metrics for involvement and participation
5. Attendance
5.1 Attended “#” monthly conference call meetings
5.2 Attended "#" working group meetings
5.3 Attended "#" other ICANN body meetings
5.4 Attended "#" RALO-related meetings
5.5 Apologies given for "#" meetings that was unable to attend
5.6 No apologies given for "#" meeting that did not attend
6. Participation and contribution
6.1 Participated and contributed to “#” monthly conference call meetings (by audio or chat)
6.2 Participated and contributed to "#" working group meetings (by audio or chat)
6.3 Participated and contributed to "#" other ICANN body meetings (by audio or chat)
6.4 Did not participate or contribute to "#" meetings that attended
7. Voting
7.1 Voted on "#" formal ICANN decisions proposed by ALAC
7.2 Did not vote on "#" formal ALAC decisions
8. Reporting
8.1 "#" reports provided to ALAC meetings
8.2 "#" reports provided to RALO meetings
9. Definition of the ALAC status:
9.1 Active 60%+
9.2 Less active 40%-60%
9.3 Not Active 0-40%
Update of ALS and RALO comments to hand, regarding ALAC metrics.
Wolf Ludwig (11/12/13) after MH request to raise ALAC metrics issue with ALSes at next EURALO meeting
I have seen your comment or “invitation”, Maureen but didn't put it on our call agenda because a) it was extensive enough already and b) there won't be any comments from our members – the subject is far too specific! EURALO defined subjects of key priorities for the region last June at our GA. This also means that we would like to concentrate on these issues! Therefore, I won’t push our members to submit a comment on whatever subject what has less relevance for us/them. I hope you will understand my point.
Olivier Crepin-Leblond (06/12/13)
Hello all,
you might be interested in seeing some of the GNSO metrics on attendance...
https://community.icann.org/display/gnsocouncilmeetings/GNSO+Council+Attendance-20+Nov+2013
Of interests is the actual implementation of the updating which is described on:
https://community.icann.org/display/gnsocouncilmeetings/Attendance+Logs
Kind regards,
Glenn McKnight (03/12/13)
Hi ..At the metrics meeting on the Sunday I tried to bring up the issue of punishment vs reward.
It didn't seem to take on great interest and it was dismissed
For the record, this onus on punishment which has been central to US jurisprudence has been reconsidered
Here is an article
Ali AlMeshal (01/12/13)
Thank you Tijani for your input. I can see that in principle you are agreeing with me in some of the below if it is not all. To comment on your feedback please allow me to crystallize my below views.
If you may notice that I have listed my points below in logical sequence , staring from what is on the table as a points to what do I see as a proposal.
My first point:
- “Attending meetings by itself is not a goal or objective therefore it cannot be part of the measuring criteria independently.”
- Also participation and contribution to the meeting discussion cannot be evaluated as a performance measure
Therefore to make it part of the criterion I suggested this
- ALAC members should be assigned to Chair a WG’s
§ With this type of assignment the member will logically be accountable and responsible of a set deliverables that he/she have to achieve, and this can be measured and evaluated for the following set of KPI’s
- · Time to deliver
- · Commitment
- · Team working
- · Developing others (member of the group)
- · Others
So by doing the above we are fulfilling the attending and contribution part of the criteria , but the major different hear is that the member is not attending or participating for the sack of showing up or to record his attendees only , No he is attending to update , discuss and participate and give feedback on action item that he is in-charge of and so on.
My Second point:
- ALAC members and chair of RALO’s jointly should be responsible for engaging the ALS’s in work and activities related to the rejoin at least and should provide a monthly call report on this.
§ This is different activity from the RALO monthly call; this should be done separately to reach out the ALS’s through a pre-plan agenda and time frame.
§ The objective is to get in touch with ALS’s members outside the official call and tries to understand their needs and requirements and also will be a good tool maybe to get them engage in if they are not active. So this is more of OUTREACH on a small scale and more of direct communication.
- On the assumption that this point is clear, then for the parameter, what I have thought about for the time being is the Call Report that need to be presented in the monthly call meeting. The call report for example should look like
Date | ALS Name | Contact Person | Comments and feedback |
12/12/12 | Bahrain Internet Society | Ali AlMeshal | 1 – 2 – 3 - |
01/01/13 | xxxxx | xxxxx |
|
Should you or any other have more thoughts of this then it would be very much helpful to enrich this fruitful discussion.
Hope I manage to answer and clarify your enquiries and doubts .
Thanks
From Tijani Benjemaa (30/11/13)
Dear Ali,
Thank you very much for your comments
- Attending meetings by itself is not a goal or objective therefore it can not be part of the measuring criteria independently.
Ali, ALAC can’t decide on anything without quorum, and members who don’t attend the ALAC meetings may break its functioning. Also, I may agree with you that attending is not a goal and shouldn’t be part of the criteria, but the attendance is absolutely necessary for any contribution during the meeting/call. It’s the elementary duty for the ALAC members. Attendance is proposed as one of the metrics element
- Also participation and contribution to the meeting discussion can not be evaluated as a performance measure
If the participation and contribution to the ALAC meetings is not considered as a performance criterion, that means that all the ALAC members can decide not meet at all and they can be considered as having good performance.
Thereafter to have a process in place that gives an indication of the member performance I would suggest the following
o ALAC members should be assigned to Chair a WG’s
This is another criterion listed in the proposal
o ALAC members and chair of RALO’s jointly should be responsible for engaging the ALS’s in work and activities related to the rejoin at least and should provide a monthly call report on this.
What are the parameters of measurement you propose for this?
Thank you again for your contribution
Fouad Bajwa (28/11/13)
I believe one of the most important metrics component should be built on the fact that if a member suggests something, how many of those suggestions were actually accepted and incorporated into the system. One small example is that I've proposed individual membership in the past during 2010, created the paperwork suggesting the policy reform in the APRALO articles, presented it and then silence. This would have enhanced participation, inclusion of more productive talent and people that are sensitive about ICANN and IG related issues. Such talent thus has to move under a narrower scope into the ncuc/ncsg and APRALO loses out.
Somehow the belief that only groups can represent the rights of the users is a fantasy. For example, in the technical community and most in ISOC circles, most of the members in a recognized ALS are not actually CS or information rights activists but people with corporate day jobs and of companies that actually should fall under contracted parties ac/oc's in ICANN. The value that comes into such an organizational system are actually the members, the system itself cannot come up with ideas and transform the ideas into workable actions.
Participation collapses when member ideas are not viewed as valuable input and organizations that don't believe in agility to change and adapt to round the clock innovation and improvements basically fall short on rationality for existence and support in the future. If the participation in meetings and tele-cons are an evaluative criteria for calculating value for money, something is seriously wrong with ICANN and the people who are attempting to address such issues by adopting such a weak and lame course of action.
KPIs are measured against activities and participation in telecons and meetings are not such an input or expected outcome. The indicators can be established on a the basis of number of members present and number of comments, suggestions and recommendations received during any calls/meetings and the actions were taken on them. The second level would be number of policy comments requested by ICANN and the number of members that voted on them, the time taken to make comments or recommendations, total number of policy requests for comments and number of comments made to policy requests.
Its actually a very simple and straightforward thing. List down all the things that ALAC and its members do. Against those activities a scorecard can be developed and bench-marked against other AC/OC work. I'll think more about this.
Rinalia Abdul Rahim (27/11/13)
Hello everyone.
I like Ali's input on metrics about leadership of Working Groups for ALAC members. I would also suggest co-chairing of WG between an ALAC Member and an At-Large community member. This will help build collaborative skills and capacity along the way.
Best regards, Rinalia
Ali Al Meshal (27/11/13)
Good day Maureen,
First of all we have always to but in mind as your correctly stated that this is a volunteer work from the members, but on the other hand I do agree that up to certain extent there should be a fair KPI’s in place to evaluate the performance of ALAC members.
Also I am quite sure that selected or nominated ALAC members for these leadership positions were based on their knowledge , commitment and experience as well as high performance otherwise they will not be part of the team.
Hence measuring criteria would always be much clearer and effective if it is set for Quantitative factors rather than Qualitative once.
So having said that then I would like to address the following:
o Attending meetings by it self is not a goal or objective therefore it can not be part of the measuring criteria independently.
o Also participation and contribution to the meeting discussion can not be evaluated as a performance measure
Thereafter to have a process in place that gives an indication of the member performance I would suggest the following
o ALAC members should be assigned to Chair a WG’s
§ With this type of assignment the member will logically be accountable and responsible of a set deliverables that he/she have to achieve, and this can be measured and evaluated for the following set of KPI’s
- Time to deliver
- Commitment
- Team working
- Developing others (member of the group)
- Others
o ALAC members and chair of RALO’s jointly should be responsible for engaging the ALS’s in work and activities related to the rejoin at least and should provide a monthly call report on this.
§ This is different activity from the RALO monthly call; this should be done separately to reach out the ALS’s through a pre-plan agenda and time frame.
§ The objective is to get in touch with ALS’s members outside the official call and tries to understand their needs and requirements and also will be a good tool maybe to get them engage in if they are not active. So this is more of OUTREACH on a small scale and more of direct communication.
….Should you need any clarification then please let me know by email or a call, I will be more than happy to discuss.
From Karaitiana Taiuru (25/11/13)
These are my thoughts based on not been an ALAC member (nor do I have time to).
I would support a more closer monitoring approach to the ALAC performance review and would like to see the evaluations distributed to the relevant RALO's as we are the ones who nominate and put our trust in our representatives to provide our views and then to report back to us.
I would expect that participation in all meetings, emails and Work Groups would be at the very minimum 90%. There are so many issues at present it would be hard not to have a voice at a meeting.
Saying this, I would also expect that any new ALAC appointments are mentored and giving an appropriate transition period and were made to feel comfortable to ask for help. Some cultural and language barriers may also exist and should be considered.
If people are under-performing, then perhaps a mechanism of someone talking to them in a non threating manner to see if there is any assistance required etc.
Perhaps too, some way of recognition for the long hours and dedication may also be a motivation. This is likely to be more of a RALO initiative though.
From Winthrop Yu (24/11/13)
I'd tried to find people here (even outside ISOC-PH) who could actively and constructively participate in the various WGs. Unfortunately, no takers so far, and i really don't like the idea of simply "tapping" someone who may not be as committed to the work or drop the ball. For myself, i'm already committed to policy issues on the local front and cannot honestly make any further commitments until ... oh, after December 2014? :)
Comments are on the workspace at
10 Comments
Holly Raiche
First- a terrific list and I think they are pretty much the metrics we are looking for.
A minor point - in 1.1 - omit the term 'or any other virtual meeting' since it is not clear what is meant.
On percentages, since there are 12 monthly meetings of both RALO and ALAC meetings, I'd suggest that 10 out of 12 is what is aimed for. And that is actual attendance. I'm not sure how to handle apologies since I'd hate to think that all one has to do is apologise and it counts - but maybe a separate metric for attendance with apology and attendance without apology. (and maybe something special for the APAC region where we are talking about meetings at 1.00am or 2.00am!)
And maybe a separate metric for participation in non-ALAC ICANN working groups?
And a question - who keeps the metrics? It's a bit of work for ICANN staff who are already busy so it may be making sure that the metrics are easy to collect and put onto a matrix to quickly show who is active - or not.
Maureen Hilyard
Yes.. exactly.. .I think the wording for each of these points is very important which is why we need to RALO inputs.. I have probably put in much more than we need, and then missed out something important, but the feedback from RALOS will hopefully tell me this
Thank you for this response Ms APRALO Chair. You have highlighted some important issues. I hope the other RALOs are as responsive.
Someone has made a start on the ALS metrics which I would be interested in getting some feedback on (after the ALAC one has been thrashed about).
Maureen Hilyard
Emailed comments:
Sent: Tuesday, 5 November 2013 12:48 p.m.
Dear Maureen,
I’m so sorry for the late reply. I read your proposal, and find it good.
I already proposed more than one year ago a similar proposal, and I updated it today and posted it on the wiki.
We will need now to concentrate on compiling all the proposed texts for a comprehensive and precise one that we walk through to enrich it till we reach consensus on a final version.
Tijani
**********************************************************************
Hi Tijani
I completely agree and thank you for your proposal. We need to get different models up that people can see the issue from different perspectives and hopefully understand the complexities. From their discussions we will hopefully achieve a text that clearly explains the process, what is required and how it will be. I know that it is not going to be an easy task.
I am looking forward to meeting up with everyone in BA. J
Regards Maureen
***********************************************************************
Dear Maureen,
Bravo for the excellent draft. We might need to explain more the definition of ALAC status. Will it be the same to just be present at meetings than be present and participate? The answer is NO but let me ask more questions. Does participate mean having something to say even if that is just "I agree" ? Should participants need to say something any how to be considered? Does significant attendance mean you attend X meetings + at least 2X comments?
I just want to better understand the logic.
Thank you, Fatimata Seye Sylla
***********************************************************************
Sent: Tuesday, 5 November 2013 1:01 p.m
Hello Fatimata
You highlight a real problem that I foresee.. how does one define “participation” and “contribution”?
I think about the number of times when my contribution at a meeting has been “+1” because someone much more articulate than I has said exactly what I wanted to say.
We also have to balance ordinary meetings with contributions that members make to working groups, as pen-holders or even if just to the online and written conversations which are much more interactive in a working group role.
Perhaps rather than measures, we should be establishing benchmarks which can look at all types of involvement (360 degrees) and which might help to identify “active” or “not so active” involvement in a much more friendly way.
These are all the things we have got to nut out. And no doubt we will, in Buenos Aires. Looking forward to seeing you there.
Regards, Maureen
***********************************************************************
Sent: Tuesday, 5 November 2013 1:01 p.m
I support Maureen position because it is often not necessary to say the same thing when there was a contribution that takes into account what is meant.
I propose that recapitulatif be done regularly after a certain stage in order to allow each other to stay in constant even as the overall dynamics.
Furthermore, I would like to know what happens in the space of wiki work. I still can not get to post my comment
SCHOMBE BAUDOUIN
************************************************************************
about points 1,4,5,6 and 7:
-What are those directly and indirectly involved?
-Als can participate in votes of ALAC?
These precisions are needed to understand the level of participation of Als in the activities of ALAC and RALO.
SCHOMBE BAUDOUIN
************************************************************************
2013/11/5 Maureen Hilyard <hilyard@oyster.net.ck>
Hi Baudouiin
Are you referring to the points in my discussion paper on
https://community.icann.org/display/atlarge/ALAC+Could+do+workspace+-+ALAC+Metrics+-+discussion
The ALSes will have their own metrics – separate from that of the ALAC members.
Maureen
*************************************************************************
Yes Maureen, I just wanted to remove the shadow of doubt in my mind and thank you for this precision.
SCHOMBE BAUDOUIN
Maureen Hilyard
RALO comments - gathered during ICANN week
NARALO
LACRALO - Sergio offered in the Regional Leaders meeting to be a liaison person for LACRALO
Contributor #1
Contributor #2
Contributor #3
Maureen Hilyard
Message to RALO Secretariats 26 Sept (placed in wrong workspace)
Apologies to the Secretariat for the very poor sound quality on Adobe Connect (26 Sep). I suspect it is our system in the Cooks. I can hear clearly what is being said, but cannot upload. And my phone connections kept cutting out too. Thanks to Silvia and Heidi for their summaries. Just to add..
The purpose of the METRICS group is to develop metrics for ALAC and as Heidi mentioned we are also doing a review of APRALO documentation, therefore I will be taking note of metrics that arise out of the decisions that are made about the ALAC to be transferred to our review of APRALO documentation. It is hoped that perhaps other RALOS may consider this as well.
Because of the complexity and sensitivity of the issues relating to evaluation of performance of people who are volunteers, but It is important that people who put themselves forward for a position on the ALAC are aware that their appointment does impose responsibilities and accountability.For this reason, it is the responsibility of the group to review the metrics that are listed in Rule 21.
In order to do this, it is important that we take the discussion to our Secretariats and RALOs and to get some feedback. At the moment the quantitative aspects of participation are recorded, but the qualitative aspects are a little more difficult. Attending a meeting, and participating, are seen as separate metrics items. How do we appropriately measure participation or performance?
Similarly, someone who may not actively participate in meetings, but may be an active member of working group activities, so there has to be some balance in what we are measuring across the many responsibilities that ALAC members have, and the activities in which they are involved.
For this reason it is really important that we gather information from Secretariats and RALOs to help us with this task. It is also assuming that all our ALSes are aware of what the ALAC does so that they can appropriately comment on the metrics.
We would like to know what kind of information RALOS would like to know from these metrics, and how they would like this information reported back to them.
But if you can assist us with any feedback we'd really appreciate it. Many thanks in advance.
Holly Raiche
Just to make life even more difficult, it would be even harder to develop metrics for participation in non ALAC WGs, where theree may be only one ALAC person participating in that WG and the only measurement that can be verified by ALAC/RALO people is attendance.
A couple of measurements that may help would be the number of times someone is a pen holder. That, too, doesn't measure the complexity of the issue that had to be commented upon, but it's a start. Another could be number of WG's that one chairs, as opposed to just participates in.
Maureen Hilyard
Maureen Hilyard
Report on data gathering for the ALAC METRICS working group from RALO ALSes
January 2014
Referring to Section 9 of the ALAC ROPS, the Working Group was tasked to:
Mixed reaction and responses met the initial request for feedback and comments. Some RALO members were suspicious of ALAC’s reasons for initiating such a task group. It had been an early expectation that our feedback would come from ALSes which is why RALO Secretariats were approached first about the issue and were asked to raise it with their RALOs. I have yet to get any feedback that they did. In general, at the F2F meetings with RALOs in Buenos Aires (LACRALO, NARALO, APRALO) responses came from current or ex-ALAC members rather than from the general electorate. AFRALO and EURALO responded on the wiki.
Strong ALS capacity building recommendations arise (for ATLASII ?):
ALSes require a greater understanding of:
With a better understanding of the workings of ICANN and ALAC, ALSes can better
**********************************************************************************
A brief summary on feedback from the RALOs
Key performance tasks: Section 9 of the ALAC ROPS, when categorised, detail:
The originally proposed metrics models on individual task outputs provided valuable feedback:
Problems were identified when relying on a statistical output for each task:
Recommendations
Maureen Hilyard
Discussion points from last meeting's chat
Definition of the criteria of involvement and participation
Meeting attendance of :
Contribution to:
Vote: Voting in the
Definition of the ALAC member status: Active, Less Active, Standby (Silvia)
Members in Standby status
How we might identify contribution to working groups to identify "active" ALAC membership
(MH - discussion starter) Perhaps - to ensure engagement by all ALAC participants:
Maureen Hilyard
As an example to demonstrate attendance and actual outputs - with ALAC members taking responsibility for their own feedback on performance - additional groups and tasks could be added. This could be done in whole or half year blocks and made available to everyone.
ALAC MEMBER: MAUREEN HILYARD, APRALO (*meeting attendance dates given in CI time UTC-10)
GROUP
POSITION
TASKS - 2014
Jan
Feb
Mar
Apr
May
Jun
ALAC
Member
Liaison reports, updates
28
ALT
Member
Liaison reports, updates
16
30
ccNSO
ccNSO Liaison
Coordinating committee; liaison reports
13
APRALO
Member, penholder
MOU APNIC, MOU APTLD
27
APRALO SINGAPORE
Showcase OC, penholder
Brochure, powerpoint, liaison with ALSes, fans
13
5
FOIWG
Member
WG updates in ccNSO reports
23
5
METRICS WG
Member, RALO liaison
Reports on RALO liaison contact (wiki)
6
BMSPC
Member
29
ATLASII Events WG
member
12
Sergio Salinas Porto
Síntesis de lo debatido en LACRALO sobre el tema Metricas
Algunos miembros de LACRALO entendemos que sobre el tema de "auto evaluación" es necesario tener en cuenta que los representantes de las regiones en ALAC son una "extensión" o "herramienta" de la región, la competencia basada en el conocimiento no debería caer sobre el representante sino sobre el colectivo que le da mandato u opinión sobre tal tema.
El conocimiento agudo sobre determinados temas le corresponde a TODA la región y no a un representante, que podrá tener conocimiento sobre algún tema en particular pero no sobre todo lo que se debate en ALAC (que es un reflejo de lo que se esta discutiendo en todo ICANN).
La evaluación externa es un elemento que no podemos dejar de tener en cuenta, pero esta evaluación externa debería tener como principio que fuera realizada por componentes de la misma región en que fue electo el representante y garantizando un marco de transparencia e imparcialidad.
Esto está basado en que si las regiones pueden elegir sus representantes también tienen capacidad para medir su desempeño e impartir sanciones si lo creen conveniente, el cual no podría ni debería ser realizado por ningún elemento externo de la región (se incluye a ALAC en esto).
Sobre el control se sugiere tener en cuenta la cantidad de horas de dedicación mínima (que no puede ser indefinida), el hecho concreto que somos voluntarios y no profesionales pagos trabajando para un empleador, etc.
Algunos miembros de la región opinan sobre el tema de “auto evaluación” diciendo que “...una auto evaluación siempre tendrá un carácter protector, y una auditoría o evaluación externa será resistida con argumentos como que "el auditor o inspector no entiende o no conoce lo que realmente sucede", y suspicacias de sesgo o parcialidad...”, entienden que “...Se requiere un cambio de cultura en el que la exigencia de actividad, participación informada y pro activa, y sobre todo resultados sólidos sea una norma generalizada...” en otro segmento del rico debate se profundizo diceindo que “... señalé que una característica de las métricas de participación debe ser la dificultad para "trucarlas", es decir, para simular su cumplimiento. Claro que esta dificultad no debe crearse mediante más burocracia sino al contrario, apelando a un principio de simplicidad...”
“...la definición de las evaluaciones debe ser hecha con una importante contribución externa que mantenga crítico y honesto el trabajo. La evaluación también deberá tener un componente externo.
Por otra parte, como en tantas otras tareas, es importantísimo incluir las demostraciones de competencia en las evaluaciones. No basta asistir a las reuniones, participar en las listas, mantener relaciones y comunicaciones con muchos participantes, votar, etc. si las participaciones no son competentes, basadas en conocimiento, apegadas a tiempos oportunos para influir en las decisiones, etc...”
Por otro lado, varios se manifiestan diciendo que “...La auto evaluación solo podría tomarse como un indicio más, que puede aportar algunos elementos, relativos a la visión subjetiva, pero no como el único medio de relevar y realizar una medición objetiva. Esas otras tareas que se realizan pero no se ven son importantes y es necesario tenerlas en cuenta en una evaluación justa...”