ALAC Metrics - Discussion
WHAT ARE METRICS AND WHY ARE WE WORKING ON METRICS?
Very basically, Metrics are measures which assess how well organisational members are performing - meeting targetted expectations and giving value.
The ALAC Rules of Procedure Section 9 is entitled "Performance, Metrics and Remediation"
This section states that the performance of the ALAC is measured by its ability to represent the interest of Internet users, and depends on strong participation from all ICANN regions.
Furthermore, the ALAC can function effectively only if all ALAC Members and Appointees meet the obligations of their positions.
Satisfactory performance is a complex concept including both objective and subjective issues and must factor the very significant personal contributions made by At-Large volunteers.
Moreover, the ALAC and At-Large receive significant funding from ICANN, both for travel and other activities, and the ALAC must be able to justify such expense.
There are three key things we have to do to get started on establishing appropriate metrics:
1.Establish what is expected of an ALAC member - key performance tasks
2. Identify what the outputs of these tasks should (specific and quantifiable)
3. Establish the criteria against which the results of these tasks can be rated (these are the measures)
FOR EXAMPLE
ALAC: Definition of the criteria of involvement and participation:
- Attendance:
1.1 Attend ALAC conference calls and any other virtual meeting
1.2 Attend face to face ALAC meetings held during an ICANN meeting
1.3 Attend monthly RALO meetings
1.4 Attend RALO and RALO-related meetings
1.5 Apologies given before a meeting when not available to attend
2. Participation and Contribution:
2.1 Participate and contribute to ALAC discussions relating to ICANN policy development through:
2.1.1 mailing lists
2.1.2 wiki pages
2.1.3 virtual meetings
2.1.4 face to face meetings
2.2 Take a lead role in ALAC-initiated working groups
2.3 Participate and contribute to ALAC-initiated working groups
2.4 Take a lead role (eg pen-holder) in working groups of other bodies within ICANN
2.5 Participate and contribute to the role of ALAC in working groups of other bodies within ICANN
2.6 Participate and contribute to RALO and RALO-related meetings
3. Voting:
3.1 Participate in the decisions and votes by the ALAC on ICANN policy issues
4. Reporting:
4.1 Report to the ALAC on ALAC -initiated working groups or other ICANN activities
4.2 Report to the ALAC on any liaison roles related to other bodies within ICANN
4.3 Report to the ALAC on any working group activities
4.4 Report to the RALO on ALAC decisions and involvements
ALAC: Metrics for involvement and participation
5. Attendance
5.1 Attended “#” monthly conference call meetings
5.2 Attended "#" working group meetings
5.3 Attended "#" other ICANN body meetings
5.4 Attended "#" RALO-related meetings
5.5 Apologies given for "#" meetings that was unable to attend
5.6 No apologies given for "#" meeting that did not attend
6. Participation and contribution
6.1 Participated and contributed to “#” monthly conference call meetings (by audio or chat)
6.2 Participated and contributed to "#" working group meetings (by audio or chat)
6.3 Participated and contributed to "#" other ICANN body meetings (by audio or chat)
6.4 Did not participate or contribute to "#" meetings that attended
7. Voting
7.1 Voted on "#" formal ICANN decisions proposed by ALAC
7.2 Did not vote on "#" formal ALAC decisions
8. Reporting
8.1 "#" reports provided to ALAC meetings
8.2 "#" reports provided to RALO meetings
9. Definition of the ALAC status:
9.1 Active 60%+
9.2 Less active 40%-60%
9.3 Not Active 0-40%
Hi everyone.. I am having problems getting into the comments boxes below. Many thanks to you all for your valuable contributions. 24/11/13 (and still on 06/12/13)
Olivier Crepin-Leblond (06/12/13)
Hello all,
you might be interested in seeing some of the GNSO metrics on attendance...
https://community.icann.org/display/gnsocouncilmeetings/GNSO+Council+Attendance-20+Nov+2013
Of interests is the actual implementation of the updating which is described on:
https://community.icann.org/display/gnsocouncilmeetings/Attendance+Logs
Kind regards,
Glenn McKnight (03/12/13)
Hi ..At the metrics meeting on the Sunday I tried to bring up the issue of punishment vs reward.
It didn't seem to take on great interest and it was dismissed
For the record, this onus on punishment which has been central to US jurisprudence has been reconsidered
Here is an article
Ali AlMeshal (01/12/13)
Thank you Tijani for your input. I can see that in principle you are agreeing with me in some of the below if it is not all. To comment on your feedback please allow me to crystallize my below views.
If you may notice that I have listed my points below in logical sequence , staring from what is on the table as a points to what do I see as a proposal.
My first point:
- “Attending meetings by itself is not a goal or objective therefore it cannot be part of the measuring criteria independently.”
- Also participation and contribution to the meeting discussion cannot be evaluated as a performance measure
Therefore to make it part of the criterion I suggested this
- ALAC members should be assigned to Chair a WG’s
§ With this type of assignment the member will logically be accountable and responsible of a set deliverables that he/she have to achieve, and this can be measured and evaluated for the following set of KPI’s
- · Time to deliver
- · Commitment
- · Team working
- · Developing others (member of the group)
- · Others
So by doing the above we are fulfilling the attending and contribution part of the criteria , but the major different hear is that the member is not attending or participating for the sack of showing up or to record his attendees only , No he is attending to update , discuss and participate and give feedback on action item that he is in-charge of and so on.
My Second point:
- ALAC members and chair of RALO’s jointly should be responsible for engaging the ALS’s in work and activities related to the rejoin at least and should provide a monthly call report on this.
§ This is different activity from the RALO monthly call; this should be done separately to reach out the ALS’s through a pre-plan agenda and time frame.
§ The objective is to get in touch with ALS’s members outside the official call and tries to understand their needs and requirements and also will be a good tool maybe to get them engage in if they are not active. So this is more of OUTREACH on a small scale and more of direct communication.
- On the assumption that this point is clear, then for the parameter, what I have thought about for the time being is the Call Report that need to be presented in the monthly call meeting. The call report for example should look like
Date | ALS Name | Contact Person | Comments and feedback |
12/12/12 | Bahrain Internet Society | Ali AlMeshal | 1 – 2 – 3 - |
01/01/13 | xxxxx | xxxxx |
|
Should you or any other have more thoughts of this then it would be very much helpful to enrich this fruitful discussion.
Hope I manage to answer and clarify your enquiries and doubts .
Thanks
From Tijani Benjemaa (30/11/13)
Dear Ali,
Thank you very much for your comments
- Attending meetings by itself is not a goal or objective therefore it can not be part of the measuring criteria independently.
Ali, ALAC can’t decide on anything without quorum, and members who don’t attend the ALAC meetings may break its functioning. Also, I may agree with you that attending is not a goal and shouldn’t be part of the criteria, but the attendance is absolutely necessary for any contribution during the meeting/call. It’s the elementary duty for the ALAC members. Attendance is proposed as one of the metrics element
- Also participation and contribution to the meeting discussion can not be evaluated as a performance measure
If the participation and contribution to the ALAC meetings is not considered as a performance criterion, that means that all the ALAC members can decide not meet at all and they can be considered as having good performance.
Thereafter to have a process in place that gives an indication of the member performance I would suggest the following
o ALAC members should be assigned to Chair a WG’s
This is another criterion listed in the proposal
o ALAC members and chair of RALO’s jointly should be responsible for engaging the ALS’s in work and activities related to the rejoin at least and should provide a monthly call report on this.
What are the parameters of measurement you propose for this?
Thank you again for your contribution
Fouad Bajwa (28/11/13)
I believe one of the most important metrics component should be built on the fact that if a member suggests something, how many of those suggestions were actually accepted and incorporated into the system. One small example is that I've proposed individual membership in the past during 2010, created the paperwork suggesting the policy reform in the APRALO articles, presented it and then silence. This would have enhanced participation, inclusion of more productive talent and people that are sensitive about ICANN and IG related issues. Such talent thus has to move under a narrower scope into the ncuc/ncsg and APRALO loses out.
Somehow the belief that only groups can represent the rights of the users is a fantasy. For example, in the technical community and most in ISOC circles, most of the members in a recognized ALS are not actually CS or information rights activists but people with corporate day jobs and of companies that actually should fall under contracted parties ac/oc's in ICANN. The value that comes into such an organizational system are actually the members, the system itself cannot come up with ideas and transform the ideas into workable actions.
Participation collapses when member ideas are not viewed as valuable input and organizations that don't believe in agility to change and adapt to round the clock innovation and improvements basically fall short on rationality for existence and support in the future. If the participation in meetings and tele-cons are an evaluative criteria for calculating value for money, something is seriously wrong with ICANN and the people who are attempting to address such issues by adopting such a weak and lame course of action.
KPIs are measured against activities and participation in telecons and meetings are not such an input or expected outcome. The indicators can be established on a the basis of number of members present and number of comments, suggestions and recommendations received during any calls/meetings and the actions were taken on them. The second level would be number of policy comments requested by ICANN and the number of members that voted on them, the time taken to make comments or recommendations, total number of policy requests for comments and number of comments made to policy requests.
Its actually a very simple and straightforward thing. List down all the things that ALAC and its members do. Against those activities a scorecard can be developed and bench-marked against other AC/OC work. I'll think more about this.
Rinalia Abdul Rahim (27/11/13)
Hello everyone.
I like Ali's input on metrics about leadership of Working Groups for ALAC members. I would also suggest co-chairing of WG between an ALAC Member and an At-Large community member. This will help build collaborative skills and capacity along the way.
Best regards, Rinalia
Ali Al Meshal (27/11/13)
Good day Maureen,
First of all we have always to but in mind as your correctly stated that this is a volunteer work from the members, but on the other hand I do agree that up to certain extent there should be a fair KPI’s in place to evaluate the performance of ALAC members.
Also I am quite sure that selected or nominated ALAC members for these leadership positions were based on their knowledge , commitment and experience as well as high performance otherwise they will not be part of the team.
Hence measuring criteria would always be much clearer and effective if it is set for Quantitative factors rather than Qualitative once.
So having said that then I would like to address the following:
o Attending meetings by it self is not a goal or objective therefore it can not be part of the measuring criteria independently.
o Also participation and contribution to the meeting discussion can not be evaluated as a performance measure
Thereafter to have a process in place that gives an indication of the member performance I would suggest the following
o ALAC members should be assigned to Chair a WG’s
§ With this type of assignment the member will logically be accountable and responsible of a set deliverables that he/she have to achieve, and this can be measured and evaluated for the following set of KPI’s
- Time to deliver
- Commitment
- Team working
- Developing others (member of the group)
- Others
o ALAC members and chair of RALO’s jointly should be responsible for engaging the ALS’s in work and activities related to the rejoin at least and should provide a monthly call report on this.
§ This is different activity from the RALO monthly call; this should be done separately to reach out the ALS’s through a pre-plan agenda and time frame.
§ The objective is to get in touch with ALS’s members outside the official call and tries to understand their needs and requirements and also will be a good tool maybe to get them engage in if they are not active. So this is more of OUTREACH on a small scale and more of direct communication.
….Should you need any clarification then please let me know by email or a call, I will be more than happy to discuss.
From Karaitiana Taiuru (25/11/13)
These are my thoughts based on not been an ALAC member (nor do I have time to).
I would support a more closer monitoring approach to the ALAC performance review and would like to see the evaluations distributed to the relevant RALO's as we are the ones who nominate and put our trust in our representatives to provide our views and then to report back to us.
I would expect that participation in all meetings, emails and Work Groups would be at the very minimum 90%. There are so many issues at present it would be hard not to have a voice at a meeting.
Saying this, I would also expect that any new ALAC appointments are mentored and giving an appropriate transition period and were made to feel comfortable to ask for help. Some cultural and language barriers may also exist and should be considered.
If people are under-performing, then perhaps a mechanism of someone talking to them in a non threating manner to see if there is any assistance required etc.
Perhaps too, some way of recognition for the long hours and dedication may also be a motivation. This is likely to be more of a RALO initiative though.
From Winthrop Yu (24/11/13)
I'd tried to find people here (even outside ISOC-PH) who could actively and constructively participate in the various WGs. Unfortunately, no takers so far, and i really don't like the idea of simply "tapping" someone who may not be as committed to the work or drop the ball. For myself, i'm already committed to policy issues on the local front and cannot honestly make any further commitments until ... oh, after December 2014? :)
Comments are on the workspace at
