Dáil Éireann Debate, Tuesday - 6 October 2020
437. Deputy Seán Canney asked the Minister for Education if an example will be published of the way in which the algorithm used in calculating leaving certificate grades works; the measures taken to preserve accuracy and fairness in the results; the way in which the algorithm was validated; the parameters used to ensure that the marks generated by the system reflected student performance; and if she will make a statement on the matter. [28121/20]
View answerTechnical details of the Calculated Grades model and standardisation process were published on the date of issue of the Calculated Grades results and are available here.
The design of the Calculated Grades model was informed by advice from a Technical Working Group comprising experts drawn from the State Examinations Commission, the Inspectorate of my Department, the Educational Research Centre, and international external expertise.
Schools provided an estimated percentage mark and a rank order for each student’s subjects. The process of national standardisation was then applied to the school information in order to ensure comparability between the standards applied by individual schools and the national standard.
The integrity, validity and reliability of the process of national standardisation was overseen by the National Standardisation Group whose role was to oversee the application of the statistical model to the school data.
In order to make sure that the standardisation process was doing what it was supposed to do, a separate additional process was carried out called validation.
Validation is an inherent element of any statistical system, and in the Calculated Grades model the purpose of validation is to ensure that the statistical model is behaving as expected, and is achieving its objectives. The validation process has been part of the design from the beginning.
The workings of the statistical model were reviewed and validated in a number of ways. There was a review of the distributions of results for each subject and level. There was also a review of the demographic characteristics of the outcomes, which included gender and socioeconomic status of the school.
The purpose of the review of the outcomes of the statistical model was to check whether the Calculated Grades model was resulting in any particular group being advantaged or disadvantaged relative to previous years’ outcomes. It is important to note that this assessment was being made relative to previous years. For example, the validation checked whether or not disadvantage effects, or gender effects are being exacerbated under the model. This was to ensure that the model presented outcomes that were as fair and equitable as possible given its constraints, and were in line with previous outcomes as much as is possible.
The purpose of the Calculated Grades system is to arrive at the grade that each student would have achieved if the examinations had taken place as normal. The validation process checked whether the interactions between these characteristics and the calculated results were similar to the interactions in the historical data between these characteristics and examination results.
The system of Calculated Grades is a complex and sophisticated system which had to be developed from scratch within an extremely tight timeframe in order for students to get their Calculated Grades results in time. The statistical model itself was developed specifically for the Irish Leaving Certificate. The Department had checks at a number of stages to ensure that data was collected correctly from schools and transferred correctly between the various stages of the standardisation process, the validation process and the grading processes.
