Dáil Éireann Debate, Wednesday - 14 October 2020
34. Deputy Mick Barry asked the Minister for Education the testing process to the calculated grade system prior to it being used; her views on whether the tests were robust enough; and if she will make a statement on the matter. [29991/20]
Amharc ar fhreagraThe system of Calculated Grades is a complex and sophisticated system which had to be developed from scratch within an extremely tight timeframe in order for students to get their Calculated Grades results in time. The statistical model itself was developed specifically for the Irish Leaving Certificate.
Technical details of the Calculated Grades model and standardisation process were published on the date of issue of the Calculated Grades results and are available at the following link https://www.gov.ie/en/collection/90d4a-calculated-grades-document-library/
The design of the Calculated Grades model was informed by advice from a Technical Working Group comprising experts drawn from the State Examinations Commission, the Inspectorate of my Department, the Educational Research Centre, and international external expertise.
The integrity, validity and reliability of the process of national standardisation was overseen by the National Standardisation Group whose role was to oversee the application of the statistical model to the school data.
The Department had checks at a number of stages to ensure that data was collected correctly from schools and transferred correctly between the various stages of the standardisation process, the validation process and the grading processes.
In order to make sure that the standardisation process was doing what it was supposed to do, a separate additional process was carried out called validation.
Validation is an inherent element of any statistical system, and in the Calculated Grades model the purpose of validation is to ensure that the statistical model is behaving as expected, and is achieving its objectives. The validation process has been part of the design from the beginning.
The workings of the statistical model were reviewed and validated in a number of ways. There was a review of the distributions of results for each subject and level. There was also a review of the demographic characteristics of the outcomes, which included gender and socio-economic status of the school.
The purpose of the review of the outcomes of the statistical model was to check whether the Calculated Grades model was resulting in any particular group being advantaged or disadvantaged relative to previous years’ outcomes. It is important to note that this assessment was being made relative to previous years. For example, the validation checked whether or not disadvantage effects, or gender effects are being exacerbated under the model. This was to ensure that the model presented outcomes that were as fair and equitable as possible given its constraints, and were in line with previous outcomes as much as is possible.
The purpose of the Calculated Grades system is to arrive at the grade that each student would have achieved if the examinations had taken place as normal. The validation process checked whether the interactions between these characteristics and the calculated results were similar to the interactions in the historical data between these characteristics and examination results.
I regret that a number of coding errors were found in the Calculated Grades process. These errors, which related to the processing of Junior Cycle data, required the process to be run again, resulting in improved Calculated Grades being provided to some students on 3 October. I announced on that date that I had asked that a comprehensive independent review of the design and implementation of the Calculated Grades process should take place when the process is complete. The full scope of that review will be clearly set out in advance.
