I propose to take Questions Nos. 231 to 234, inclusive, together.
The decision to adopt a model of Calculated Grades by my Department was a result of COVID-19, which prevented the State from running the conventional Leaving Certificate Examinations this year. The statistical model underpinning the system of Calculated Grades is a complex and sophisticated bespoke development. Developing a brand-new system for providing examinations results to students in the circumstances of not being able to hold examinations was always going to be hugely challenging. The system was developed specifically for the Irish Leaving Certificate, and within an extremely tight timeframe in order for students to get their Calculated Grades results in time. In light of the intensity of the timeframe for this work, the code was being written, developed and refined on an iterative basis.
The code was written for the system of Calculated Grades by Polymetrika International Incorporated based in Canada. The principal of Polymetrika, Fernando Cartwright is considered an international expert in the field of statistics and psychometrics, the science of educational testing. This was a bespoke development in response to the inability to run the 2020 Leaving Certificate.
Technical details of the Calculated Grades model and standardisation process were published on the date of issue of the Calculated Grades results and are available at https://www.gov.ie/en/collection/90d4a-calculated-grades-document-library/.
The code operationalizing the algorithms in the analysis used a combination of custom-developed ECMAscript, standard numerical and statistical libraries, and SQL. The algorithm implementation used custom-developed code. In all, some 50,000 lines of code were contained within the complex statistical estimation system required to operationalise the standardisation process.
Following the discovery of errors in the way in which junior cycle data had been extracted and used within the model I (the Department) contracted with Educational Testing Services, the world-leading US-based non-profit organisation which specialises in educational measurement, to provide an independent expert opinion on the adequacy of the coding underpinning the standardization process, so as to provide a level of reassurance that students’ results are as intended by the standardization process. ETS evaluated the corrections made to the code and determined that the data extraction works as described in the technical document. It is the professional and expert opinion of ETS that the revision is now correct.