Leaving Certificate 2020 Calculated Grades: Statements

I am grateful to the Business Committee for scheduling this opportunity and this session so quickly. Yesterday was a very difficult day for the leaving certificate class of 2020. The announcement that was made has inevitably generated much anxiety and worry for them. While the announcement yesterday was important, we could not provide all of the answers to their questions at that time. I apologise sincerely once again to the leaving certificate class of 2020 for what has happened.

Deputies will be aware that two errors in the calculated grade system for this year’s leaving certificate have been identified. They are errors that should not have occurred. I am very clear on that point. As part of work being done for the leaving certificate applied fifth year cohort, an inconsistency in how the data was performing was detected by Polymetrika International Inc., the contractor for the calculated grades process. Polymetrika contacted the calculated grades office on Tuesday evening to notify it that the inconsistency had been identified. It was agreed immediately that Polymetrika would seek to investigate this anomaly further and report back to the calculated grades executive office the following morning. This report was received by the calculated grades executive office on Wednesday morning and identified that Polymetrika had made a mistake in the writing of code for the standardisation process.

On Wednesday afternoon last, I was told that a mistake had been identified by Polymetrika. At that point, we knew that one line out of 50,000 lines of code had a mistake in it. We knew that mistake would impact on the results of some students but we did not know exactly how many would be affected. We knew it was important to find out as much as possible about the error before making an announcement. Polymetrika and the calculated grades executive office then began a detailed analysis of 50,000 lines of code, affecting more than 400,000 exam results.

The system was meant to take a number of factors into account in computing the leaving certificate results. Among these were the results of the students' junior cycle examinations, which were considered at an aggregated class level. It is important to say that the junior cycle results of individuals were not used to predict or influence their leaving certificate results. The data was to be used at an aggregated class level. The system was meant to draw on the core subjects of Irish, English and maths and combine them with students' two best non-core subjects. The coding error instead combined them with the students' two weakest non-core subjects.

In the course of the review which the calculated grades executive office immediately undertook on Friday, staff found a further error in the code. The results relating to civic, social and political education, CSPE, were meant to be disregarded by the system. They were not. They had been included in error. Following discovery of the second error, the calculated grades executive office undertook a detailed walk-through, taking every parameter and rule underpinning the standardisation model and verifying, in conjunction with the contractor, that it was working correctly.

While nothing further was found as a result of this exercise, I wanted a further level of assurance. As such, I instructed my Department to seek an independent expert to review the code, most importantly of all, to give our students the assurance they would now require. My Department has engaged Educational Testing Service, ETS, to provide a review to offer an independent expert opinion on the adequacy of the coding. Furthermore, it is my intention that a full review of the calculated grades process will be conducted in due course. ETS is a US-based not-for-profit organisation that conducts research and develops assessment programmes such as the scholastic assessment test, SAT. ETS is one of the largest testing and assessment services in the world. It develops and administers more than 50 million achievement and admissions tests each year at more than 9,000 locations in the United States and 180 other countries. We expect to have the outcome of ETS's work as soon as possible and I will make that information available at the earliest opportunity.

While we do not yet have the final figures, and we will not have them until the independent review which is under way has been completed, our checks to date indicate that the error has affected approximately 7,200 grades. There were over 400,000 calculated grades in total issued on 7 September. As a result of both errors, approximately 6,500 students received at least one result which was one grade lower than they should have. That has been rectified.

In addition to the students who received lower grades than they should have in this year's leaving certificate, some students received higher grades. They will not be affected in any way; their grades stand. Those who received lower grades will have their proper grades restored. When all the grades are completed, we will issue the corrected results to the students affected. As soon as it is possible to do so, every student will be contacted by text message and informed whether they are impacted upon. Those who are affected will be directed to the calculated grades student portal, where they will find a new statement of provisional results, which will replace the one they received on 7 September.

I would have liked to have had complete and final answers to all of the issues so that I would not be leaving students with questions once the issue was announced. I would also like to have had the certainty that everything had been cross-checked but that would have taken further days. We were conscious that the Central Applications Office, CAO, was issuing round three on Thursday and might need to incorporate what our analysis had revealed in their plans. The decision was therefore made to inform CAO on Wednesday morning of what we knew at that point and to call a press conference at 4 p.m. that day. Furthermore, a dedicated helpline for students was set up within my Department. This came into operation at 4 p.m. yesterday. All students received a text message yesterday advising them that an issue had arisen with the calculated grades. The Opposition were briefed in advance of the press conference.

The information I have outlined here is available to students on gov.ie/leavingcertificate. The dedicated helpline for students is available on 01 889 2199 and is open today and tomorrow from 8.30 a.m. until 5 p.m. and on Saturday and Sunday from 11 a.m. until 4 p.m. In addition, the National Parents Council Post Primary, in conjunction with guidance counsellors, has kindly agreed to reopen its leaving certificate helpline for next week running from Monday to Friday from 11 a.m. to 4 p.m. each day. I understand that approximately 200 calls from students and parents were made to the helpline yesterday afternoon and evening. Approximately a quarter of the calls sought further information on when the review process would be completed and revised results issued. Other topics included whether those with exemptions in Irish had their results properly reflected, when revised CAO offers would issue and a separate issue of when calculated grades appeals would be completed.

The announcement I made yesterday changes things for students. Even if for many of them it improves their situation, it should not have happened. On behalf of the Department of Education and Skills, I apologise sincerely to our leaving certificate students for the situation we are now in and for the upset it has caused. I appreciate that students and their families now want to know what happens next. The results data have now been rerun through the corrected model.

Once the review by ETS has been completed and the results are finalised, the Department will have full information on which students will benefit from improved grades and which specific subjects are involved for each student. The Department will then contact all students, advising them when they will receive a higher grade or grades, or whether they are not impacted. The Department will send a corrected file of student results to the CAO in order that it can work with higher educational institutions to determine if a student is due a new offer and to do everything possible to facilitate his or her admission.

Any student who would have been entitled to a different offer in previous CAO rounds if he or she had received the correct grade on 7 September will receive this offer or a deferred offer as soon as is practicable under the updated results. Equally, if students receive improved offers but would like to defer their places until next year, I would encourage them to contact the institutions concerned in order to establish if deferral is possible. It is important to say that this is in line with the practice that occurs in the appeals process every year.

Where a student who has already started college receives a deferred offer as a result of an upgrade, the student can continue in his or her existing course this year and, in respect of next year, remain eligible for free fees and Student Universal Support Ireland, SUSI, funding as if he or she was starting college for the first time. Any student who had applied to a higher educational institution outside the State who believes his or her grade change will impact upon those arrangements can contact the Department through a dedicated email address, lc2020@education.gov.ie.

Students continue to have the option to register for the postponed written examinations. These will commence on 16 November. The closing date for this registration was to be tomorrow at 5 p.m. I have asked the State Examinations Commission to extend that registration deadline until next Wednesday.

I acknowledge that students have had an exceptionally difficult year and they have coped unbelievably well with the challenges posed by Covid-19. As was confirmed by the Taoiseach to the House yesterday, we will do everything to support students who now have the chance to change their course and might wish to do so.

I thank the Minister. We will move to contributors from Sinn Féin. Deputy Ó Laoghaire is sharing time with Deputy Conway-Walsh.

The leaving certificate class of 2020 has been through a terrible ordeal. Every year the leaving certificate is difficult but no other leaving certificate class has had to contend with a pandemic, the postponement and cancellation of exams and the ongoing chaos of the calculated grading process.

Sinn Féin expressed serious reservations about calculated grading from the beginning. We argued for a different approach that would have focused on allowing as many students as possible to achieve their first choice course at third level and university-based assessment in courses with high competition. We communicated this in a detailed proposal to the then Minister, Deputy McHugh. When the Government was determined to stick with calculated grades, it became our priority to ensure maximum transparency and minimise the risk as much as possible. That is why we called for all the details of the model to be published before the results were released. I cannot help but wonder whether, if the Minister had published the algorithms much sooner and been transparent and open about the grading process instead of keeping it under lock and key, some of these issues might have been identified a lot sooner and the consequences less severe. Even if the Minister had communicated to us these errors when she learned of them seven days ago, it could have made a big difference to students who were waiting on round two CAO offers and to students who signed leases for student accommodation in the past week. The Minister knew that the consequences for many students would be massive. Why did she keep this information to herself and why did the Taoiseach do likewise?

I also want to ask how Polymetrika came to take up this job. We now know that it was not subject to tendering. We know that the company was well paid. It was meant to cost €71,500 and cost €163,000 in the end. How did Polymetrika come to be selected? What positive factors led to the Department choosing the company? Was it chosen because it was the cheapest option? This is a back-and-forth, question-and-answer session.

Will the Minister deal with those questions?

The Deputy asked when the information was disseminated. As he is aware, the anomaly was first discovered on Tuesday of last week and the calculated grades office was informed. All that the office was informed of at that stage was that there was an anomaly. It was worked on through the night and on Wednesday morning an official of the Department was informed by the calculated grades office and the Secretary General was informed in the morning. I was then informed on Wednesday afternoon. At that point, what we knew was that an issue had arisen. That issue was an error in the code. Information about how many students it impacted, what its scope and potential might be, and who the affected students might be was not available. It was necessary that further and continuous checks be run to ensure that we had all of the information available so that we could make it available, as I accept is necessary. Following further checks on Friday, a second error was identified. At that stage, I sought and received an opportunity to source external expertise and oversight on the errors that had arisen. That was secured through a company called ETS from the United States, whose staff are international experts in the field of statistics, psychometrics and educational testing. Those checks are still running and theirs are the third pairs of eyes, if you like, on the process, along with the contractor and the calculates grades office.

We were aware that CAO offers were becoming available on Wednesday and we made the decision at that stage, as I have outlined, to release not the full information, because we still do not have it, but the information we had at that point. When we did not have the full information, it was impossible to make the information available. I wish there was more information available but we did make available on Wednesday, as soon as we had concrete data, the information that we had at the time.

The Minister has announced an external review and appointed ETS to conduct it. The company is entirely independent of the Department and this process. A paper from the national standardisation group to the independent steering group and the programme board stated that Dr. Kentaro Yamamoto was appointed in an expert role, which included a level of contingency cover for Polymetrika, as technical adviser on the standardisation process and to provide expert advice and oversight regarding data integrity. He was part of checking the data integrity in the process that the Department set up. Am I right in saying that Dr. Yamamoto is an ETS employee, the same ETS that is essentially externally reviewing the calculated grades process? Does the Minister see a problem with that in terms of a conflict of interest?

It is my understanding that Dr. Kentaro Yamamoto is a retired individual. ETS is a not-for-profit company with more than 1,000 employees. The doctor worked with that company at one stage but is a retired individual.

Will the Minister explain how Polymetrika came to take up the role and how it was chosen?

It is my understanding that before any decision was taken to postpone the State examinations, Polymetrika had been engaged by the State Examinations Commission to advise on calculated grades. That was part of the State Examinations Commission's contingency plans. The decision to postpone the examinations was taken by the then Minister on 8 May and calculated grades was the process with which it was chosen to move forward. In order for the calculated grade model to be of value to students, it was known that results had to be issued by early September to make Irish and international deadlines for entry to higher and further education. There was no time to run a normal, full procurement process, as I understand it. The area of psychometrics and field of educational testing are highly specialised.

I thank the Minister.

I confirm that it is my understanding that, on that basis, it was agreed in May that the Department would avail of the procurement process known as the negotiated procedure without prior publication. This is used in circumstances-----

I thank the Minister. We must move on to Deputy Conway-Walsh.

-----where there is a cause of extreme urgency.

I thank the Minister.

I cannot believe that the Minister did not have this matter discussed at the Cabinet meeting. Will she confirm that it was not mentioned at any stage of the Cabinet meeting?

Whose decision was it not to discuss it at the Cabinet meeting?

Cabinet meetings are confidential. We cannot have discussions here about Cabinet meetings. We cannot ask Ministers about Cabinet meetings. We have a constitutional protection.

Okay. It is a question people want answered. I will put it in another way. People find it absolutely incredible that the Cabinet sat around collectively and did not think this was serious enough to discuss.

The Minister.did not even get the basic data right. We have to remember that an awful lot of pressure was put on teachers and schools to submit all of the results before the end of June. The Minister has had all of that time since. There was a further delay to get this right and rerun the model. The basic constant piece of data in the model was not right and that is what people find quite incredible.

There has been collective Cabinet responsibility for this mess and the students are the victims. The anguish and uncertainty that the Minister has consistently released on students since May is completely unacceptable in terms of delays, accommodation, costs, fees and all of that.

Is the Minister sure that additional places will be made available when it was already stated that could not be done for leaving certificate students from prior years? They have been left out of this situation. What is happening with the legal process? What if the legal process means that even more additional places have to be made available?

The Minister is admitting to two mistakes. Is it possible that there are more mistakes than that? Has the Minister looked at the North in terms of capacity in the universities and colleges there?

I am afraid the Minister has just 30 seconds.

She did take some of my time.

To clarify, the seriousness of the situation is not in any way underestimated. I would say to the Deputy that at all points I acknowledge that the primacy and principle in the situation is and has always been the students and their best interests.

In terms of not getting the basic detail right, in an ideal world there would be no mistakes and no errors. We do not live in an ideal world. Therefore, mistakes and errors can occur. It is very important that when an error occurs or a mistake is made, every possible proactive action that needs to be taken to correct that error is taken, and that is what is occurring now.

Everybody in the House would accept that the Minister has had a lot on her plate. She tried to reopen schools and deal with a lot of inherited messes on her desk. Nobody is suggesting for a second that all of this is her fault.

There are things she has to address in the coming time. She has to address too-large class sizes in primary and secondary schools. A teaching union is balloting for industrial action because of the nature of what is going on in second level schools. She has to sort out pay inequality. She has to provide proper pay and conditions for school secretaries. Her Department has to begin to deal with special needs assistants in a more respectful manner and vindicate the rights of children with special educational needs to get a school place. There is a huge amount on her agenda.

In terms of the leaving certificate, from the beginning people in opposition had strong views on this and were not listened to. The then spokesperson on education, and current Minister of State, Deputy Thomas Byrne, said in a tweet that the written exam should be cancelled. We in the Labour Party said from the beginning that we did not agree with that assessment because we were completely unconvinced that a proper, fair and robust manner could be found to replace the leaving certificate in such a short amount of time.

From the off we said that school profiling was a bad idea. It should have been deleted by the Minister's officials immediately but it was not. To her credit, she recognised it was wrong after the debacle in the United Kingdom and deleted it at the last second. That timeline is part of the problem. I had to force out of the Minister in the Dáil an admission that the leaving certificate result date was being delayed until 7 September because of the calculated grades system. I have come to the conclusion that if school profiling had been deleted earlier in the process, the process would have been more robust.

The Minister engaged the services of this crowd, Polymetrika, without a procurement process or tender document. This process was overseen by a national standardisation group and an independent steering committee. A Dr. Janet Brown was appointed as the external reviewer. The Minister relied on the work of Polymetrika International. Apparently, one individual, who is the head of the Northern Ireland Council for the Curriculum, Examinations & Assessment, CCEA, was on the steering committee. At the same time he was apologising for the Northern Ireland debacle, he was signing off on the system in the Republic.

The Department of Education and Skills placed the dreams, ambitions, emotions and mental health of 61,000 young people in the hands of their teachers and then Polymetrika. Guess who we could trust? We could trust the teachers. The body that the Minister employed to do this work obviously made a mistake. Was the system not road-tested immediately? Did the Department not say that this is the first time in the history of the State that we are not having a written examination, and therefore we have to road-test this system?

Some of the commentary from other Government spokespersons was to the effect that mistakes and these things happen. It is a bit like a builder telling someone a mistake with the foundations was made, but 95% of the house is fine. There seems to be an attitude that these things happen, mistakes happen and we can all just move on.

One of the biggest educational decisions ever made was the decision not to go ahead with a written leaving certificate. One would imagine that the momentous nature of that decision would have reverberated throughout the Department of Education and Skills so that there was no possibility that an error could have come into play. Yet, two of them surfaced. After all of this, 6,500 students will have their grades changed and improved.

I have a number of questions. The Minister said, in what was a remarkable statement from a Minister for Education and Skills but is a statement of fact, that there are students who will not be able to attend the course they have legitimately obtained through this system because of an error from the Department of Education and Skills. That is a devastating thing for somebody to hear.

Deputies told the Minister about this timeline. We told her that it was wrong for students to get their school-based assessments and see the differential between that and their final grade after the CAO offers came out. We told the Minister that was a mistake because at that stage it was too late to appeal.

What is the Department planning to do for the cohort of students who may have already set up shop or made a decision to spend three or four years living in Galway, Cork, Limerick or Dublin and now discover, because of the Department's foul up, that their lives will now be completely different? Is the Minister going to say that these things happen, it will only affect them for a year, mistakes happen and that is the way goes?

The Government was told from the off that a system of calculated grades was not a small decision. It is not something that somebody tweeted about and then decided that somebody else could sort out the ramifications. It was a monumental decision. When one includes the fact that school profiling was included and then taken out, the decision was taken so late that the system could not then correct itself in time for the results to come out.

The CAO results came out and students did not know that there were discrepancies between the school based assessment and what the Polymetrika bunch had churned out.

Now we understand that her Department does not trust anybody. It does not trust her Cabinet colleagues or anybody in the education sphere because nobody knew this was coming. Nobody from the third level institutions, the students union bodies or the teacher unions knew this was coming. Obviously, the Department of Education and Skills does not trust anybody. It certainly could not tell us but it could not tell anybody about what was coming.

Let us forget about the political system for a second and focus on the young people and those who are going to be hurt most by this. In the minute or so the Minister has left to respond to me, can she address one central issue? The Minister is a compassionate and capable politician who is going to deliver much good change in the education system over the coming period of time. Many of the things she has done in her short period of time in office have benefited children and young people across the land in getting back into school buildings. I want to work with the Minister to achieve more. What is the Minister going to say to somebody who discovers over the coming days that he or she is stuck in Galway when he or she wanted to be in Limerick or some other place? What practical support is the Minister going to give that person because telling him or her that there was a mistake and the Government is sorry is not going to cut it?

I would not mind being stuck in Galway myself. The Minister to respond, please.

I acknowledge - the Deputy has referred to the whole calculated grades process - that many people had principled objections to it and voiced those, including many teachers, and they put these to one side in what was an extraordinary time when there was a need to introduce an extraordinary measure because of the pandemic. I salute the teachers who made it possible and put aside their objections and did it in the best interests of their students. I fully accept what the Deputy is saying in that regard.

I am conscious of time but I will deal specifically with the question the Deputy raised in respect of students who may have taken up one particular offer and may now, as a consequence, be in a position to receive a higher preference offer, and I will give this issue some context. Last year 17,000 students appealed, 3,000 of whom were upgraded and 600 offers were made. As a consequence of that, all but three of those received offers within that timeframe, or within last year’s academic year. We are going to move might and main, and I know that the Minister with responsibility for higher education has said likewise, to ensure those students will be in a position to receive those offers this year, within this academic year. We will do that to the absolute best of our ability. I appreciate some students have already taken up offers, as may have occurred in previous years as well.

It is important to note that as a special arrangement this year, any student who receives a higher offer, either as a result of taking the exams in November or as a result of an upgrade, will not have to pay any fees, registration or otherwise, and will of course be eligible for SUSI again. It is not an ideal situation which I absolutely know to be the case. I would never have wanted students to be in this position. All of us have an agreed agenda here which is that we only want what is best for the students. They are at the centre of all that we do in the Department of Education and Skills. I have worked all my life in the education sector and I want to ensure, through working with people like the Deputy and the Members of Dáil Éireann, that we deliver for the students. This is not an ideal situation and I regret it.

We are over time now. I call Deputy Gary Gannon.

The comments, criticisms and potentially the frustration I am expressing are not aimed personally at the Minister but rather collectively, at the Government of the day, the Department and indeed the ideology that has left our system of education crumbling at the seams at this moment. I was interested in the Minister’s opening address. Yesterday was again a very difficult day for leaving certificate 2020 students. She rectified the situation at the end when she acknowledged that this leaving certificate year had been an awful year, as a whole.

It would be remiss of me if I did not go through the timeline of what has been experienced by leaving certificate students in 2020 to encapsulate the full horrors of what they have experienced. Schools closed on 13 March. On 19 March the orals were cancelled, although they were due to take place a short time later, between 23 March and 3 April. On 8 May, the leaving certificate was cancelled, although it was due to start on 3 June, and was replaced by a model, including standardisation and historical data. On 16 July an announcement was made that the results would come three weeks later than scheduled. On 1 September the school historical data was removed and on 7 September the results were issued. Yesterday an announcement was made that there had also been a cock-up in the results.

I accept mistakes can be made. What is massively frustrating for me, probably for other Members of the Opposition and for people who have been commenting, asking for changes to be made and highlighting flaws and inadequacies in the system almost since March, was that at every one of those steps and every time we highlighted flaws we were met by expressions of great confidence. We were informed that everything would be okay on the day by both the Minister and her predecessor. It was only at the eleventh hour that an announcement was made that things were going to change, and that the commentary and the suggestions made by opposition parties and leading experts in the field of education, which had been made all the way through, were being acknowledged. That was grotesquely unfair to the students and to the Opposition. If there had been parliamentary scrutiny, some of the flaws that have been revealed today could have been discovered. If governance in this country is to follow the script given by the Department, to show up and cut ribbons and to get a small bit of pork barrel for our constituencies, then it is flawed and unacceptable. In many ways, it has rendered us in the situation we are in.

I have some questions on the leaving certificate process but before I put those, I want to acknowledge that in our education system now, some very dark clouds are gathering, which we can all understand. My colleague, Deputy Ó Ríordáin, touched on these a minute ago. I do not have the confidence that the Department and the leadership there now have the capacity or the ability to address the dark clouds that are gathering on a multitude of fronts, which I want to touch on now.

The Fórsa union showed up at the Workplace Relations Commission, WRC, today to engage in discussions on the school secretaries. It was met by blockages. There were retrograde steps as to the relationship that had broken down-----

The Deputy is moving away from the matter under discussion which is calculated grades. Union issues have nothing to do with this.

We are going to be here in three weeks' discussing this. Can we not pre-empt the problem?

Let us be here then.

This is a problem in our Parliament at the moment. We raise problems, we are shut down, and then we come back and get apologies. This is ridiculous.

I ask the Deputy to please listen. I do not normally interrupt Deputies. We need to focus on the subject matter before us. Union issues are not part of the-----

The subject matter before us is education as a whole.

I beg your pardon, Deputy.

The subject matter we should be discussing is education as a whole.

That is not, however, what we are discussing. Sorry, Deputy-----

What we are going to see in the next couple weeks is schools being closed.

-----but we are discussing calculated grades and the difficulties that have arisen.

When we are discussing school strikes in three to four weeks' time it is right that we should remember this moment.

We are not discussing school strikes.

We should remember this moment in a couple of weeks-----

I appeal to the Deputy, please. With respect, we are not discussing school strikes. We may have future discussions-----

-----when this happens.

-----on school strikes but that is not what we are discussing now.

We will discuss them when they happen then. When Humpty Dumpty falls off the wall, we will all come running.

Deputy, please.

On Polymetrika, what is the Minister’s understanding of this company? Why did it get this role? When I looked it up on LinkedIn today I could find one employee. Please tell me it is not just one person who is in charge of this whole process. How many employees does Polymetrika have and how did it get this contract?

I thank Deputy Gannon and call on the Minister to reply.

On Polymetrika, as the Deputy is aware and as I said earlier, the decision was taken in April to postpone the leaving certificate examinations until July, followed rather quickly on 8 May by the decision by the then Minister to move towards calculated grades. Prior to that, contingency planning was being undertaken as to what might happen if the Minister had to make the decision at that time.

It is important to point out, in reference to previous points the Deputy made about the difficulties and issues around calculated grades, that calculated grades were introduced by the Minister on 8 May but they were first mooted by students and, further to engagement with representative bodies such as teachers, principals, deputy principals, managerial bodies, parents and student representatives, it was an agreed process to move forward with calculated grades. It is important to acknowledge that this was the best view at the time given that we were living in an extraordinarily difficult time. The decision was made to ensure the students, the class of 2020, would have a pathway to move forward in terms of further or higher education or whatever other pathway they would want to choose. In the interests of balance it is very important to mention that.

Polymetrika and its principal, Fernando Cartwright, are experts in a highly technical and specialised field in the area of statistics, psychometrics and educational testing. Polymetrika had worked in Ireland previously and its expertise in statistics and psychometrics is especially in the high stakes fields. Having initially been engaged in an advisory capacity with the technical working group prior to April or May and as part of the contingency planning for the 2020 leaving certificate, it was then engaged to implement and deliver the system when the decision was taken on 8 May to postpone the leaving certificate.

In terms of software design, and I had to check with experts today so some of these terms might be quite technical, it relies on three standardisation processes: clarity of specification with a formal sign-off for same being a responsibility of the client, that is, the Minister's Department; professional coding, in this case by a company that her Department decided was the best in the business for this particular type of business; and, critically, quality user acceptance testing following which the client, the Minister's Department, and nobody else, approved the system. Can the Minister confirm that the specifications signed off by her Department were explicit as to the requirements and the date of that sign-off? Who within her Department approved the sign-off, on what date and at which particular meeting? Will she be able to provide the minutes of that meeting at which the specifications were approved?

It is my understanding that the then Minister signed off on the calculated grades process on 8 May. The exact time and date of the signing of a contract with Polymetrika is not available to me at this point in time. The Deputy will appreciate that it was some considerable time ago but I will make every effort to furnish that to him should he require it. I will ask my Department to do that.

In respect of the aspects he has raised, I am not a statistician or an expert in any aspect of psychometrics but the Deputy mentioned the notion of oversight, quality, validation and so on. In terms of oversight, a series of mechanisms were put in place comprising the national standardisation group; the independent steering committee chaired by Dr. Áine Lawlor, formerly of the Teaching Council; the international external reviewer, Dr. Janet Brown of the Scottish Qualifications Authority; and the Educational and Research Centre, which was running a parallel system. Through that system, validations and checks, etc., were taking place. All of that was put in place and documentation relating to this has been made available on the website. If there is further specific information the Deputy requires, I will ensure my officials make it available to him.

I will start with a question. In addition to the students who received a lower grade than they should have in this year's leaving certificate, some students received a higher grade. How many?

As I said previously, it is difficult to know the finite number of students until all the checks, reviews and balances have been put in place. In terms of the students the Deputy has referenced who will go up by at least one grade, we estimate at this point that we are talking about approximately 6,500. That is an estimate at this stage.

I can only estimate that, when students' grades go up, we are perhaps talking about a similar number but, again, I am not a statistician and I would not have that information fully available to the Deputy until the review is completed.

I am not expecting an exact figure. To be clear, when the Minister says that in addition to the students who received a lower grade some students received a higher grade, she thinks that as many as 6,500 students could have received a higher grade.

I cannot say that definitively but I am going to assume that if the estimate is that that number will go up by one grade, there is every chance that we might be talking about a similar number. When the absolute figures become available they will be made available to the Deputy, the general public and, most importantly, to the students.

Could it be the case that a student who applied for a course was squeezed out of getting that course because students received a higher grade than they should have got? Is that something that could have happened?

To clarify, if a student received a higher grade than was their due, that student retains that grade. We have been very clear that no student will be downgraded. Equally, it is an established practice that where a student is awarded a higher grade as a consequence of an error through no fault of their own, and this would have happened in previous years, they retain that grade. That would be the position now and I think that is only fair.

I understand that that is the established practice. I want to be clear that I am not in favour of any student who got a higher grade being downgraded but in previous years there would have been a certain number of students who received a higher grade than they should have got. It is not an exact figure but the Minister might be talking about a ballpark figure of 6,500. The point I am making to her is that there must be students who applied for courses and would have got their courses if that error had not been made, but were squeezed out of getting their courses because of that error. That must surely be the case. Does the Minister agree that it probably is the case that students have been squeezed out of courses they would have wanted as a result of that error?

I absolutely confirm to the Deputy that there are students who have benefited and whose grades have increased. I reiterate that those students will retain those grades. No student will be downgraded. I can only speak in the absolute here. That is an established practice and it is what I can confirm for the Deputy.

That is not what I am asking the Minister. Perhaps I am not being clear. To be clear, does she accept that it is at least theoretically possible that students who applied for courses and who would have otherwise got the courses have been now squeezed out of those courses because of an error that might affect approximately 6,000 people being upgraded?

In the absolute I know that students have been upgraded. In the absolute I can tell the Deputy that no student will be downgraded and in the absolute I can say to him that it is an established practice. Where there might have been some issue in terms of students getting places that they might not have got if the system had been otherwise, that is very possible.

I put it to the Minister that it is very possible not for one or two people, or even dozens or scores of people, but potentially for hundreds of people or even more. That is an issue that needs to be debated more but I will park it there for now.

It strikes me that it is a standard testing practice, not something done once in a blue moon or regularly. It is a standard practice whereby a sample set of data is taken and run through the system, and then what is got is tested against the expected results. If that were done in this case, it would be discovered very quickly that there was something wrong. Clearly, that did not happen. My question is related to the individual in the Department of Education who signed off on an arrangement that did not involve sample testing of data, such as that described, and the company. How can that be explained? I would just like a brief comment on that before my final question.

I would be very surprised if the Deputy were seeking to identify individuals employed by the Department.

I am not seeking to identify anyone.

I would be surprised if the Deputy, of all people, were looking for that.

To be clear, I am not in favour of identifying any individual. The point I am making is that the Department has signed off on an arrangement, and a company signed up to a deal based on that arrangement. My question is whether sample sets of data were used to test results. Surely that would have shown up in the system very quickly.

On the issue of checks, validation and oversight, I must confirm again that there was a series of layers of such oversight. I referred already to the national standardisation group, the independent steering committee, the international expert reviewer and the Education Research Centre. The Education Research Centre sampled aspects of the code. There was the initial code. Then those concerned were running what I have been informed was a process in parallel to the system being operated by the contractor. Therefore, there were checks and balances put in place. I absolutely accept that it is beyond unfortunate that errors arose. Those errors were found internally, and when they were found, immediate and proactive action was taken. That is important also.

The clock is running down. I have a final question.

It is important to clarify that when an error was identified, it was identified internally within the system. It is regrettable that the errors exist but they were identified internally within the system, and that does speak to the validation, checks and balances that were in place.

In fairness, this is running down the clock.

I am not running it down; I am answering the Deputy's question. It is very important to give the context and, on an issue like this, to give the correct information.

Now there is only one minute and 23 seconds left. My final question-----

The Minister is here to answer everyone's questions. She is dealing with Deputy Barry's also.

We are not in a position to say when the final report and full scope of the situation will be clarified, but obviously this is not a situation that can run for months, or even weeks. Will the Minister give a rough idea as to when we are likely to have a result and how quickly the extra university places needed might be made available when we have the results? The Minister with responsibility for further and higher education, research, innovation and science, Deputy Harris, referred to up to 1,000 places the other day.

Educational Testing Service may have completed its analysis as early as tomorrow. It could be the following day. I am very hopeful. I want this to be as done as speedily as possible, but as correctly as we would all want. Once we receive the information from Educational Testing Service, the calculated grades office will have to complete its work. Thereafter, I would like to see things move as speedily as possible. I appreciate that it is in the best interests of the students that we move as quickly as we can.

To clarify, does the Minister accept that once the number of extra places needed is absolutely clear, it will be necessary for them to be found and filled not in a matter of months or weeks but in the matter of days?

I confirm that, in co-operation with my Department and the Minister with responsibility for further and higher education, research, innovation and science, whose co-operation I acknowledge, we will work closely to ensure that the maximum number of places that need to be made available to those who might receive higher preference offers will be made available.

Serious questions remain and need answers. I thank the Minister for attending tonight to try to answer them. When did Polymetrika find out about this error? How did it find out about it? What safeguards had it in place to ensure its system was watertight? Why did the safeguards not kick in? How much was the company paid for the development of this system? In what way is it accountable as a result of its system not being watertight?

I thank the Deputy. The error identified by Polymetrika on Tuesday evening of last week was identified as the company was working through the leaving certificate applied results, which are based on a two-year programme. These results are generally uploaded after the leaving certificate results become available. The fifth-year results from the subject area were being worked on. At that point, as I said, Polymetrika identified what it classed at that early stage as an anomaly. In other words, it became apparent that there was something not correct. As regards what that was or meant, there was no clarity at that early stage. The calculated grades office was informed. The system ran through the night and the following day work ensued, as I have outlined.

The Deputy asked about the contract and payment. To the best of my knowledge, the original contract was around the €170,000 mark. I will check that and revert to the Deputy. For absolute clarity, I just want to check that.

What was the Deputy's third question?

It was about the development of the system. Had the company checked it out? How was it accountable?

There was, of course, a contract. Obviously, as with any contract, there are penalties. All that will obviously be pursued to the absolute extent. On what the Deputy asked about oversight, I have outlined that there was oversight in terms of the full model.

What checks did the Department carry out prior to issuing the leaving certificate results to ensure they were watertight?

There was oversight of the process from the very beginning. That included oversight by the national standardisation group and the independent steering committee, which was important and which was chaired by Dr. Áine Lawlor, formerly of the Teaching Council. Equally, an international external review was conducted by Dr. Janet Brown, formerly of the Scottish Qualifications Authority. There was a further external review here by the Education Research Centre. There was validation and there were checks and balances. A number of layers were put in place. As I said previously, all the information and data concerning each of these groups have been made available and are available online. The reports they have produced and opinions they have offered are also available.

Yesterday I received a very detailed response to a parliamentary question on how the junior certificate results were taken into account in the standardisation process. The Minister and Department, despite being aware of the error, failed to mention it in the answer, despite its being directly related to my question. Can we not trust the responses from the Department? Why was this not mentioned?

I do not have sight of the specific question the Deputy is referring to.

On Tuesday of last week, when the matter first arose, an anomaly was identified in the first instance. It was only on working through it that the exact nature of the error and the number of students who would be impacted became apparent. Indeed, we are still working on further, complete, absolute, substantial data.

Until the fullest of data is available, we cannot make the fullest of data available. As I said, I am hoping to do that as quickly as possible.

I have concerns about parliamentary question replies in general. Are they accurate and as accountable as they are meant to be?

Yesterday, a text message was sent to students informing them that they had been affected, my own daughter included. Shockingly, the Department got that messaging wrong in that it sent an incorrect link to the students and they then had to be sent a second message. Why could the Department not get this right? We are dealing with the lives of young students. There has been one blunder after another. Why could the Department not get that messaging right? As the Minister said in her opening remarks, this is a very sensitive time for students. I acknowledge that the Minister has apologised. It is a trying time for every student, not just my daughter. That the Department sent the incorrect link was very sloppy and clumsy. It shows a lack of serious care for the students it was addressing.

I thank the Deputy for raising that point. I acknowledge that he is 100% correct that this is an incredibly sensitive and anxious time for the students. At no stage do I want to diminish the anxiety that students are experiencing. I would go beyond that because I know that it is not just the student but the entire family who are drawn into this. I appreciate that all households become involved when a student is taking the leaving certificate examination or any examination. I acknowledge the anxiety that exists. As the Deputy has acknowledged, I have apologised previously and I will continue to apologise for the stress and anxiety that has been placed on students.

On the text message, it was transmitted yesterday to students so that they would be made aware of exactly what was happening. I reiterate that students are the central and principal concern in all of this. I want to abide by that throughout this process. On the specific instance raised by the Deputy of the transmission of an incorrect link, I am not aware of that. I am aware of the text message being sent to the approximately 61,000 students. The issue of an incorrect link has not been raised with me. If that is the case, I am sorry to hear that. I will investigate that with my officials and revert to the Deputy on the matter. I again thank him for raising the issue.

Will the students affected receive their rightful course choice and be able to take up their place this year? If not, those students are disadvantaged? If students must move to another city to take up a place in an alternative university, who will compensate them for the accommodation for which they have already paid? Everyone had to pay for their accommodation last Friday only to be told on Friday evening that they would be taking their courses online. I ask the Minister to clarify that?

If students receive an upgrade, the completed file will be sent to the CAO. The CAO will make higher preference offers to those who are deserving of them at that point. We will move to do that as quickly as possible because I appreciate that it is important students receive those offers as quickly as possible.

I mentioned previously that students who receive an upgrade as a special arrangement this year, either as a consequence of the calculated grade or of taking the examination in November, will not have to pay any fees, registration or otherwise, and they will be eligible for the SUSI grant. On the number of places, I confirm and acknowledge the preparedness of the Minister with responsibility for higher education, Deputy Harris, and his Department to work with the Department of Education, the CAO and the Higher Education Authority to do all that is within our gift to ensure that the maximum number of students receive their offer and placement this year. That is the objective. It is important to reiterate that last year there were 17,000 appeals, of which 3,000 were successful and 600 students were upgraded and became eligible to receive higher preference offers. All of the 600, bar three, received their offer within the academic year last year. It is my absolute objective, in co-operation with the Minister, Deputy Harris, to ensure that, if at all possible, similar success will be achieved this year.

Deputy Barry asked the Minister a number of questions. While she gave the context, she did not actually truthfully answer them. I do not want-----

Please do not say that.

I will tell the Ceann Comhairle the reason why-----

The Minister has truthfully answered any question put to her.

I am not suggesting any lack of truthfulness.

That is the-----

No, I am not suggesting that, absolutely not. I would never imply that, Minister.

The Deputy is clarifying that he did not say that the Minister has not answered truthfully.

The key point of the question was not answered. The context was given and the structures that existed and the people who were responsible for testing was outlined but the Minister did not tell us whether the type of testing that should have been happening did happen. That is the key question that Deputy Barry was asking. There is a piece to this that I am not getting. In response to me earlier, the Minister, Deputy Ryan, said the flaws were spotted during the process of the leaving certificate applied. I am not sure why that was being processed at that stage given that the results were already out but perhaps it was appeals. The bit I am struggling with is that any formula of this kind should be tested numerous times. It should be run on a few sets of data. It is not complicated; there is a formula. For example, a small dataset of, say, a dozen, 20, 40 or more students, is run through the formula and one should know what the outcome is meant to be because one knows the inputs and outputs of the formula and the process that should happen. If it does not produce the results expected, one works back through the formula to identify the problem. That is what should have been happening. It is exactly the detailed walk-through that the Minister says has happened. Given the way that this transpired, it is obvious that was not happening during this process. Unless the Minister can tell me otherwise, it seems to me very plain that that type of testing during the development of this process, formula or code, was not taking place. What am I not getting?

In regard to the Deputy's confusion as to the reason the leaving certificate applied aspect was involved, he will appreciate the leaving certificate applied is a two-year programme and the fifth year aspect of it was being uploaded. That is why that was happening at that time. The Deputy is correct that it was at that time the error was identified. In terms of the processes, oversight and so on, I have said previously, and I say so honestly again, I am not a statistician. I am not an expert in psychometrics or examination testing. Therefore, I can only give the Deputy the layman's version as I understand it. My understanding, without any expertise in this area, is that there was oversight of the process by the national standardisation group, the independent steering committee and the international external reviewer and, most important, the Education Research Centre was running the data in parallel with the process being run by the contractor. That is my understanding of what has happening at the time. An error occurred and it was identified. Something went wrong. We are in this position now because something went wrong. I accept that. I have acknowledged that and once the error became known, positive and proactive action was taken.

If the Minister cannot provide the answer on the floor of the House, perhaps somebody from the Department can do so after the fact. It seems to me very plain that the testing that should have been happening was not happening. The Department employed an outfit that did not test as it should have tested. It bought an application that did not work. That is the reality of what happened. I am speaking not about the Minister but the Department. We are dealing with the consequences of that process at this point.

The key point in all of this is the solution. I urge the Minister to work with the Minister, Deputy Harris, to do all in her power in regard to third level places. It is the only game in town in terms of resolving this matter.

Every step must be taken. People talk about capacity and, obviously, that is an issue. However, it is not just about physical capacity but also staffing in terms of feedback to students. That must be the focus now and I urge the Minister to continue that work.

Thank you, Deputy Ó Laoghaire. That is a very positive note on which to conclude. I call the Minister.

I guarantee that I will work with the Minister, Deputy Harris, as I said earlier. I acknowledge his and his Department's co-operation in working through the current situation. Every effort we can make will be made to ensure that students who are offered a higher preference place will receive it within this academic year. There is an absolute determination to do that.

My opening remarks gave me the opportunity to acknowledge the errors that have occurred in the calculated grades system and, importantly, to apologise for the impact these errors are having. I appreciate having the opportunity to do that once again. I have listened to the concerns of Deputies and also to the comments and questions from those with whom I met remotely yesterday afternoon. The Taoiseach and the Minister, Deputy Harris, are committed to ensuring that we support students in addressing the challenges that lie ahead and when the revised offers are made through the CAO. It is important to remember, with regard to the CAO and what happens every year, that following the examinations appeals process some appeals are successful and students receive revised CAO offers. In essence, that is the same sequence we propose to follow this year. The scale is considerably bigger and that is a challenge.

Almost half of those who received calculated grades received their first preference choice as a CAO offer and 80% received one of their first three choices. Some of these students will receive revised grades. As regards those who receive revised grades, it is not limited to any single group of students or to a particular subject area. There will be revised grades across all subjects and students.

I appreciate the earnestness and sincerity of all Deputies who have engaged in the discussion tonight, of those who engaged earlier with me and of those who engage on an ongoing basis with positive and constructive contributions for the sake of education, to which we are all greatly committed. I appreciate the wisdom, experience and, indeed, the genuine nature of the contributions Members make in the House on all that happens in the education sector.

Creating and building the calculated grades process and supporting systems was a major logistical undertaking. That has been a much-used phrase over the last while but, even so, that does not take from its veracity. The calculated grades process was a direct response to challenges presented by Covid-19 and it was designed and implemented in a very short timeframe. Undoubtedly, it was done in good faith by the Minister at the time when he made the announcement on 8 May last. My focus now is on ensuring the work being undertaken by my Department, Polymetrika International and the external reviewer can be concluded as quickly as possible. I want students to appreciate that all that must be done will be done to ensure that the maximum number of students who are entitled to a higher preference offer will receive those offers.

I conclude by apologising again to the students of 2020 and acknowledging the impact the two errors have had on them. We are working to resolve the situation. I appreciate the resilience, forbearance and patience students have shown to this point. I thank the Deputies for the opportunity to be here this evening.

The Dáil adjourned at 10.25 p.m. until 2 p.m. on Tuesday, 6 October 2020.