Leaving Certificate Calculated Grades 2020 and Preparations for Leaving Certificate 2021: Department of Education

On behalf of the committee, I welcome Dr. Harold Hislop, chief inspector at the Department of Education; Mr. Dalton Tattan, assistant secretary general at the Department of Education; and Ms Andrea Feeney, CEO-designate of the State Examinations Commission and former director of the calculated grades executive office at the Department of Education.

I will shortly ask Mr. Tattan to brief the committee on the 2020 leaving certificate calculated grades, specifically on the calculated grades process; measures that are in place to review the process and to indemnify students who were wrongly graded; as well as on preparations for the 2021 leaving certificate. The format of the meeting is that I will invite Mr. Tattan to make a brief opening statement, which will be followed by questions from members of the committee. Witnesses are probably aware that the committee will publish the opening statement on its website following the meeting.

Before we begin, I want to remind members of the long-standing parliamentary practice to the effect that they should not comment on, criticise or make charges against a person outside the House or an official either by name or in such a way as to make him or her identifiable. I would like the officials to note that they are protected by absolute privilege in respect of their presentations to the committee. This means they have an absolute defence against any defamation action for anything they say at the meeting. However, they are expected not to abuse this privilege and it is my duty as Chairman to ensure that this privilege is not abused. If their statements, therefore, are potentially defamatory in regard to an identifiable person or entity, they will be directed to discontinue their remarks. It is imperative that they comply with any such direction.

I ask members to switch off their mobile phones. I ask Mr. Tattan to make his opening statement and I will cut him off after six minutes because we want to finish the meeting swiftly at 3.25 p.m. as members have to be in the Convention Centre for 4.00 p.m.

Mr. Dalton Tattan

I thank members for the invitation to engage with the committee on the 2020 leaving certificate calculated grades process and on preparations for the 2021 leaving certificate. I am assistant secretary at the Department of Education with responsibility for curriculum and assessment matters. Attending with me are Dr. Harold Hislop, chief inspector in the Department of Education; and Andrea Feeney, CEO-designate of the State Examinations Commission and previously director of the Department’s calculated grades executive office.

The Government decision of 8 May 2020 to postpone the leaving certificate examinations and to adopt a model of calculated grades was a result of the Covid-19 pandemic, which prevented the State from running the conventional leaving certificate examinations this year. The Government decision was made following an assessment of public health advice, advice from the National Educational Psychological Service and other likely implications for holding the examinations from the previously rescheduled date of Wednesday, 29 July 2020. Critical to this decision also were the strongly expressed views of the advisory group for contingency planning for State examinations 2020, comprising all stakeholders, including students, teachers, school managers and others. This advisory group met frequently from April and it shared concerns that it was not possible to hold the conventional examinations safely.

It was decided that leaving certificate candidates would be provided with the option of receiving calculated grades and certification that would allow them access the Central Applications Office, CAO, system, higher and further education, and the world of work. Candidates would also retain the right to sit the conventional leaving certificate examination as soon as it was safe for the State examinations to be held. These written examinations will commence next Monday, 16 November, when approximately 2,800 candidates will sit 7,300 examinations in 600 schools and other settings.

The leaving certificate class of 2020 faced a unique set of circumstances and the decision to provide students with calculated grades was taken with the best interests of students at heart and in full consultation with the partners in education. The design of the calculated grades model was informed by advice from a technical working group, comprising experts drawn from the State Examinations Commission, the inspectorate of the then Department of Education and Skills, the Educational Research Centre and international external expertise.

At all times, the calculated grades system had concern for this year’s group of leaving certificate students who completed their second level education under the most unprecedented and difficult circumstances. Through the calculated grades process, the Department sought to ensure that the grades students received are fair and comparable representations of their levels of achievement. Every effort was made by the Department to provide calculated grades to as many students as possible, provided there was credible and satisfactory evidence from an appropriate source and on which an estimated percentage mark could be based. Calculated grades results were issued to more than 60,000 students on 7 September 2020, ensuring deadlines for the CAO, the Universities and Colleges Admissions Service and other jurisdictions’ college entry systems would be met as far as possible.

Very regrettably, a number of coding errors occurred in the calculated grades model. On 3 October, the Minister for Education, Deputy Foley, announced that 6,100 leaving certificate students would later that day receive details of improved calculated grades following the correction of those errors. As part of the round 4 offers made by the CAO on 8 October, 485 of these students received a CAO offer on foot of their improved grades. Each of them was given the opportunity to take up his or her offer in the current academic year. The Minister also announced that she had asked for a comprehensive independent expert review of the design and implementation of the calculated grades process to take place when the process is complete. Some initial scoping of the review has taken place and legal advice is currently being sought, having regard to certain litigation in being.

On 21 August, the Minister announced a series of changes that will be made to the assessment arrangements for the 2021 State examinations. These changes are detailed in the assessment arrangements for junior cycle and leaving certificate examinations 2021 which forms part of the roadmap for the full return to school. The arrangements are designed to take account of the disrupted learning experienced by students during the 2019-20 school year and to factor in possible further loss of learning time in the 2020-21 school year as a contingency measure. As the loss of learning through school closures will have affected students’ engagement with their course of study in various ways, the adjustments put in place will play to students’ strengths by leaving intact the familiar overall structure of the examinations, while incorporating additional choice. Project briefs for relevant subjects will be issued earlier than usual and practical course work will be submitted earlier than normal.

The preference of the Department and all stakeholders would have been to hold the June examinations in 2020 for several reasons, not least of which was that students are familiar with the examination format and the format of associated practicals and tasks. That is why the Department and the Minister, Deputy Foley, are determined to run conventional examinations in 2021 and planning is progressing for the running of the 2021 leaving certificate examination.

My colleagues and I are happy to answer any questions committee members may have.

I thank Mr. Tattan.

Mr. Tattan, Dr. Hislop and Ms Feeney are very welcome. The committee appreciates their time. This is a very important topic. It is also an emotive one. I am thinking of the 60,000 students who are preparing to take the leaving certificate in 2021 and the 60,000 who completed their leaving certificate in 2020 but still have questions regarding the process that was in place. I have had several conversations with teachers, principals and pupils and have received several emails in recent days on this issue. I believe I owe it to those who contacted me to raise the particular points they made. The leaving certificate students who contacted me were incredibly articulate in terms of defining their issues and anxiety and they were very clear about what they wanted for themselves and their peers. I was struck by their contributions. I will tabulate those contributions and send them to the Department because all that correspondence needs to be read.

The particular concerns of the students who are to take the leaving certificate in 2021 are around the loss of class time and the content overload. I know the Department has made some changes in terms of the curriculum that will be assessed in the leaving certificate examination but, having listened to teachers and students, I contend that is not enough.

There is now a slow-down correction process for homework and tests and that adds to it as well. There is a lack of access to laptops. Perhaps there are a number of students within a family when there is only one laptop at home, which is causing difficulties.

There are difficulties about being back at school in terms of masks and the lack of lockers. I know we are all conscious of this. There is no more group work in language classes. Again, there are concerns about the orals. Will they be held? Will they be held in Zoom format in a one-to-one context? There is definitely an increase in stress and anxiety since the return to school in September. We must address the mental health of students and ensure they have positive mental health. The fact that they cannot participate in other activities such as gym or sport is definitely affecting them. There are students who missed additional time due to Covid cases in their school or family and they are particularly concerned about that. Other affected areas included are specific subjects such as field study for geography. The agricultural science sample paper has only just been released. The details of a new research project for economics to be completed by December were given this week. The practicals in science and music are also affected.

I received a number of emails, particularly from parents and students who were downgraded. There is concern about that and the November exams. Some people felt that they should be cancelled and held in June. In general, teachers were very positive about being back in school and everything they could do but there was a feeling that there was a lack of consultation with school leaders around some aspects of it. Again, they are concerned about increased anxiety and mental health concerns.

The very specific questions I have are around responsibility for the errors found. Does responsibility lie with the Department or Polimetrika? Was the model tested before it went live? Can the witnesses from the Department outline what needs to happen to ensure the model is fit for purpose? I appreciate that the witnesses said that the Department is planning to hold conventional exams in June. How will they comply with public health and social distancing measures? If students are out due to Covid, will there be a second set of exams during the summer?

If the witnesses do not have time to answer in full, they can send a written reply to the Senator.

Mr. Dalton Tattan

Regarding the format for 2021, planning is under way through the State Examinations Commission. We know there have been calls for certainty around the exams for 2021. Obviously, with the pandemic, it is difficult to have certainty about anything. As we know, every day has a different challenge but we are trying to bring about that certainty. We absolutely want to have exams in 2021. We need to engage with the stakeholders on that because in order for that to work, it must be sustainable for schools to be able to run. That includes the orals.

Very unfortunately, there were a number of errors. The Minister has apologised for that and I reiterate that apology. It caused a lot of stress. The Senator spoke about student anxiety. The errors occurred in the coding but it must be recognised that the degree of pressure everyone involved was under, including the company doing the coding for us, over a very short period of time was extreme. There were quality assurance checks. We may have an opportunity later on for Ms Feeney to talk more about that. Those are probably the main points relating to that but we will follow up on the others.

I thank the witnesses for attending this afternoon.

We all know the Department was left in an invidious position earlier in the year regarding the public health crisis and the exams. It has been my view, and I communicated this to the former Minister for Education and Skills, Deputy McHugh, that using calculated grades was not the right approach. Unfortunately, students who worked extremely hard and effectively gave up a year of their lives did not get the results for which they were hoping. They were able to look at the teacher grades, however, and see that the results they received were not what their teachers thought they deserved either.

That was exacerbated by the error in the coding. I still find it difficult to fathom how that error managed to slip through, given that this process was meant to have been reviewed and tested. Before Mr. Tattan answers, it is important we acknowledge that two Departments and multiple agencies were involved with last year's leaving certificate examinations and the fallout from that process. I hope, therefore, that Mr. Tattan will acknowledge in his answers that there is still a responsibility on the Department of Education to make this good for those students who were failed by the coding error and by the model as a whole.

I have a million questions, but I will send most of them in an email. I will focus on just three. Some 8,000 higher grades were issued. Does Mr. Tattan acknowledge that there was a ripple effect for other students? The CAO is a system which ranks order in allocating courses. There will be a round of that process in February after the leaving certificate examinations in November. I raised this question with the Minister for Further and Higher Education, Research, Innovation and Science, Deputy Harris, as well. Have there been discussions between the two Departments regarding whether it would be possible to provide deferred offers to those students who lost out on courses because they were pushed out by those with higher grades? Is it possible to undertake the mathematical exercise of excluding the error and running not the entire CAO process, which has already allocated places, but a process to figure out what points those students affected would have had if the error had not taken place and if they can now be offered deferred places? I think that would be a way of making things good for those students who worked extremely hard, who may have lost out because of the error and who were looking at their teachers' grades and wondering how it went wrong.

The November examinations will have a relatively small cohort, some 2,800 students. It is an atypical cohort. From my experience, these are people who have missed out on courses with high points and who are now taking the earliest opportunity to try to remedy that situation. Most students are taking one or two subjects, and not all. It would seem perverse to apply the usual standardisation, the bell curve, in that situation. There is no way it can match the standardisation of previous years. I hope that will not be the case, but perhaps Mr. Tattan can enlighten me.

A practical issue already rising and causing issues concerns those students who have already missed out. I refer in particular to those interested in studying medicine because that is a competitive and hard area. As I said, some of what has happened was because of the model and the errors. The health professions admission test, HPAT, to get into medicine is already demanding enough. Considering the fallout from this situation, therefore, surely the Department of Education can have some discussions with the universities and the other Department to ensure that the results from this year's HPAT can be carried over. I think that is an important remedy.

Mr. Dalton Tattan

On the ripple effect, there certainly were some discussions on the issue at a high level between two Departments, particularly during the time when the coding errors were discovered. The difficulty lies in trying to do what the Deputy referred to without having to run a full CAO process. The view we eventually came around to was that it was not possible to do that without a full CAO rerun. The Deputy referred to ripple effects, and those also occur from round one to round two etc.. People may turn down an offer in round one and that offer gets passed on to somebody else in round two, and so on. Trying to rerun the process, therefore, without involving everybody seemed to be impossible to us. The view coming from the higher education wing as well is that what the CAO takes every year are the leaving certificate results, which is effectively what the State Examinations Commission in a normal year-----

I will clarify that what I have in mind is something to the effect of establishing the points that would have existed without the error. The students who achieved those points could then be allocated those courses.

Mr. Dalton Tattan

I do not think it is possible to do that without a full rerun, because without doing a full rerun one would not know where the points would have landed. As part of the process, the examination results are issued in their entirety. That file is handed over to the CAO, which does an initial pass on it, and it is then sent to the higher education institutions, and based on that, offers are made about a week later. It would be a question of trying to do it without involving everybody in it. Some parents asked whether the first round of offers could be run again for those who just missed out in the ripple effect mentioned by the Deputy, but that did not seem possible to us. However, we have explored it and I am happy to discuss the issue in more detail with the Deputy.

I will pass the question on the issue of standardisation and the November examinations to Ms Feeney because she is directly involved in leading on the examinations that are being held next week.

Ms Andrea Feeney

As the Deputy has acknowledged, this cohort of students is very different because it is small. These examinations are being undertaken by a disparate group of students, including those who were disappointed with their calculated grades and those who did not initially have an opportunity to get calculated grades because of their circumstances. There is a small group of students who did not want calculated grades and opted out of that process. Their perspective was that they wanted to sit their examinations.

The normal standard-setting approach that we follow is a combination of the marking schemes matched to papers, which have been set in advance; the statistical norms from year to year; and the responses of students. We do not have that this year. Even in the biggest cohort subject, which is biology at higher level, it is not statistically significant. We have appointed very senior and experienced examiners to mark these examination papers. They will be bringing their professional expert judgment to the exercise of marking to ensure that students are appropriately rewarded for the efforts they make in these examinations, and that the outcomes are fair and valid. That is the approach that we are taking. We are not looking at a statistical norm, but at the expert judgment brought by the examiners.

If it is the case that there is no adjustment whatsoever after that, I will be happy but I want to clarify that is what Ms Feeney is saying.

Ms Andrea Feeney

It is through that process, so we are looking to make sure that the outcomes will be fair and valid and that students will be appropriately rewarded.

I welcome the witnesses. We have the expertise of an assistant secretary, a chief inspector and a CEO-designate of the State Examinations Commission. I believe Ms Feeney worked in the Department's calculated grades executive office.

We are here today to discuss the calculated grades process and the measures that are in place to review it, the indemnification of the students and the preparations for the 2021 leaving certificate. I understand that this year was very challenging. I pay tribute to what has been delivered for students. I know that has been done with help of principals and teachers. Students and teachers have worked together. I appreciate that the organisations engaged with stakeholder groups in developing this system.

As someone who comes from a project management background, I am concerned about how this system contained errors to such an extent that over 6,100 students were affected by coding errors. Polymetrika found that Irish, English and maths were being used with the two worst subjects, instead of the two best subjects. When the Department went through the system at a later stage - maybe this should have been done earlier - it found that civic, social and political education had been left out. Finally, the educational testing system reviewed it and found another two errors. If students had not sat those three subjects and instead took the next best subject, instead of an average, that was also an issue.

I am really shocked here. From a pure project management perspective, I wonder how on earth was it not tested. Was €75,000 the amount that was spent? Does the Department think that was sufficient for 60,000 students? It appears quite low to me. What other costs were there after that? What section of the Department had oversight and governance of the systems here? Of the 485 students who were offered a place through CAO, how many of them have taken up those offers? I will stop there and follow up with questions afterwards.

Mr. Dalton Tattan

I will start and then will let Dr. Hislop respond. There were three errors.

There was a further issue. The Educational Testing Services characterised the fourth as an issue rather than an error but it was something that was at slight variance with what was described but did not have any impact on students. It was something that we sought to clarify with them. On the three errors, obviously it was shocking. It was certainly shocking to those of us involved to find that out at that point in time.

All of the errors occurred in the small section of code. That is not in any way to reduce or excuse them or not talk about the impact because the impact, for the students involved, was significant. We did everything we could to resolve that for the students and in as timely a fashion as possible. They all related to one small section of code. It was described to me as a kind of family of coding and was all around how the junior cycle data was used. It was a particularly complex piece of code. Polymetrika is a company that we have used to do the coding for us. It worked extensively and intensively over a very short period of time to do that because other iterations of the standardisation models had looked at other ways to do this. It had looked at using a number of junior cycle subjects, using English, Irish, maths and the one best result, and eventually it was brought to English, Irish, maths and the best two excluding civic, social and political education or CSPE. It proved a very complex piece of code to the point where, at one stage, we thought it may not be possible to fully enact that and give it effect but ultimately it was possible. To be fair, Polymetrika spotted the first error. We would not, I do not think, have spotted the other errors only for them coming to light and immediately alerting us when it was discovered that there was an anomaly.

On the money issue, €71,000 was the initial cost. It was always envisaged that once we moved into implementation that there were going to be further costs involved. To my knowledge, the current amount invoiced to date is about €193,000 in total. I shall now pass over to Dr. Hislop, if that is okay.

Dr. Harold Hislop

The three errors that were found were all connected with how the data was taken from the datasets dealing with the junior certificate results and then using it within the mathematical modelling. As Mr. Tattan has mentioned on the degree of changes that we were going through, we were trying to work out the best way to use the junior cycle data under what was an intensely short time period. We had discussions at the national standardisation group about the relative benefits of using all of the subjects, of using Irish, English, maths and one subject or Irish, English, maths and two subjects, and eventually came down in favour of the Irish, English, maths and two subjects. Those discussions were really not possible until we had run the model a number of times and seen the effects. It would have been preferable if we had been able to run complete checks on the code but that proved very difficult, in fact impossible.

We were conscious that we had Polymetrika working on the modelling and we attempted to get standby or contingency cover for Polymetrika, including a contingency that would have carried out independent checking of the code for us. At the time it was impossible to get staff after we made approaches. We made approaches in the University of Oxford and we also made other approaches. Each of the agencies in the UK and Ireland, at the time, that would have provided that expertise to us, such as Ofqual, the Scottish Qualifications Authority or the Welsh equivalent was taken up with its own calculated grade process and was also hoovering up the international expertise to do so.

I ask Dr. Hislop to conclude. Next is Senator Pauline O'Reilly and she will be followed by Deputy Aodhán Ó Ríordáin.

I thank Mr. Tattan, Dr. Hislop and Ms Feeney for joining us today.

I believe that the wrong decision was made to cancel the leaving certificate. It has been proven now to have been detrimental, I would say, to an awful lot of students and young people in terms of their mental health.

Steps could have been taken but we are where we are now. That was not a decision for the witnesses and I appreciate that but I just wanted to make that point.

In terms of next year and the issue of standardisation, all the commentary and rhetoric at the moment is that students attending DEIS schools have done better under the current system. What steps are being taken with regard to standardisation for next year to ensure we do not see a backsliding in the advancement for those students?

Students have told me about the negative impact on their self esteem of being downgraded vis-à-vis the grades their teachers gave them or of not being graded very highly by their teachers even though they were doing the hard slog. I was one of those students who did the hard slog but was not noticed by teachers as being particularly bright. I have spoken to the quiet students who are very bright and who knew they were putting in the hard work. They have told me that the rhetoric is that most people have seen an increase in their grades and that makes them feel ashamed that they have done worse than expected. My understanding from previous briefings is that one of the issues taken into consideration was a ranking of grades within classrooms and that is where the problem seems to lie for these students. In a class of five students, for example, who are all very well performing, the fear is that the ranking of grades results in a downgrading. Is that correct? I know that outliers were taken into consideration and have heard that from ministerial advisers but was the situation of classes with numerous high-performing students taken into consideration too?

Many students have contacted me not only about missing out on their schooling in the run up to the original leaving certificate date but also about missing out on the help that students would have been given in other years in the run up to the exams. Is that correct? Were any steps taken to help the students who will be sitting their exams next Monday?

Mr. Dalton Tattan

I do not want to re-argue the decision to cancel the leaving certificate exams but in terms of mental health and student well-being, the decision that was made in May by the Government was informed by NEPS advice and concerns. The public health advice we were getting led us to believe that we would have had to run very truncated and limited exams or exams that did not bear much resemblance to a normal leaving certificate. Undoubtedly, the leaving certificate experience is extremely stressful for students every year-----

Leaving that aside-----

Mr. Dalton Tattan

-----and the lack of certainty would not have helped in that regard.

Regarding DEIS schools, the point the Senator made is important. While we know that we have made great progress through DEIS, we want to make further progress in genuinely closing that achievement gap so that all students have equality of opportunity. Of course, we will be very mindful of the standardisation approach we take in 2021, not least because we will have students who have just done the calculated grades or who are sitting the exams next week who may seek to compete in next year's CAO process.

We must ensure fairness there too. That will be part of the planning done for State exams. Ms Feeney will speak about the ranking process.

Ms Andrea Feeney

There was a question about the ranking order model. The school estimates work was the most important information we had. It was not only the school estimate of the students' likely level of performance, had they sat the examination, but also their places in the classes, as indicated by the teacher by reference to the rank order. A commitment we gave to schools in advance of the process starting was that we would preserve that rank order.

We noted that schools were unable to standardise their results against each other. A person in one school would not know how the teachers in the school down the road might approach the marking process. We needed to have some mechanism for being able to standardise and, effectively, the standardisation was done by reference to the rank order. By providing a rank order, teachers were telling us how the student performed relative to the peers in class. It was an important piece of information.

The system allowed for outliers, such as if there was a high-performing student in a potentially lower-performing group of students. The system was capable of recognising that. If a strongly performing group of students were in a particular class, we had the junior cycle data within the model. While some changes were made later in the process in removing the school historical data, there was information left in the model that allowed us to scientifically and statistically approach the standardisation process.

I welcome the witnesses. It is no secret the Labour Party did not support the cancellation of the exams when the announcement was made on 8 May. We felt the written exams should have gone ahead.

I will go through the timeline. On 8 May, the announcement was made and there was a school profiling element for the calculated grades. The delay to leaving certificate results was announced on 17 July and the school profiling element was dropped on 1 September. Was there not a suggestion in that process that the school profiling element should have been dropped from the entire process much earlier to allow the system to adapt?

Was it as a result of what happened in the UK that there was a feeling that the Irish system should change? We saw what happened in Wales, Scotland, Northern Ireland and England. The results came out and the discrepancy seemed to see disadvantaged students being doubly disadvantaged. Was that a big influence on the decision?

I will ask all my questions at once. Today, the Welsh education minister announced there will be no exams in Wales in 2021 and I am interested to know that if this is repeated across England, Northern Ireland and Scotland whether the Department of Education will also be minded to follow suit because of that rationale.

Junior certificate results were used as part of the mechanism for determining leaving certificate results. How does it make sense to use those results if 2017 - the junior cycle cohort in question - was the first year of the new English cycle? It is also the case that 13% of students do not sit an Irish exam. These are two of the three core subjects within the mechanism as chosen by the Department and they already have question marks over them.

I have a few questions on the process for choosing Polymetrika, and the witnesses have gone through that. It was indicated that the preference of the Department and stakeholders was to hold the June examinations in 2020 for a number of reasons. There was mention of the National Public Health Emergency Team and the political element, and these contributed to the final decision. The decision on school profiling was announced on 1 September but was the decision made much earlier? Was it felt that 1 September was the first time the Department could go public on that?

Mr. Dalton Tattan

I will pass some of the Deputy's questions over to Dr. Hislop but I will address the issue of the timeline for the decision. I will preface what I am saying by advising that account be taken of the fact that litigation is ongoing, some of which is focused on the decision to which the Deputy is referring. I will do my best to answer while having regard to that difficulty. Over the summer, there were calls in respect of the school profiling element. There was quite a degree of commentary in the media with regard to whether it would have an unfair impact on schools, and particularly on disadvantaged schools in disadvantaged areas serving disadvantaged communities. We were very cognisant of that with regard to the standardisation model. We did not categorise the results coming out on 7 September as a delay. There was a very tight timeframe as it was. We had always said publicly that we would issue the results as close to the normal date as possible but we had not committed to that date until the announcement to which the Deputy referred. That was as much time as we needed to get everything done while also making the Universities and Colleges Admissions Service, UCAS, deadline and the deadlines for entry into higher education in other jurisdictions.

The decision to remove the historical school-by-school data was effectively made in the middle of August. The Minister made that decision.

That was in mid-August.

Mr. Dalton Tattan

Yes. Her decision, however, also required a Government decision because the Government decision of 8 May incorporated the historical school-by-school data as a key element of the standardisation model. We knew that the decision would have to be returned to the Cabinet for approval. That is why the decision could only have been made with the approval of the Government. At a policy level within the Department, the decision was made in mid-August. Of course, the public commentary was a very significant factor in that decision. We could see what was happening in Scotland and what had happened in England, Wales and the North. Other factors were also involved, to which others had referred and of which the Minister had cognisance. For example, there was the question of whether, regardless of the impact, it was right that the results of other students, who had been through the system prior to 2020, would have an impact on the 2020 cohort. One thing I can say about the leaving certificate is that people go in with a blank piece of paper and it is their effort and their results that get counted at the end of the day. That was certainly a factor as well.

Dr. Harold Hislop

I will comment on the discussions that were happening. It was very clear from our earliest discussions with the current Minister that she had questions about the inclusion of the historical data. It was the subject of ongoing discussion and something at which we had to look as we developed the model.

The Deputy asked about the announcement regarding examinations the Welsh authorities have made today. The English authorities had already announced that the A level examinations will go ahead in England. One of the examinations in Scotland, the national 5 examinations, which are the equivalent of the GCSEs, will move to a school-based assessment but the higher qualifications examinations, highers, will be held as a traditional examination. We are determined to run the leaving certificate examination. This was also the determination and preference of all stakeholders last March and April. The advisory group that was referred to was determined that examinations be held if at all possible. The group did, however, reach unanimous agreement that this was not safe or feasible for 2020, which is why we had to move ahead with calculated grades.

There was also a question about the junior certificate results and the combination involved. It was not simply a matter of scores from the revised junior certificate subjects. Over the course of the implementation of the revised junior certificate, certain cohorts of students have been taking the revised subjects as they came on stream while others have still been taking the subjects in the old manner. There was, therefore, still a full range of subjects from which to draw data.

I thank the Department representatives very much for coming in and presenting to us today.

We have heard concerns about the calculated grades system from other members. I believe there were a few faults in the calculated grades system. It was done in a very short period of time and everybody was under pressure. As young people in working-class areas were able to reach their full potential, it is not all bad news. For example, a few young women in Ballyfermot got 600 points. Their teachers graded them at the level they know they were capable of. These are bright young women who will go on to become leaders in their own community and will go on to better our society.

We can look at some of the issues or flaws with the calculated grades. The experience with the leaving cert assessment shows that not all young people start on an equal footing within the education system. If anything, it has shown there was always a fault in how the leaving cert exam was done. Most Traveller children from my community do not complete second level education. Moreover, many studies have shown that achieving higher grades and good degrees depends on money and investment in a child over time.

In the middle of a pandemic we need to take into account factors such as halting sites, people's accommodation, homelessness, people living in direct provision, access to laptops and people in rural areas not having access to broadband never mind laptops. We are talking about young people's mental health and well-being. There is one solution for this. People might think I am a bit crazy for saying it, but I will say it anyway. We have an opportunity to put the leaving cert on an equal footing for all young people in Ireland.

Last night I was speaking to the vice principal of a school who believes it should be assessment based. If I were measured on my academic ability, I would not be here today. One of the teachers said to me last night on the phone: "If I was to look at you, Eileen, from your academic ability, I would never have pictured you being in the Houses of the Oireachtas. I learned from you as a student and how we must go forward believing in students." We must look to doing the leaving cert differently. We are talking about pressures. There is one way to settle all these pressures. Listening to Deputies and Senators, I believe it would be better to have continuous assessment.

What are we doing to address inequalities when we are assessing students' performance? That is down to accommodation and resources for young people. There is a bigger picture beyond what we are discussing today. I thank the witnesses. Based on the experiences of young people in working class areas, I am really happy with the calculated grades system.

Mr. Dalton Tattan

I will reply first and then pass over to Dr. Hislop. It is very heartening to hear the stories the Senator mentioned with the opportunities that people have. It underlines the point I was making earlier. The approach with standardisation was very much to try to ensure we did not exacerbate any disadvantage this year. We were very pleased that we were able to achieve that through the calculated grades. We cannot mistake that for genuine equity in society and ensuring that people have those opportunities in more normal times.

Part of that informs the approach we have of being very determined to run exams and, as far as we possibly can, run all components of exams.

We know that students can demonstrate their abilities, skills and talents in lots of different ways. It was not possible in November to do anything other than simply to run the written examinations, but for next year we want to run as far as possible all components of the process to ensure that those sort of inequities do not start to reassert themselves in the system. I will hand over to Dr. Hislop to speak about continuous assessment.

Dr. Harold Hislop

I think it is worth noting too that while leaving certificate exams are starting next week, we are also running junior certificate exams for students who are early school-leavers and for whom the junior cycle examination would possibly be their only qualification. They have been given that option, and a small number have taken it, but it is still important, as have a number of adult learners who are eligible to take either the leaving certificate or the junior cycle next year, again for whom it is a return to education and perhaps their only qualification.

One of the weaknesses in the leaving certificate system at the moment vis-à-vis comparable systems in other countries is that we have fewer episodes of assessment throughout the two years of the leaving certificate cycle. We do have practicals and orals, which are useful and give us a lot of information, but some countries have different assessment points at different times. We did not have that option, but we are determined to run the oral exams and the practical subjects for next year because they do form another important part of measuring the learning of students.

Ms Andrea Feeney

The senior cycle reform process is ongoing and the work that was continuing apace was affected by the Covid interruption. The senior cycle is being reformed with the partners and stakeholders to look at issues such as continuous assessment and how we can have a more equitable approach to how we address assessment that looks at intervals of assessment rather than everything depending on a final, written examination. It is not that the final, written examination does not have a place, but we are looking at supplementing it with a more spaced-out approach to assessment.

I apologise for being late. Thank you, Chair, for accommodating me. I welcome the three witnesses. I come from an education background myself. We have all heard good news stories in the committee in that more people than ever before have received higher offers in terms of the allocation of third level places. We have also come across tough stories from people who in many cases seem to have a legitimate grievance in terms of being disadvantaged by the system that was applied. As other speakers have acknowledged, as the crisis was unfolding, decisions had to be made very quickly. I commend the Department and the Ministers on the actions they have taken heretofore. That said, I believe it is important to have reviews to learn from mistakes. I apologise if my question was asked earlier. In terms of errors that were made, I assume mistakes like the coding error would not happen again. Looking ahead to 2021, I understand a review is currently under way. Could we get a brief rundown of the status of that and when we are likely to see a report or any recommendations? It is not just for the purposes of finger pointing, but to get things right if we do have to engage in the process again in 2021.

Mr. Dalton Tattan

The Minister has already said it is her intention to proceed with a comprehensive review as soon as is practicable. As Deputy O'Sullivan says, it is to learn the lessons rather than to play a blame game, both in relation to calculated grades but there are other things too such as assessment more generally and the experience of teachers in that regard. To do that we need to complete out the process. We have two key remaining steps, one is the exams themselves, which as I said are on the brink of starting next week, and then there is also the final stage of appeals. There has already been a stage 1 and stage 2 appeals process, but there is a third independent stage operated by independent scrutineers and that is under way at the moment. We expect it will complete out over the coming weeks.

There is one question that we have to figure out.

Given that there is litigation, to which I referred earlier, does that in some way hamper our ability to proceed with the review? We are very keen to get the review under way and for it to be meaningful we want it to move as quickly as possible. We will need to check that aspect, however, to ensure we can proceed fully on all the parts that we want to do.

My second question concerns the company implicated in the error in the coding. Has Mr. Tattan any information about that process? I am sure this will come out in the review, and I am not suggesting that there will be sanctions, but what are the likely repercussions? Did the company meet the obligations of its contract at the time? Is Mr. Tattan in a position to comment on the role of Polymetrika?

Mr. Dalton Tattan

I will make some general comments. Again, advice will need to be sought on that issue and on the terms of the contract on which we engaged the company. As I touched on earlier, the timetable which we were set was very ambitious. Ms Feeney and her colleagues in the calculated grades office and others, including Polymetrika, were working flat out through the summer to try to achieve what was needed.

As I mentioned, the coding in question was very complicated and it was written in such a way as would have allowed for some independent external scrutiny. We had tried, as Dr. Hislop said, sourcing others who could have been there as contingency or as a way to check that aspect of the process. It simply did not prove possible in the time we had, and with the demand and the expertise required. It is something that we will, ultimately, have to consider.

Initially, teachers were given the impression that their ranking order was not going to be made available, but subsequently it was. I would like an insight into when that changed or when it became clear it was going to become that transparent. Teaching unions were initially of the impression that the ranking order was going to remain private. Will Mr. Tattan give us an explanation regarding that aspect?

Mr. Dalton Tattan

We had a series of engagements with the two post-primary trade unions on this matter. They had an understanding that the ranking would only be available through a subject access request under data protection legislation. That was not, however, the understanding we had. We had a misunderstanding between the two parties involved. We had a round of engagements with the unions on that aspect. We sought legal advice on it at the time, which ultimately said that we were bound to release it because it was personal information. We did release it, albeit a couple of weeks later than we had originally intended.

To follow on from and add to what was said by Senator Flynn, who has left just now, it was interesting to note what happened with many of the practical subjects. Ideally, that might be the way forward out of this and lessons may be learned. We need to engage in more continuous assessment in practical subjects etc.. I taught a subject with a practical component when I was a teacher. This is more a point than a question. If we do have to make changes to the original system regarding how we deliver exams, and the oral and practical components, I suggest we give schools some flexibility in deadlines and dates for submission of those practical and oral components. We could also let them start as soon as they would like to and make it more practical in a school-based setting. I ask if that could also be considered.

Mr. Dalton Tattan

The Deputy has made an important point. The planning is under way for 2021 and we are cognisant that whatever we do regarding arrangements for the exams they must be sustainable at a school level. Schools are under pressure every day because of arrangements with Covid-19 and meeting the existing health and safety requirements.

I apologise for having stepped out. The witnesses will appreciate there was a vote in the Dáil. The representatives will forgive me if I repeat questions that have already been asked. I ask the witnesses to pretend they have not already answered the questions.

My focus is not on leaving certificate 2020. It is done. There were regrettable errors. We know that, and we also know that some of them were not caused by the Department. The rest of them were caused by Covid. I am not going to dwell on that. In the context of leaving certificate 2021, and taking on board the step taken in Wales today to suspend any plans for a seated examination in the summer of 2021, one must inquire about the process.

One has to ask and to forward plan. If it is not possible for us to have an examination on the normal date in early or mid-June, what contingency plans are in place? Further, if the 2021 leaving certificate cannot take place in an examination hall, are the witnesses confident that the predicted grades system is robust enough, given what we have learned, to be used again?

My final question relates to long-term effects of Covid on our student population and the need for a one-day examination of a student's ability at a senior secondary level. Should we recognise this opportunity and take it as the wake-up call we may have needed? I mean no disrespect to the representatives from the Department and do not doubt the professionalism they bring to their roles but we need to have that conversation. We all know it. Mr. Hislop is nodding. I would appreciate if the witnesses could spend some time on that third point, which is probably more important than the prior ones. I leave that question open to any of the witnesses who wish to answer it.

Mr. Dalton Tattan

I will deal with the first two questions and then hand over to Dr. Hislop. Our determination is to run the exams next year. We cannot be absolutely sure and if this year has taught us anything it is that there is uncertainty in the world but we are absolutely determined to run the exams and all their components as normally as possible. That said, given the experience we have just gone through, we would be mad if we did not make contingency plans and think about alternatives.

We intend to carry out a review of the calculated grades process this year. We have obviously learned a huge amount but we had to learn it very much on the hoof as we were doing it.

I am sorry to interrupt. Will that report be presented to this committee or to the Houses of the Oireachtas?

Mr. Dalton Tattan

We intend that the report will ultimately be published. We have to wait until we get to the end of the process, which still involves the exams and the final stage of the appeals. It is ultimately something that should be in the public domain and should be shared. However, we need to understand the process better. We have some sense of it, having gone through the experience, but we are still just coming out of it at this point. We need to understand what it was like, what worked and what could be improved. We obviously had issues like the errors, which nobody wanted. We would have liked not to have had to face into those but we had to correct them at very short notice. We need to learn those lessons but I reiterate that the intention is to run the exams unless it proves absolutely impossible to do so. We do not anticipate that that will be the case. Schools are open. They have gone through their first half-term successfully, they continue to be open and that is the intention. Next week is an important milestone for us as well. Small numbers of students will take the November exams but that will prove that it can be done while schools are also running.

Dr. Harold Hislop

It is important to emphasise that schools have made huge efforts and have learned a lot about how to operate in a Covid environment. We simply did not have that information in June and it was not safe to run the exams in the current year. Exams are an important way of assessing students. There is no doubt about that. They serve a useful purpose and they have their strengths as well as their weaknesses. As students have been preparing for doing an exam for the previous two years, it is the fairest way of assessing them. That includes the other components of the exams as well.

The Deputy asked about the long-term effects of this situation. There are obviously gaps. We have recognised already that there are gaps in students' learning and that is why the exam arrangements for next year have been altered so that students will have a greater range of choice in those examinations. They will have to cover fewer areas and yet will still be able to do the exams. We thought that was the fairest thing to do, by adjusting the assessment so that, irrespective of the order in which students had covered elements of their syllabi, they would still be able to do their exams and represent their learning fairly. The other side of it is that this whole experience has shown that if we had other episodes or modules of assessment that were properly set, moderated and examined over the course of the two-year period, we would not have been relying on a single three-hour exam that we had to cancel.

Other countries have those measures in place and they were able to cope better. That was not possible in Ireland, which is an important lesson in regard to the review of senior cycle in years to come.

The committee and the witnesses, as individuals within the Department, must recognise that the fifth years who are now sixth years are at a distinct disadvantage to their peers. I welcome what Dr. Hislop said, but I would urge caution and suggest members might pause their consideration of this matter and how it might affect the current sixth years to a later stage, perhaps with the student representative body which, I know, has been in touch with the committee in writing. It would be very helpful if we engaged in a meaningful discussion with it on this matter at a later point. I thank Dr. Hislop for his responses.

I have a number of questions and I would appreciate it if the witnesses could answer them as succinctly as possible. In terms of the number of legal cases that are in the system, how many are outstanding and when is it expected they will be concluded?

Mr. Dalton Tattan

It is a difficult question, Deputy, but I will do my best. There are approximately 20 cases. There are a number of cases that have been granted leave for judicial review by the High Court and a number of others that have not been granted leave. The latter are waiting in the wings. They are not formally initiated at this stage. There is a lead case, which the High Court judge has identified and that is proceeding at the moment so it is in a discovery process. We do not have a hearing date. We had one, but it was vacated by the High Court judge a number of weeks ago. I cannot anticipate when it is likely to happen but it is unlikely to happen before the end of this year, I would think.

There are 7,000 to 8,000 students who were wrongly upgraded, a number of students who were downgraded and, in particular, a number of students who feel disadvantaged and that their education was sabotaged because there was such a differentiation between the teacher's assessment and the school's assessment and the grade they ended up getting. If any of the court cases impact that, what resolution will they have?

Mr. Dalton Tattan

I do not think I can get into a discussion on that because it is speculative. There are cases with a number of different grounds to them. There are variations. They all broadly argue against the standardisation model in one way or another, but they are different in how they are approaching it. Some are taking issue with one aspect of it and some with other issues. There are some cases involving 2019 leaving certificate cases as well. I do not think I can get into a discussion on it.

Dr. Harold Hislop

It is worth noting that the estimated grades that came from teachers were only adjusted where there was solid evidence on which to base that adjustment. They were not adjusted if there was no evidence that indicated that such an adjustment was not warranted.

The evidence that teachers and schools had in terms of testing that had been done previously to substantiate the grades they had given was not considered. The students in respect of whom I have seen testing over two years and the results they had achieved feel that they have not been taken into account. I expect that those issues will be tested within the court cases. In regard to the students who are severely disenfranchised, did the Department at any time consider taking into account the assessment of the schools and the teachers and using those grades for students in order to draw a line under this?

Dr. Harold Hislop

Yes, that was considered.

Why was it not done?

Dr. Harold Hislop

It was considered. The possibility of using teacher estimated marks unadjusted was considered but it was very clear when that data came in from schools that different schools and different teachers within individual schools had over-estimated the likely performance of students at very different rates.

We would have ended up with some cases of schools and teachers, and individual teachers within schools, awarding grades with little or no overestimation and others awarding grades in the same subjects where severe overestimation had happened. We would have been leaving that inequity in the system unadjusted.

What was done to address severe underestimation?

Dr. Harold Hislop

Those students were moved up.

Is there any hope for the students still left with severe underestimation?

Mr. Dalton Tattan

A group of about 3% or 4% was moved up. It was 4.4%.

Was that through the appeals process?

Mr. Dalton Tattan

No, it was through the standardisation process. It was part of the results when they were issued on 7 September. About 17% of grades were moved down.

Will Mr. Tattan acknowledge that many students are still in a limbo situation having been downgraded and with their marks bearing no relation to the assessment of their teachers?

Mr. Dalton Tattan

No, I think-----

Is Mr. Tattan saying he does not acknowledge that many students have been severely disadvantaged because they have been downgraded?

Mr. Dalton Tattan

That is a characterisation of it. From what we saw and how it was talked about, there was a significant degree of overestimation. That is not surprising because it happened in other jurisdictions.

It is the underestimation I really want to get at. I am happy regarding the overestimation aspect, but I want to see if Mr. Tattan can give any hope to those students affected by underestimation. Will they have to wait for the end of the legal process?

Mr. Dalton Tattan

If there are students who were underestimated, where the school-estimated marks were lower than what they ought to have received, that was corrected through the standardisation process. About 4% of students did move up, and that was a small proportion of those who moved down. There are students, however, who feel that the standardisation process did not give them a fair representation. Their argument, if we take some of those legal cases, is that the teachers' marks or the school-estimated marks were at a higher level than the results the standardisation gave them. Those students feel aggrieved and some have chosen to litigate the issue, and that is their entitlement. On the evidence we have seen, however, what came through was that there was a high degree of overestimation. That was particularly the case in some schools, where the results bore no relationship to the typical results in those schools in previous years.

We are not going to agree on this issue. I refer to the correspondence we get from so many students and parents concerning the impact on mental health. They give us all the documentation they have from their tests and everything else from within the schools, and their teachers' marks bear no relation to their final results. If Mr. Tattan cannot acknowledge that, we cannot make progress on this issue until we have the outcomes of the legal cases.

Mr. Dalton Tattan

If we were-----

I ask Mr. Tattan to respond later, because I want to move on. I call Deputy Nolan.

I thank the witnesses for their presentations. The calculated grades system was far from ideal, but we were in far from ideal circumstances. I acknowledge that it did not disproportionately affect disadvantaged children, which I was concerned about. I state that because we were hearing reports from England regarding children from disadvantaged backgrounds being adversely impacted by the process and system used in that country. It is good to see that did not happen here.

I have some questions, however, and one concerns the number of appeals. It was stated that appeals are outstanding and there may be a delay in the review coming to light as a result. How many appeals are outstanding and what efforts are being made to resolve them as soon as possible? Several students in my constituency also studied subjects with a tutor outside of school. The decision regarding those students happened late in the day, as did proposals to deal with the situation.

The uncertainty caused most of the anxiety for the students. If they had known they would be going through the calculated grades process sooner, maybe they would have adjusted sooner and come to an acceptance.

On those students who studied outside of the schools, are there many of those affected in terms of appeals and downgrading? What methods were used to ensure those students were treated fairly in the calculated grades system?

Mr. Dalton Tattan

On the students studying an additional subject or two outside of school, we made every effort that if we could at all, we would give them a calculated grade and there were a number of routes through which that could be done. Unfortunately, it was not possible in every situation. Usually the stumbling block was that the student in question may not have had a registered teacher involved in overseeing his or her education and therefore, there was no means of verifying the work he or she had done. We had some litigation around that too and we resolved some cases as a result of the litigation. I will pass to Ms Feeney to deal with the numbers and the appeals.

Ms Andrea Feeney

On the appeals and how many appeals are outstanding, we are stage 3 of a three-stage appeal process. Stages 1 and 2 have already been concluded. We had 12,000 students who made an application for a review of about 33,000 results. It was a technical process-based appeal looking at issues with the transmission of the data through the process and checking all of the points of data transfer. The professional judgment of the teachers was outside of the scope of the appeal process so if a student felt that a teacher had underestimated his or her performance, that could not be accessed through the appeal process. That was part of the compact with teachers in agreeing to engage in the calculated grades process to begin with. The statistical standardisation process was also outside of the scope of the appeals process. As a result of the processing of the stage 1 and the stage 2 appeals process, there were 18 upgrades. From those 33,000 individual subjects, there were 18 upgrades. There is a stage 3 appeals process which is for a referral of the procedures that were undertaken by the calculated grades executive office in the processing of the appeals to independent appeal scrutineers. Approximately 400 of the 12,000 students have undertaken that option and have made an application for further review against 900 of their grades. That process is under way but we cannot say how long it will take. Each stage of this has been new so it is a little bit uncertain how long the process will take. As soon as we know, we will alert students to that.

I refer to students who were studying either one or all of their subjects outside of school, either with a tutor or completely on their own. My colleague, Mr. Tattan, has already referred to the fact that every effort was made to accommodate those students within the process. As a result of our processing, for the students who were total out-of-school learners, we had 173 cases where we could not provide them with calculated grades in any of the subjects in which they were entered. We had 21 cases where they were part approved or part refused. We had 51 students who did not want to get calculated grades and opted out of the process and there were about 135 students who we contacted on a repeated basis over the summer about their applications and where we had no engagement in return. It is interesting that of the 2,800 students who will sit examinations starting next Monday, about 200 of those are in that category where they were studying all or some of their subjects outside of school. They have returned for the November exams and none of them has appeals ongoing. We have gone through as many procedures as we can with them and we have looked for every opportunity available to engage with them to make sure they were aware of the process but ultimately, for reasons of inter-candidate equity and to make sure we were treating other students fairly, we had to have a basis on which we could ground a calculated grade. There had to be some evidence available to us in order for us to look at running that information through the estimation process. I hope that answers the Deputy's question.

I would like to start by acknowledging the huge job of work the inspectorate and the Department had to get through in order to produce the calculated grades system.

It was by no means perfect but as the last speaker said, we were not in a perfect situation. On the whole, a very good job was done. The inspectorate and the Department deserve praise and I wish to begin by doing that.

I have two questions on points of information and another on Dr. Hislop's remarks about the increased variety in the questions for the forthcoming leaving certificate. Some 2,800 candidates are looking to sit 7,300 examinations. That is slightly less than three examinations per candidate, but I assume the median is much different from the average. Is a large cohort repeating the leaving certificate in full while many others are taking one or two examinations or what is happening? The second question relates to communications with schools and supports for schools in preparing for sitting examinations. Schools are experts in putting on these types of events but are we happy that the schools have been sufficiently prepared and supported to allow the examinations to go on?

On the increased range of question choice, I am concerned specifically about a subject such as maths. How is the secondary school system communicating with the third level sector? If a greater number of questions is made available in examinations, there will be uncovered gaps in the curriculum. It sounds very much like it will be decided on a school-by-school basis. Taking maths as an example, will we end up with a cohort entering third level courses that have a mathematical component, such as economics, physics or straight maths degrees, with non-standardised gaps in their knowledge? If so, much of the first year of teaching at third level may involve finding those gaps and filling them in. Is there ongoing communication with the third level sector to identify where those gaps will be in order that they can be addressed in first year to allow this cohort of students to hit the ground running when they begin their third level courses?

Ms Andrea Feeney

I will have to get information for the Deputy on the median number of examinations being taken per candidate. Having looked at the data recently, I am aware that most students are taking examinations in two or three subjects. Those who could not get calculated grades in the summer are more likely to be doing their full suite of examination subjects.

On resourcing and planning in schools, the State Examinations Commission has been engaging with schools since the summer to plan for these November examinations. We have put resources in place for pre-planning, an examinations aid for pre-planning and a logistics person to be there and engage in the school over the course of each day the examinations are on. The examinations will run from Monday, 16 November to Friday, 11 December, inclusive, and will run on evenings and weekends. It is a question of ensuring supports are available in schools for each of the times and dates on which examinations will be sat. Examinations will not be sat in all schools. Some 600 venues out of 740 will run examinations. In most, the number of candidates taking examinations will be very small. The protracted nature of the timetable means that there will not be a huge load, but it will be more likely to be at the earlier part of the timetable. The school can appoint a local superintendent and the State Examinations Commission can assign a superintendent to each venue to be the commission's eyes and ears, to be the custodian of the examination papers, to mind candidate entries, to look after the scripts and to return them to the State Examinations Commission. We have also made provision for additional costs arising in the school in respect of heat and light, PPE, cleaning equipment and additional sanitation which is required over the course of the examination. I believe the supports have been made available and we are engaging with schools on that.

That is a great, comprehensive answer.

Dr. Harold Hislop

I will explain the approach we took towards the content of the examinations. We were conscious that individual teachers and schools will teach the elements of each syllabus in a different order.

They are quite free to do units of work in whatever order they wish. It would have been impossible for us to say X or Y or Z units are to be dropped and they do not have to do those units of work. Instead, the approach was to adjust the assessment and to acknowledge everybody had missed months of study from March to May or June of this year, so we will give greater choice in the examinations for June of 2021. That approach will allow the student to say that he or she does not have to have covered the whole programme, but will still be able to answer and participate in the examination fully.

The downside of that is the point the Deputy makes. Different students in different schools will have had different experiences. In fact, there has been quite an amount of research on the impact Covid has had on the learning of students generally, and this has come up quite clearly. The ESRI has published reports on it, and we have produced reports about it. Part of the reason for that is to highlight to the system for third level, teacher education and so forth that there are going to be gaps in students' learning.

There are other impacts on students as well. We have had to make a strong investment in the well-being of students. Many online resources have been put in place to help students with their well-being and to help teachers to support student well-being. It is not simply academic or cognitive knowledge and skills. We hope, however, that by allowing the wide range of assessment through practicals and orals, as much as possible akin to normal, that we will allow students to demonstrate their skills across all their ability ranges and their competences within each subject as well. Certainly, for universities that receive students, there will be less learning. It is inevitable. However, it is not possible to determine that on an individual case basis.

I apologise for the time, Chairman. I was trying to give a comprehensive reply.

I am conscious that I will not be able to go to the Chief Whip and tell him that Dr. Hislop held us up in the committee. It can be a written note. I call Deputy Jim O'Callaghan.

I thank the witnesses for attending the meeting today. Does Mr. Tattan believe that any students were treated unfairly by the calculated grades process and, if so, what cohort of students?

Mr. Dalton Tattan

I do not believe so. We went out of our way to try to ensure fairness as far as possible, accepting the limitation that, ultimately, students did not sit examinations during the summer of 2020. What we were asking teachers, principals and schools to do, therefore, was to give their best estimate of what those students were likely to attain. Then we drew on data we had to try to ensure fairness throughout the cohort. Dr. Hislop talked earlier about how we tried to do that to ensure there was as much equity as we could achieve throughout the 2020 cohort and some level of comparability with other years too.

Does Mr. Tattan not think students who received certain marks from their teachers and who were subsequently significantly downgraded were treated unfairly?

Mr. Dalton Tattan

I do not believe so. We saw through the data that there was some level of underestimation but there was a far greater degree of overestimation. However, the overestimation was not even. In other words, with some teachers within schools and across schools there was a greater level of overestimation in some schools compared to others. To leave that without applying a standardisation certainly would have been unfair.

Does Mr. Tattan think that, perhaps, the schools were treated fairly, but individuals within each school may have been treated unfairly?

Mr. Dalton Tattan

No, I believe they were treated fairly. There was the rank order that was used, the extensive guidance given to schools at the time, as well as an in-school alignment process to try to ensure there was as much fairness coming through at school level too.

Mr. Tattan may have answered this question already, but is an independent review being conducted on it and, if so, when will it be complete?

Mr. Dalton Tattan

There is to be an independent review. We are scoping that out at present. We have two remaining pieces to be done with the calculated grades process and the 2020 leaving certificate. One is the examinations to take place next week and then there is a remaining independent stage of the appeals. Once those are done, and subject to any legal advice we get that might preclude us because of the litigation, we intend to proceed.

Who is conducting the independent review?

Mr. Dalton Tattan

We are scoping that at the moment. We are looking at possibilities. One route would be to tender out to a commercial company to do that, but that might take more time and we are not sure we will get the expertise. A different approach might be to look for an international panel of experts.

When does the Department envisage the process of review will be completed, or can Mr. Tattan say?

Mr. Dalton Tattan

Both the appeals and the exams will, we hope, complete before the end of this year. The exams are due to finish around 11 December and the appeals are being worked through at the moment. Unless we are told we cannot proceed for legal reasons, we intend to be ready to go then. We are doing the planning work for that now.

That is why there is the independent review.

Mr. Dalton Tattan

Correct.

I do not want to speak about the circumstances of cases that have commenced before the High Court but if some of those students succeed what will be the impact of that decision on students who have not instituted proceedings?

Mr. Dalton Tattan

One of the Deputy's colleagues, Deputy Conway-Walsh, asked a similar question-----

I beg your pardon.

Mr. Dalton Tattan

-----and I would prefer not to go there. I do not believe it is appropriate given that the matters are before the courts and there is a range of different types of cases from different schools, and involving 2020 and 2019 cohorts of students.

Obviously Mr. Tattan will agree that the circumstances of the leaving certificate 2020 class have been unfair in that they missed out on three months of scheduled schooling because the schools were closed due to Covid-19. Some of those students are now going to proceed to do the leaving certificate in November. Has anything been done to make the pathway to that leaving certificate easier for those students in the same way that I understand the leaving certificate students of 2021 have had the course slightly amended to reflect the loss of schooling they had?

Mr. Dalton Tattan

Maybe Dr. Hislop will come in on that.

Dr. Harold Hislop

There have been no adjustments made to examinations for that cohort of students. On the grounds of equity with the students who got calculated grades, the teachers were asked to estimate grades, marks and rankings on the basis of what the student would have achieved if he or she had done the full syllabus and participated in the full examination as normal. They were given an estimate on that basis. The same basis would, therefore, apply for the students of the November cohort. The November cohort is atypical. It is not the normal group of students, obviously, as some students will be taking the exam because they were not able to take those subjects as a calculated grade or not able to be given a calculated grade, and some of the students preferred to take the examinations, but there have been no adjustments made.

On that topic, the students who will sit the examinations in November will get their results in February. Is that correct?

Dr. Harold Hislop

Yes. That is the expected-----

Do they then have the opportunity or sufficient time to repeat the leaving certificate if they wish to do so in 2021?

Ms Andrea Feeney

Yes.

When does applying for the CAO kick in for them or will they need to apply again?

Ms Andrea Feeney

It would be up to them to make that decision at the time to reapply for the CAO. Once they have an application in for the CAO, I understand they can change their minds on that up to July.

I thank the witnesses.

When one reflects back, the past six months have been very challenging for the Department of Education. If the officials here were to make any changes or do anything differently, and hindsight is a great thing as they say, what are the top two or three things they would do differently today, as they reflect back on what has happened over recent months?

Mr. Dalton Tattan

I suppose we would have started earlier. That is something we were just not able to do when the pandemic came. When the first lockdown hit us, and the school closures happened with no notice, we were all caught in a sort of whirlwind. The leaving certificate was caught up in that too. It took quite some time to figure out if we could really run a set of exams. We really did try to run exams for some of the reasons the members have spoken about, such as the need for fairness and the concerns about what calculated grades might mean. Ultimately, we came to the view that they simply could not be run.

If we had had more time, it would have given us a greater chance to plan and avoid some of the issues and we would have had earlier results as well. Students started in higher education a few weeks later than they would ordinarily have. That said, if we had run the exams they would not have started even at this stage of the year. They would not have started until December or January.

Dr. Harold Hislop

That is one of the key lessons from this. We attempted to make the examinations work because we thought that would be the fairest way to operate for students, but it just did not prove possible. We probably spent longer than we might have needed to trying to run the exams instead of spending that time developing the calculated grades model.

The second lesson is the point I made about having a range of assessments for leaving certificate students across their two years of study. That is a big learning for us. This situation has highlighted the vulnerability of relying on one exam at one point in time. As a long-term piece of learning that is very important. Within the calculated grades process itself, we were very good and successful. I pay tribute to the teachers, school managers and union leaders who worked and collaborated with us as stakeholders. We tried to keep them and the school system informed but we found it challenging, particularly as secondary schools shut and we had to make them work into June. Keeping those lines of communication open and running with them was challenging and we probably learned how to do that better. Within the grades model itself, it is obviously a matter of regret to us that we did not find the error before we issued the results. That was undoubtedly a failing in the system and of course we have learned from that.

Following on from that, what checks and balances were carried out on the company that did the coding and what options does the Department of Education have now? Are we looking at a legal case? I am aware that we are conducting a review and everything like that. I ask for a very brief answer because I have another question about the upcoming leaving certificate exams starting next week.

Mr. Dalton Tattan

As I touched on earlier, we will seek advice on this matter and the contract we have with the company. It has to be looked at in the round and everyone, including that company, was under that very intense pressure and was working around the clock to achieve this. Education Testing Services, ETS, the independent company that came in to check and give us assurances when the coding errors were discovered, recognised the level of complexity in the code. We have to look at all those things in the round in making a call on it.

Regarding the 2,800 candidates who will be doing their leaving certificate exams in 600 centres across the country, I presume there will be a number of students in the same classroom doing the exams. What contingency plans are in place if one of those students is diagnosed with Covid? I was a member of the Government when the decision was made to cancel the leaving certificate and in my nine years at the Cabinet table it was probably one of the most difficult decisions the Cabinet had to make. It had to think of the thousands of students, families and parents and the years of work they had done. Some of my family members were doing the leaving certificate as well and they were on to me daily asking me what was happening and were really concerned. What contingency plans has the Department put in place for the students? If one of them is diagnosed with Covid in the next two or three weeks, what happens to the rest of the students? Do they carry on? As close contacts they are meant to go into isolation for ten days. I ask the witnesses to comment on that matter.

Ms Andrea Feeney

It is a difficult issue. Things happen within the examination system every year whereby students cannot undertake or continue with their examinations for other reasons, not just because of Covid. This year, this sitting of the leaving certificate is the contingency. We had calculated grades and ultimately there is an opportunity for students to sit their examinations. It would be very difficult to make arrangements if a group of students had to exempt themselves because they were ill. Their only option may be to come back and sit those exams in 2021.

Of the 2,800 candidates, are more students from private schools participating in the upcoming leaving certificate or is there a mix across the board?

Ms Andrea Feeney

There is a mix across the board. There are probably more coming through from certain quarters. I do not have the data in front of me but, as the Chairman is aware, the numbers are not huge in the overall scheme of things. I have not looked at it in terms of a breakdown between private schools and other schools.

Returning to the calculated grades, the Institute of Education holds weekly exams for all subjects for the leaving certificate. Could they not have been taken as an example for predictive grades announced in the past number of months? They would have been very accurate because they hold regular exams in all subjects. I am a great believer in continuous assessment and believe we must go down this road. Could Ms Feeney comment on that?

Ms Andrea Feeney

I would not want to characterise in the context of any one school.

That is okay.

Ms Andrea Feeney

One of the underlying principles relating to calculated grades is that schools are good at knowing how schools perform in their local context. They know how their students do in the context of an examination held in the school because they are able to make that call in their local frame of reference. However, schools do not have the knowledge to be good at understanding where the school is performing in respect of the national standard, which is why we had the national standardisation process.

Dr. Harold Hislop

Yes, there has been. Even as the Covid pandemic unfurled last April, May and June, NEPS developed a lot of material and online supports for students that it made available to teachers, schools and students. These materials are accessible through the gov.ie website. One very persuasive reason for postponing the leaving certificate and moving to calculated grades was the extraordinary anxiety among and pressure on students at the time. Deputy McHugh was the Minister who made that decision and this was clearly a very important factor he took on board.

Dr. Hislop spoke about the well-being of students. The leaving certificate happens every year and there is significant pressure on students. Have we learned lessons this year? This year, we have really concentrated on the candidate doing the leaving certificate. Are there lessons for the future regarding the type of well-being supports we can put in place for students?

I preface my five questions by saying that I appreciate that an extraordinary amount of work has been done in unprecedented circumstances. However, I am under siege from emails from people who have been downgraded and my questions pertain to this cohort. Based on replies to Deputy Conway-Walsh, what I hear is that the Department does not accept that there is a group of students with remarkably similar patterns of achievement who have been downgraded. What I am hearing is that this downgrading is due to an overestimation on an individual basis by teachers. This stands in stark contrast to the profiles of the schools and students. I have heard Ms Feeney say that schools know their own but not necessarily where they appear in the national average. However, if the school has a high number of H1s on a consistent year-on-year basis and suddenly it does not, I would venture to say that the national average does not apply in those instances. I am curious to know how the witnesses can make a generalisation. If they are saying it is on an individual basis and that there was overestimation, how they can say this about this cohort? I am at a loss as to how to understand that and need that clarified.

Did the Department carry out a data processing impact assessment with regard to the rights and entitlements of data subjects because there seems to be a large number of people who were denied Form A, Form B and other information when they sought it? Moreover, a number of complaints have been made to the Data Protection Commission. If there is a data processing impact assessment, has it been published or can it be published?

Regarding the exams in November, at some point in the future - perhaps in the context of the review - I would be interested in finding out the profile of the 2,800 students and how many of them are coming from schools with a historical pattern of high achievement because I would venture to say that there are quite a lot of them. These students have been disproportionately downgraded and disproportionately affected and are not in the courses they expected to be in. I am astonished that no adjustment has been made for them despite the fact that they have been out of school since March. They were promised three weeks of some time in school before they sit their exams but they have not been at that and there was no comment as to why that was the case. The timetable is not aligned to the normal June timetable. There are things like Maths 1 and 2 happening on the same day so that seems to be disproportionately affecting them as well while some of them are coping with Covid and everything else. I would like comments on how and why that was set.

Given that there are far fewer students sitting the exam in November, why will it take until the end of February before they will get their results? Those who are relying on the HPAT will only get this year's HPAT carried over into next year if they get a deferred place arising out of the leaving certificate results and the closing date for applications to the HPAT is before the results of the leaving certificate, so the timing prejudices this same group of students.

Will the terms of reference of the independent review be published and is that review merely a procedural one, as was the appeals process? In the absence of a substantive element, I believe it was not fit for purpose. Will the independent review be a substantive one as opposed to merely a procedural one?

Mr. Dalton Tattan

Regarding downgrading, essentially what we see in terms of the data coming through is that schools did overestimate. Some schools very significantly overestimated. If we compare the estimates they had in the calculated grades process with results from the past number of years, we can see that in some cases, they do not bear much resemblance. In some cases, there was a very high degree of overestimation. To have left that there would have created a huge level of unfairness because there were other schools and teachers who did estimate fairly and accurately based on the standardisation model that was applied. We should bear in mind that about 79% of grades were unchanged as a result of the standardisation model so four out of five grades remained unchanged after they had been put through the standardisation model.

Except that there is a very small percentage within that very high-achieving group.

Mr. Dalton Tattan

I will come to that. The Senator asked about a breakdown of the 2,800 and those coming from a high-achieving group. I think we would need the Senator to really lay out to us what she means by that. It is a phrase that is used but we are not sure we fully understand what is meant by that because other people have made that argument to us as well.

We could list the schools.

Mr. Dalton Tattan

There are breakdowns in the national standardisation group report between DEIS and non-DEIS schools and others, which include non-recognised fee-charging schools but other settings as well so there is some degree of breakdown there. We would need some greater articulation, which would certainly help us.

Dr. Harold Hislop

It is also worth noting that in the national standardisation group report, the estimation was greatest at the level of higher level 1 and higher level 2 - the highest performing students. If we take a subject, over the course of 2017, 2018 and 2019, H1, higher level 1, we can see that on average, across the country, 4.7% of students who took that subject got a H1. In the estimates coming from students, that figure was 13.8%.

There is no way on earth that students who did agricultural science in 2020 are three times better than the cohorts for 2017, 2018 or 2019, yet the calculated grade process alters that point to 9.1%. Even with the adjustment, the students received marks that are higher than they would be going by the historical pattern.

On the point raised by Senator Seery Kearney regarding the HPAT, I do not think it should be that conditional, given the year it has been. Students should just be allowed to carry the HPAT forward.

On the November examinations, there are a few difficulties with them that I do not quite understand. Question choice has been expanded for those who will sit the leaving certificate in 2021. Given that the cohort sitting the November examinations faced the same difficulties, why was question choice not expanded for the November examinations? They also have no opportunity to demonstrate their language skills in an oral examination whereas those sitting the leaving certificate in 2021 will have such an opportunity. Was that considered for the November examinations?

On the issue of stakeholders, what contact has the Department had with the Irish Second-Level Students Union, ISSU? What contact does the Department plan to have with it in the future to get its opinion on things? Many of the students who are sitting the November examinations feel aggrieved because they were given 100% in respect of oral examinations in March but now they are being penalised for sitting the examinations as that is being taken away. I find it really disconcerting that 12,215 people have appealed and that we are just trying to dismiss them and not acknowledge what they have been through and the injustice that has been done to them.

Mr. Dalton Tattan

On the issue of the HPAT, as raised by Deputy Ó Laoghaire, we can discuss that with our sister Department. I will take that issue away with me.

On the November exams and oral exams, we also have to ensure equity as far as possible with those who have just gone through the calculated grades process. It also simply was not possible for us to run oral examinations. We could not give the 100% because that would create an inequity with the group who have just gone through the calculated grades process where the 100% did not apply. Neither could we run the oral examinations because the schools are up and running and teachers are fully occupied. We have had to enable schools to engage more teachers because of the Covid restrictions.

Deputy Conway-Walsh asked about contact with the ISSU. The ISSU has been central throughout the process. It sat on the advisory group. We had several bilateral meetings with it and continue to engage with it. It was it through its surveys that argued for calculated grades and that was a big influencing factor in the decisions that we made.

Will it form part of the review and be central to it?

Mr. Dalton Tattan

Absolutely.

I offer a sincere thanks to Dr. Hislop, Mr. Tattan and Ms Feeney for their presentation. We had a good exchange of views and opinions. I thank them for their comprehensive answers. I apologise for having to cut them off on several occasions. I hope they understand. I thank all the members as well.

The joint committee stands adjourned until 1.20 p.m. on Thursday, 12 November 2020 when we will have a public meeting with the Minister for Further and Higher Education, Research, Innovation and Science, Deputy Harris, regarding the approval by Dáil Éireann and Seanad Éireann of a motion relating to the draft Technological Universities Act 2018 (Section 36) (Appointed Day) Order 2020. Before that there will be a meeting of the select committee to deal with the 2020 Revised Estimates with the Minister for Education, Deputy Foley, and the Minister, Deputy Harris.

The joint committee adjourned at 3.29 p.m. until 1.20 p.m. on Thursday, 12 November 2020.