Committee on Public Petitions díospóireacht -
Wednesday, 21 Feb 2018

English Junior Certificate Examination: Discussion (Resumed)

Chairman

The next item on the agenda is engagement with officials from the State Examinations Commission and the Department of Education and Skills.

Before we begin, I remind members, witnesses and persons in the Public Gallery to switch their phones to airplane mode please.

I welcome, from the State Examinations Commission, Mr. Aidan Farrell, chief executive officer and Dr. Tim Desmond, acting head of the examinations and assessment division. Ms Rita Sexton, assistant principal officer, is from the Department of Education and Skills. I thank them for appearing before us.

The witnesses have been invited here today to discuss petition No. P000013/17 from Ms Tara O'Sullivan on how to make the new English junior certificate examination fairer by adding 30 extra minutes. Ms O'Sullivan is requesting that the Department of Education look at the possibility of extending the length of time of the English paper for the junior certificate by 30 minutes to allow students enough time to complete the paper.

Before we commence, in accordance with procedure, I am required to read the following. By virtue of section 17(2)(l) of the Defamation Act 2009, witnesses are protected by absolute privilege in respect of their evidence to the committee. However, if they are directed by it to cease giving evidence on a particular matter and continue to so do, they are entitled thereafter only to qualified privilege in respect of their evidence. They are directed that only evidence connected with the subject matter of these proceedings is to be given and asked to respect the parliamentary practice to the effect that, where possible, they should not criticise or make charges against any person or an entity by name or in such a way as to make him, her or it identifiable.

Members are reminded of the long-standing parliamentary practice to the effect that they should not comment on, criticise or make charges against either a person outside the Houses or an official, either by name or in such a way as to make him or her identifiable.

I remind our guests, as we indicated in the committee's invitation, the presentation should be no more than five to ten minutes duration. I am proposing that, rather than read out the submission verbatim, because all of the members have received the submission, if the witnesses were in a position to summarise it in five minutes or so then we can have a question and answer session on the nature of the submission. If that is okay, I will call Mr. Farrell to open the proceedings.

Mr. Aidan Farrell

I thank the committee. Good afternoon on behalf of myself and my colleagues. We are pleased to be back with the committee this afternoon to answer any further questions. I refer to the petition that was signed by more than 10,000 candidates in advance of the 2017 junior certificate English examination and the subsequent appearance then of the petitioners before the committee in November. The committee then raised a number of issues with ourselves for further response.

At the outset, I am pleased to reiterate that the first examination of the revised junior certificate English examination took place last June. We were pleased to report to the committee that the examination transacted just as we would have wished, in terms of the time allocation, the demand placed on candidates and the conduct of the examination itself. We had taken steps prior to the examination, based on the petition, to make sure that would be the case. We were pleased that transpired.

The petitioners raised a number of issues in November. The committee asked us to comment on those and we are happy to do that. I will briefly mention what the issues were and then I will deal with them in turn. The petitioners felt they had been passed from Department to Department and their concerns were not adequately addressed, so they contacted us directly in March 2017.

A day later we were in the position to confirm to the petitioners that all steps would be taken to make sure that the examination could be completed by candidates within the allotted two hour time and that the exam would be fair to candidates. Subsequently, that turned out to be the case. There was some further engagement from the petitioners following my initial response later in March. Again, we replied a number of days later and reassured them of the particular situation. I am pleased that we were able to give that level of reassurance to candidates in general, and not just the four petitioners, about the conduct of the examination.

The next topic that the petitioners raised, which the committee has asked us to comment on, was that they understood the key philosophy of the junior cycle examination would be that candidates would have the opportunity to carefully plan, draft and redraft their responses to questions. In a response we set out that the assessment in the new junior cycle examination and framework is broader than what was traditionally the case. In the past, the assessment of junior certificate English would have taken place over two terminal examinations in June. Under the reformed junior cycle programme, assessment is much broader in terms of the planning of assessment and the requirement on candidates. Assessment takes place over the full cycle as opposed to just the terminal examination at the end of third year. The reason for this reform is twofold. First, we want to reduce the focus on terminal examinations and the relentless preparation and attempts to maximise performance in the examination, which research has shown does impact on student learning. There is now a focus on having assessments at various points over the course of the programme and giving feedback to students at the time. All of that helps students to learn better. We get feedback on how they perform and they learn how to improve their performance. The change reduces the focus on purely a terminal examination.

In the context of junior cycle English, there are two classroom-based assessments at school level, which are marked by the teachers in accordance with the national guidelines issued by the National Council for Curriculum and Assessment, NCCA. A broader suite of skills are tested by the school-based assessments than are tested in the final examination. In third year there is a classroom-based assessment where students undertake a number of pieces of creative writing and submit two of them. By way of preparation, they actively work on preparing, drafting, redrafting, finessing and finalising their pieces of creative writing, which leads up to the assessment event that takes place in the classroom and is marked by the teacher. As a consequence, in the terminal examination set by the State Examination Commission, SEC, in June of third year, we do not have to ask students to respond to extended writing tasks. We consider their application of knowledge and skills at a broader level than simply in terms of extended essay responses at that particular point. That scenario is very much in line with the whole philosophy that underpins reform of the junior cycle. The assessment has been designed to improve student learning and engagement with learning as opposed to just preparation for final examinations.

The third issue that petitioners raised, which the committee has asked us to comment on, is that the official sample papers were not consistent in structure, which meant that the mock examination papers did not reflect the final examination. I can categorically state that the SEC is happy with the quality assurance measures that we have put in place, both in relation to the preparation and development of the sample paper and also with the final examination, that took place in June of last year. We made sure that the examination could be undertaken within the allotted period, that it would test the syllabus objective and learning outcomes required in the syllabus specification for testing through that assessment, and it would be completed within the two hour timeline provided for the examination. As I mentioned at the outset, we are very pleased that turned out to be the case because of the various steps that we had taken.

When there is a change of syllabus or specification, as it is now termed, or a brand new subject is introduced, students and teachers cannot rely on a previous suite of examination papers over a long number of years. The purpose of sample papers is to assist teachers and students and, indeed, the providers of mock examinations and let them know the structure, style and format of the final examination. Sample papers aim to give as much help and assistance to students and teachers, in particular, in advance but recognising that there is always some level of uncertainty in terms of what the assessment of a revised or a new subject will look like when first examined.

Chairman

I understand that approximately 60,000 people completed the examination.

Mr. Aidan Farrell

That is right.

Chairman

Plus four individual complaints were made about time constraints.

Mr. Aidan Farrell

Yes.

Chairman

Was there a geographical spread in terms of the four complaints? Did they come from one entity?

Mr. Aidan Farrell

No. We received four individual complaints via email. Therefore, I have no idea of their geographical background.

Chairman

Mr. Farrell's colleagues can also respond to my queries.

Mr. Aidan Farrell

Sure.

Dr. Tim Desmond

There was no indication that the complaints came from a particular quarter.

Chairman

It is reasonable to presume that if there were only four complaints from various parts of the country then the vast bulk of the 60,000 people who participated in the examination were satisfied with the two hour allocation. I do not mean to put words into anybody's mouth here. I just want to get a sense of the matter from the witnesses.

Mr. Aidan Farrell

We last met the committee in June of last year, which was after the examination had taken place but prior to the marking. We have reviewed in some detail actual student experience and performance across the examination. We now have clear data that indicates that time was not an issue during the examination. The candidates did not run out of time before the end of the examination. The proportion of marks that were achieved in the later questions in the examination were not disproportionate to those that were awarded for the earlier questions in the paper, which might show some element of student fatigue or flagging towards the end of the examination. We are very happy that the examination did transact as we would have wished.

Dr. Tim Desmond

We closed the last meeting in June with the observation that we had the ultimate feedback because we had scripts from every candidate. Interestingly, and a slight digression, we moved to online marking this year. Therefore, the scripts were marked onscreen by viewing images of the script. One advantage is quality control. Another advantage is that one ends up with a phenomenal amount of data because data is trapped on every part of every question. At the time of the last meeting as many as 10,000 random scripts would have been marked or in the process of being marked, and we got some statistics. We considered the data in two ways. First, the attempt rate across the questions and across the paper linearly. There was no indication of candidates running out of puff or time from the point of doing things.

Second, we considered the statistics for the type of scoring that happened across the questions and across the paper linearly. We noticed that there was no noticeable drop in performance rates either. When one combines the two, whereby the questions were attempted and there was no reduction in quality, then the only conclusion to be reached was that there was not an issue.

Chairman

Some of us are under time constraints today and I apologise to the witnesses. Time is always a big issue here. I know the committee has sent correspondence to the witnesses about other issues as well but I want to open up the debate to other members. I call on Senator Kelleher to commence.

Senator Colette Kelleher

I thank the State Examination Commission for its paper and response. The witnesses have given a comprehensive response to the matters at hand.

They have presented good evidence to address the questions raised by the petitioner. My question is not on that because we can accept what the witnesses have provided as a very comprehensive response. I wish to welcome Ms Elena Roche who, coincidentally, is on work experience with me today. She sat the junior certificate examination and she signed the petition as well, as I discovered when we were on our way in. They can have a chat afterwards. However, I am interested to know what structures are in place to ensure that the users' experience in an examination is captured. Obviously, there is the evidence of the examinations being done, but what informal mechanisms are in place?

Second, presumably the commission has a board. Is there student representation on that board? Is that something the witness might consider?

Mr. Aidan Farrell

First, in terms of our interaction with the school system our role as the State Examinations Commission is to provide a national service to over 120,000 candidates every year. We operate in partnership with the education system so we work closely with school management, principals and teachers and then provide the service to the candidate. Regarding interaction with and feedback from stakeholders, that happens every day in our day-to-day business in the delivery of the examinations. Students are entered for the examinations through their schools. In the Irish system and tradition schools take a strong advocacy role on behalf of their students and interact with us in terms of providing the national service in a strong advocacy way with regard to particular issues arising for individual candidates, class groupings or at a broader system level, if they see that as important.

We also engage with teachers through teacher unions and subject associations. Immediately after every examination takes place in June we get feedback from the subject association for that particular subject. There is also feedback organised through the national media. There is a strong focus in Ireland on the examinations so there is a high level of reporting on them every year. There might be a separate debate there regarding the stress levels that might generate, but there is that strong level of feedback to us and the examination system in terms of how students have encountered the particular examination, how they have experienced it and the professional educator's view on it. That is important feedback for us because it helps us in framing the marking scheme that will ultimately be applied to students' work. In the Irish context we do not pre-set examination questions and pre-test them in advance of the examinations whereby we can test the standard of questions or how students will respond to them. Our question setters will have an expectation of how students will respond to a particular test item but that can change in terms of what actually happens in the examination hall. There can be a different experience. We get feedback from the individual teachers, the expert teachers operating through the media and through the subject associations. We will often get feedback directly from candidates or their parents if the candidates have had a particularly nervous or upsetting experience during the examination. These are all factors we take into account in framing the marking scheme.

Senator Colette Kelleher

Would the commission consider having a more formal consultation or feedback loop with candidates? The witness referred to the schools, but the school is first and the pupil is second. It is similar with the teacher. The media is a very direct route. This petition clearly hit a nerve given the number of people who signed it. As I said the witness addressed the questions beautifully, so there is no issue there. However, would more direct feedback from the candidates be something the witness might take on board and consider?

Mr. Aidan Farrell

Feedback comes to us in a number of ways in terms of the examination event. The first is that we have the 60,000 scripts. We had 60,000 scripts for English last year and so on for the other subjects taken by candidates, so we know in a sense how they have responded to the examination, where they have scored well and where they have scored poorly. If there was a question that we believed would work well in eliciting the knowledge or skills we were looking for that did not perform as well as we would have wished, we have that evidence in front of us. We engage with the subject associations, which represent all the teachers of the subject and who are hearing the feedback from their students. There is a strong level of teacher support at individual school level and then advocacy on behalf of their students. That all filters up to us and it happens immediately after the examination.

We have broader levels of engagement in terms of school management, school leaders, teacher unions and so forth across the span of the year. One thing we are considering this year, and we have put it in our strategic statement, is engaging in some stakeholder feedback through some formal processes this year. I anticipate that happening on an ongoing basis but outside-----

Chairman

For clarification, is that under the statement of strategy?

Mr. Aidan Farrell

Yes. It is something we will be doing this year. It is in the statement of strategy that it is to be commenced in 2018.

Senator Colette Kelleher

Would that include candidates as well?

Mr. Aidan Farrell

Yes. We would see ourselves engaging with students. To be frank, it will not happen immediately after an examination because-----

Senator Colette Kelleher

I understand. However, it is under consideration.

Mr. Aidan Farrell

There will be a way of drawing in the student voice, most likely through the Irish Second-Level Students' Union, ISSU. We will have a formal engagement with the ISSU with regard to the delivery and operation of the examination system as a whole, not regarding individual examinations.

Senator Colette Kelleher

I understand why that would be the case.

Ms Rita Sexton

I wish to add to my colleague's comments. The junior cycle framework meant quite significant reform within schools. As the English examination has been through the first cohort of students, the National Council for Curriculum and Assessment, NCCA, is undertaking a review of it and how the process worked out. That is under way at present. There is a public consultation element and focus groups with teachers so it will take account of all aspects of the junior cycle English in terms of the content, examination experience and experience of the classroom based assessment.

Chairman

Deputy Eugene Murphy wishes to intervene but with regard to your point about content, one of the issues we highlighted related to the role of Parliament and parliamentary committees. We perceive that there is a phasing out of that knowledge, as it were, at junior cycle level. I am relying on my notes but with the new well-being programme the current civic, social and political education, CSPE, is being phased out under the junior cycle reform programme. Where lies the ability of students to glean knowledge on the role of Parliament and parliamentary committees, for example?

Ms Rita Sexton

Yes, it is true that the junior cycle CSPE examination is being phased out, but CSPE is not being phased out. In fact, it is a core part of the well-being programme. Under that programme schools must include a number of core elements, including physical education, PE, social, personal and health education, SPHE, and CSPE. The schools have a lot of flexibility around that. They can undertake the new short course developed by the NCCA. In addition, I realise we are discussing the junior cycle today but there is a new subject at leaving certificate level, politics and society. I should also mention that there are very strong links between the education officer who is appointed to the Houses of the Oireachtas and the Department of Education and Skills. I foresee those links strengthening. There is much scope there for education on the parliamentary committee structure.

Deputy Eugene Murphy

I thank the witnesses for attending the meeting. There has been an open expression of views with the committee. I thank Mr. Farrell for the knowledge he has imparted to us. It is clear that there is a very good ongoing process, and it is good to hear that. There is no question that the vast majority of students can work within the timeframe, given the figures that have been presented to us relating to complaints. That is obvious. Four complaints is very low. However, would he accept that there might be a considerable number of people who will not make a complaint?

Many people have a fixed notion that they should give X amount of time to answering each question. I accept that the witnesses are the experts in this area. Dr. Desmond referred to students attempting to answer questions but not finishing their answers. Is it envisaged that the number of marks for project work will increase in the leaving certificate? It is great that examinations are moving in this direction because a significant number of students freeze when confronted with an examination paper and find it difficult to deal with examinations.

Mr. Aidan Farrell

I will respond first, after which I will ask Dr. Desmond will contribute. As I indicated to Senator Kelleher, a significant level of media coverage and interest is shown in the lead-up to the State examinations and in their conduct every year. All the main broadsheet newspapers and other newspapers provide daily feedback on how the various examinations went, how students responded to them and what their experience was. If individual candidates or their parents and teachers have concerns, the State Examinations Commission hear about them because the structures we have in place invite feedback.

Dr. Desmond referred to candidates attempting questions in the English examination. The average number of marks awarded in questions on the English paper was not lower for question No. 10 when compared with question No. 1. In other words, candidates scored as strongly at the end of the examination as they did at the start. There is no evidence to show that candidates flagged in any way.

Deputy Eugene Murphy

They may have done the final question first.

Dr. Tim Desmond

As it turns out, given the nature of the examination paper, there was a completion booklet and no choices were offered as a matter of course. In such cases, the majority of candidates tend to attack papers in a linear fashion, working from front to back. The average marks stacked up solidly across the paper and there was no obvious diminution of quality. It is not just a question of attempts but actual outcomes.

As far as feedback is concerned and the suggestion that some people who are not happy do not raise their concerns, it is perfectly true there are almost certainly a number of such persons. My colleagues and I have managed the emergency desks or complaint desks, for want of a better description, for the past six or seven years. If there is an issue with an examination paper, the complaint desks will receive significant communications traffic. Notwithstanding that this examination was new and, therefore, a flight into the unknown and that we had petitions, media reporting and uncertainty in advance of it, we experienced the quietest afternoon after an English examination that I have experienced. Beforehand, we wondered what would happen.

Chairman

The complaint desks did not receive 11,000 calls.

Dr. Tim Desmond

No, it was almost disturbingly quiet and highly unusual. We have occasional issues with papers where things have not been quite right or people were not happy and it is not unknown to be still at the desks at 7.30 p.m. or 8 p.m. on the night of an examination. It was remarkably quiet, however, after the English examination, with only a couple of emails received on the day and a further couple of emails received subsequently. It was very quiet. As I stated, the statistics also stack up. Much comes back to the initial disparity between the mock and sample papers. I would argue that it was this and not the sample papers themselves that spooked people.

Senator Colette Kelleher

Does the board of the State Examinations Commission consider the voice of students? Does the low level of complaints give the commission pause for thought that its complaints process may need to be more prominent or visible? Sometimes organisations make it difficult for people to complain, although I am not arguing that is the case with the State Examinations Commission. While the low level of complaints is great, is the commission satisfied with the visibility of its complaints process? If I were to go on its website, would I conclude it is easy to make a complaint?

Chairman

To be fair, given the degree of parent and pupil power we have these days and given that 11,000 people signed the petition, I expect people know their rights.

Senator Colette Kelleher

I am only asking a question, which is the job of members.

Chairman

It is a fair question.

Mr. Aidan Farrell

Dr. Desmond referred to our way of dealing with complaints or emergencies that arise during examinations. My experience has been that if there is any sense of unease among students regarding the difficulty of a paper or particular question or if a question does not work out or there is an error in the paper, our lines will light up and we will receive between 2,000 and 3,000 complaints and calls within a matter of hours. That is our experience and it shows that people are alert to the examinations and do not hesitate to contact us if a particular issue arises with one of them.

The Senator asked a question about the composition of the board of the State Examinations Commission. This was decided by the Government of the day when it established the commission in 2003. In setting up a separate body to run the State examinations body, the then Government examined the structure of other boards in education and broader civic society. Unlike, for example, the National Council for Curriculum and Assessment, NCCA, which is a representative and advisory council that includes representatives from third level interests, school management, school teachers, educationalists, industry and so forth, the State Examinations Commission is a delivery body. Our responsibility is to operate and deliver the State examinations system. The NCCA sets the policy, if one likes, the Minister then determines what will be the policy and our role is to deliver and operate that.

I understand that at the time of the commission's establishment, the Government of the day was strongly of the view that the role of the board was to provide oversight on behalf of the population regarding the conduct of State examinations. It was not intended to be a body representing particular sectoral interests, especially in the area of examinations. The board needed to be separate from that in terms of providing assurance and reassurance to the population on the conduct of what is regarded as a very important national educational service. For this reason, representative groups do not nominate members of the board. The Minister makes nominations to Cabinet which then determines who will be the five members of the board.

Deputy Denise Mitchell

I thank the witnesses for appearing before the joint committee and addressing our concerns. I am heartened to learn that only four complaints were received about the English examination. I thank them for their work.

Mr. Aidan Farrell

Deputy Eugene Murphy asked a question about project work, which I will answer in two parts, if I may. I will deal first with the broader revision of subjects and subject specifications that the National Council for Curriculum and Assessment has been undertaking for many years, particularly regarding the leaving certificate. I will then address the junior cycle because it differs significantly from the leaving cycle. Leaving certificate subjects are revised on a rolling basis by the NCCA.

It has certainly been the practice and policy over the past long number of years that when subjects are being revised we moved from a system of the traditional three hour terminal exam in June, which was the only assessment element where the person is tested on a written paper, to a system with project work, course work or skills-based activities in which we set out to assess a broader range of skills and knowledge than we can assess in a written examination. In practical subjects it can be determined by asking the student to design a communication graphic using CAD systems, or if the subject is construction or engineering the student may be asked to make an artefact, or in language subjects the student is given aural and oral exams as well as written exams. The idea is that the student is tested on a broader range of skills than was the case traditionally, and it also allows the student to bank some assessment activity separate to the three hour examination in June. This reduces the stress levels that are, in the case of the leaving certificate, undoubtedly part of the high-stakes examination.

The reform process around the junior cycle is fundamentally quite different. Research has shown that focusing on the terminal examination system at the end of third year does not encourage the sort of learning that we hope to develop as a society moving in to the 21st century. There is a broader range of assessment and reporting to students and parents in relation to the junior cycle than is possible through an external model such as the one operated in the State examinations system. Classroom-based assessments are now happening at school whereby students can be assessed on a broad range of skills and upon which they can get feedback at the time. Students learn better when they get feedback at the point of assessment and at the point of learning, on how they are performing and where they can improve, as opposed to feedback at the end of the third year. The research at junior cycle, in the context of performance in English and maths, has shown us that students are stabilising in first year with some students then disengaging in second year. Traditionally a lot of the focus in third year, especially from Christmas term onwards, was on preparation for the examinations. In many respects it was preparation for performance in the examinations as opposed to developing a deeper learning and engagement with the subjects. The research shows that the skills set is trying to encourage in students a deeper engagement in learning with their schools and teachers that is broader than the system as a whole and broader than the exam system. Research shows that this is possible through a broader range of assessment, through feedback and through formation than is possible through a terminal, external model. That is very much the focus there.

Chairman

I thank Mr. Farrell very much for that comprehensive response. Do members have any further questions?

Deputy Eugene Murphy

Such assessment is a fantastic idea and well done to Mr. Farrell on the way he has addressed it here today. I will tell the committee why. In the past, I believe that many students failed an exam because they just could not face the stress of questions and trying to put the answers together. Assessments, however, gives them an opportunity to excel and many of them do excel in that area. We all accept that students need to do some written work. Written work is very good, but assessment gives a fair chance to the students who may not be able to handle sitting an exam for two or two and a half hours to try to answer. There are a considerable number of such students. It causes them a lot of stress. The assessments are really good.

Chairman

There has to be some deconstruction of the thinking that existed heretofore where everything was geared towards the terminal exam. It is quite a struggle for many schools to try to deconstruct the pedagogical element and the content aspect of this. Where a school was geared towards trying to squeeze as much as possible in to the terminal exam, it was in the confined space of time because they were getting the desired points or desired results for the onward trajectory towards third level institutions and so on. The previous junior cycle was part of that trajectory or continuum. Maybe some schools continue to struggle in this regard. I do not refer specifically to Loreto College, which generated this petition, but maybe some challenges remain in the system around trying to deconstruct some of that thinking. This gives voice to petitions of this nature, and I speak in general terms.

The committee has heard absolutely compelling evidence here today. By any reasonable or objective analysis, the witnesses have given a very robust account of the nature of the correspondence entered into with the petitioners and have engaged willingly and openly with the petitioners. The witnesses have answered all of the questions on the time challenge. The research and qualitative analysis carried out clearly shows that the time was not such an issue as it related to last year's English exam and that people were able to get to the questions in a timely fashion. I believe the committee members are in agreement with that analysis.

The procedures in place for individual students to interface with the State Examinations Commission are robust; in this instance in this exam, there were only four people in the State who made a complaint about the time element out of 60,000 students who sat the same exam - and protecting their anonymity I assume they were not four individuals from one school and that they represent a wider geographical spread. The vast majority of students who took the exam were satisfied about the time constraints under which they were operating. I believe this to be a fair assessment.

Having regard to the evidence submitted to the committee by the State Examinations Commission and the Department of Education and Skills the committee deems it appropriate to close the petition and to send a copy of the submission made by the State Examinations Commission and the Department to the petitioners. The committee will thank the petitioners for their engagement with the committee. On behalf of the committee members, I express my gratitude to the witnesses for coming before the committee today. We strayed in to other areas also, but it was very useful to engage with the witnesses because it is helpful to members, as parliamentarians, to understand the dynamics of the State Examinations Commission and the Department on the exam structures. It was a useful tutorial for us and we are grateful for it. I thank the officials for coming before us today and for engaging with the committee.

The joint committee adjourned at 2.28 p.m. until 1.30 p.m. on Wednesday, 7 March 2018.