Paragraph 7.1 reads:
7.1 Examinations Branch
Introduction
The Examinations Branch of the Department of Education and Science is responsible for administering all second level examinations in the State, including those leading to the award of Junior and Leaving Certificates. Other examinations catered for by the Branch include those covering Leaving Certificate Applied, Leaving Certificate Vocational Programme, National Council for Vocational Awards, Ceard Teastais, other teacher training and PLC courses (on an agency basis).
The Branches work includes organising the setting and distribution of examination papers, co-ordinating the set-up of examination centres, organising marking, issuing results and dealing with queries and appeals.
Table 1 illustrates the diversity and scale of the work undertaken by the Branch for the main examinations in 2001. Students from some 800 post-primary schools sat the Junior and Leaving certificate examinations in over 4,400 centres. In all, over 2 million examination components (written, aural/oral, practical and project) were marked
Table 1: Throughput for the Main Examinations in 2001
|
Junior Certificate
|
Leaving Certificate
|
Total
|
Examination Centres
|
-
|
-
|
4,415
|
Examination Superintendents Employed
|
-
|
-
|
4,434
|
Examiners Employed
|
-
|
-
|
5,682
|
Chief Examiners
|
44
|
62
|
106
|
Advising Examiners
|
197
|
197
|
394
|
Assistant Examiners
|
1,730
|
1,477
|
3,207
|
Oral Examiners
|
-
|
1,033
|
1,033
|
Practical/Project Examiners
|
-
|
-
|
942
|
Candidates
|
60,124
|
56,670
|
116,794
|
Subjects Examined
|
26
|
40*
|
66
|
Examination Papers
|
126
|
203
|
329
|
Grades Issued
|
582,765
|
390,371
|
973,136
|
Appeal Applications
|
1,712
|
8,974
|
10,686
|
Upgrades Granted
|
609
|
2,432
|
3,041
|
*Includes 8 non curricular subjects
The successful operation of the Public Examination System cost some €27 million in 2001 after taking examination fee receipts of €6.4million into account.
The direct costs incurred in running the 2001 examinations are summarised in Table 2.
Table 2 - Costs of the 2001 examinations
Expenditure Table
|
€m
|
Examinations Branch Payroll
|
4.3
|
Superintendent and Examiner Fees
|
16.2
|
Travel and Subsistence
|
8.2
|
Attendants Fees
|
0.8
|
Incidentals
|
4.1
|
Total
|
33.6
|
Review of the Branches operations
In August 1995, following operational failures in that years Leaving Certificate Art examination, the Departments of Finance and Education and Science agreed that the Centre for Management and Organisational Development (CMOD) of the Department of Finance and the Organisation Unit of the Department of Education and Science should undertake a joint review of the operations of the Examinations Branch "to make whatever recommendations are necessary for the future conduct of examinations".
At about the same time consultants were commissioned to carry out a specific review of the Arts Examination in 1995. The resultant report issued in 1996 recommended an extensive and comprehensive programme of change in this area. The joint Department of Education and Science/CMOD review (the Joint Review) took cognizance of the recommendations in this report.
The review team, which started its work in April 1996, concentrated on work practices, systems and procedures of the Branch. Areas covered were work volumes, the allocation of staff to different tasks, the incidence of overtime and the use of temporary staff by the Branch. It also covered the internal structure of the Branch, the level of supervision and the relationship between the Branch and the Inspectorate.
The Joint Review, published in January 1998, contained thirty-three recommendations, all of which were accepted by the Department. Implementation of the recommendations was delegated to the Branch. A further review of operational systems in the Department of Education and Science was carried out in 2000-2001 by a former Secretary of the Department of Finance, at the request of the Minister for Education and Science. This review concluded that the Examinations Branch was heavily overburdened and that, while the Branch has successfully drawn on new technology to enhance the efficiency and transparency of the system, additional benefits may arise from further outsourcing of specialist tasks including printing and logistics management. The review noted that there was on-going consideration by the Department of the desirability of achieving a better balance in the Junior Certificate by introducing elements of school certification rather than placing over-reliance on terminal written assessment, as at present. The review involved publication of a discussion paper and a countrywide consultation process. Such a rebalancing of the Junior Certificate to school certification would reduce the workload of the Branch.
The report by the former Secretary favoured a more radical solution, originally proposed in an earlier externally commissioned report, that a specialist body outside the Department manage the State Examinations. This suggestion has since been taken up by the Minister who announced in June 2001, that a State Examinations Commission with operational responsibility for the running of the examinations was being set up. This will bring the Department into line with best practice internationally. The new body is expected to come into being in October 2002.
Objectives and Scope of Audit
The principal objectives of the examination were to establish the extent to which the Department has implemented the Joint Review Groups recommendations and the effects of their implementation on overtime and staffing levels and on general operation of the Branch.
The scope of the audit included a review of files, analysis of statistics and financial data and discussions with senior officials in the Examinations Branch.
Audit Findings
Implementation Status
The Department did not set out a list of priorities or a timetable for their implementation. In the light of the 1995 Arts Examination crisis the Branch, under the direction of the Examinations Management Group, chose to implement those recommendations which impacted most on the public perception of the Branch, followed by technology measures to reduce the burden on the Branch. Issues impacting on industrial relations within the Branch have not yet been resolved.
While many of the recommendations have been wholly or partially implemented in the last four years some important areas have yet to be fully addressed which have the potential to improve the efficiency of the Branch.
These include cutting overtime, the development of a management information system and better organisation of workflows. These issues are considered in the following paragraphs.
Overtime
The review team considered that the level of overtime worked in the Branch was unacceptably high and that a radical revision of arrangements needed to be implemented, including increasing the number of permanent staff in the Branch.
The review team recommended an increase in staffing in the Branch, which it deemed was necessary to administer the examinations system. By February 2002, the total permanent staffing of the Branch was 90 officers compared with 73.5 in 1996. This staffing level is somewhat higher than the review teams recommendations and is attributed to the heavier workload since 1996.
Table 3 illustrates the trend in the overall level of overtime worked in the Branch from 1996-2001 in monetary and hourly terms. While this trend is currently downward, from a high in 1997, the total number of hours worked in the Branch in 2001 was 16% higher than in 1996. Costs incurred on overtime continue to be significant.
Table 3 - Salary and Overtime Costs 1996-2001
Year
|
Salaries ’000
|
Overtime ’000
|
Overtime as % of basic salary
|
Normal hours
|
Overtime hours
|
Overtime as % of normal hours
|
1996
|
1,864
|
536
|
29
|
157,244
|
47,120
|
30
|
1997
|
2,145
|
750
|
35
|
157,244
|
61,412
|
39
|
1998
|
2,383
|
907
|
38
|
189,335
|
59,823
|
32
|
1999
|
2,631
|
762
|
29
|
189,335
|
52,106
|
28
|
2000
|
2,952
|
769
|
26
|
189,335
|
55,290
|
29
|
2001
|
3,512
|
785
|
22
|
189,335
|
48,076
|
25
|
The review suggested that it should be feasible to eliminate the overtime worked between October and end-May if a concerted effort was made to substantially shift forward the cycle of preparation for the examinations. It also believed that a significant reduction in costs could be achieved by employing, in the period June to September each year, less expensive temporary clerical officers (TCOs) in place of overtime working by non-Examinations Branch permanent staff. There are insufficient volunteers from Examinations Branch willing to do overtime and, as a consequence, staff from other Departmental areas have to be deployed on overtime working. Industrial relations issues limit the TCO input to the Branch to 65 persons, working a maximum of 1,100 weeks during a 28 week period. Of the temporary staff deployed only porters may carry out paid overtime duties. These restrictions limit the Branchs scope to achieve payroll savings.
Since the Joint Review report the workload of the Branch has expanded through
· increases in the number of examinations
· candidates getting the right to view their original scripts
· internet access to results
· introduction of dial-up examination results.
Overtime has not been eliminated in the October to May period and there was no significant reduction of overtime in the June-September 2001 period. 127 officials were involved in the overtime roster over the period April to mid-October 2001. Twelve of these worked in excess of 600 hours in this period (an average weekly rate of 28 hours).
The Department is fully committed to reducing the Branches reliance on overtime working and points to a substantial reduction (about 22%) in overtime hours between 1997 and 2001, despite a considerable increase in the Branchs workload.
It notes that the work cycle and tight timescales between holding examinations and providing results and the manual nature of some of the processes will always generate a need for overtime working. The use of shift-work for TCOs was tried in the Branch in the mid-1980s but was found to be not satisfactory and was not pursued thereafter.
The Department considers that a formal agreement on the use of TCOs negotiated with the union representing the majority of staff in the Branch is a considerable achievement in view of the strong reservations of the union nationally. Recent legislative change and the establishment of the standalone State Examinations Commission will provide new avenues for progressing the issue of greater flexibility with the union in question.
General Operation of Branch
Aspects of the recommendations which impact on the general operation of the Branch may be considered under the following headings.
Management Information Systems
The review team was critical of the effort required of the Branch to provide it with basic management information relating to workflows, volumes of work, allocation of staff and overtime payments. In what is essentially a production line operation, it considered that the information it had sought to conduct its review was crucial to knowing the state of progress of work being undertaken at any point in time. It was unaware of any norms that had been set for staff which would help indicate productivity levels and highlight incidences of unevenness of workloads or less than full productivity. The report called for improvements in the management information available to the management of the Branch.
The Department accepts that while it has considerable informal information systems available on workflows, allocation of staff, progress of operations etc. there is a need to formalise these systems. It is intended that this will form part of the State Examinations Commissions agenda.
Examination stationery
The review team recommended that examination materials such as graph and tracing paper, answer sheets etc. should be sent directly to examination centres from suppliers in quantities agreed by Examinations Branch. This recommendation has not been acted on because of industrial relations issues involving the work of temporary porters. However, the Department has created a new examinations aide function and authorised the deployment of this resource for up to 15 days per examination year in each school.
The Department states that the objective of this new function is to provide for a greater role for schools in examinations delivery, to decentralise the current system, and to provide opportunities for the Department to improve the efficiency of examinations delivery. The new model also opens up the possibility of greatly reducing the Departments vulnerability in the area of examination paper security and non co-operation by teachers in supervising the state examinations.
While the approval of the Department of Finance has been obtained for the expenditure involved, the impact of the additional expense in the overall context of the examination system has not been computed.
The Department decided to defer further action on the examination stationery issue until agreement was reached with the unions on bringing into effect those elements of the 2001 Report approved by the government for implementation. Agreement has now been reached with the unions in that respect. It is now open to the State Examinations Commission to pursue outstanding issues relating to examination stationery. The introduction of the examination aide for schools was also a key dimension in ensuring school acceptance and positioning the external elements of the system for further internal change.
Production of question papers
The review group noted that as the Inspectorate saw no significant reasons to delay the setting of examination papers, these should be prepared before 1 April of the year preceding an examination. This would allow finalisation by the Inspectorate by the following 1 September and enable all printing work on these to be completed before end January when the demands on the Branch are normally low.
The review team felt that if printing of examination papers was completed by this deadline, the Branch would have sufficient time to have these packed before the end of May, as part of its routine operations. Bottlenecks continue to occur at the point of moving papers from draft to proof stage at Inspectorate level and as a result bulk printing is not being completed by the end of January. The Branch strives to meet this time frame but sees the role of the Inspectorate as vital to achieving the objective.
According to the Department there has been a strong improvement in the flow of examination papers to the printing stage in the past few years and further improvement is anticipated with the establishment of the State Examinations Commission. This improvement is due, in the main, to the introduction of Chief Setters at Junior Certificate and Leaving Certificate Applied level and improved schedules by the Inspectorate in dealing with draft examination papers. The Commission will have a dedicated professional/assessment cadre. Currently inspectors work as chief examiners is always competing with other Department wide pressures.
Collating data on pupils candidacy intentions
The Examinations Branch needs to get accurate information on the number of pupils sitting the various subjects, the levels being taken by each and the examination centre at which the pupils will sit the examination. The Branch relies on the Departments Post Primary pupil Database for most of this information but must seek confirmation from schools of any changes in the choices made. In order to do this the Branch uses what are referred to as turnaround documents to obtain updated information from schools and candidates.
The review team recommended that these documents be simplified for ease of input to the databases and that they be returned from the schools before Christmas for the Leaving Certificate and by the end of February for the Junior Certificate.
For the 2001 examinations, the Leaving Certificate documents were issued in the period 1 December 2000 to 29 January 2001 to be returned within 2 weeks, while the Junior Certificate documents were issued on 28 February to be returned by the 30 March 2001.
The Department states that the Branch is dependent on the flow of examination candidate information through the collection of pupil data by the Post Primary Pupil Database process. This process piggybacks on the capture of other data from schools each October. It is of the view that there would be considerable opposition at school level to a separate earlier process. Furthermore, Junior Certificate candidates have until the end of March to decide on the level at which they will sit their examinations. It has not been possible to achieve an earlier date because of the opposition of schools, teachers, parents and pupils, who argue that a decision on the level to be taken must be left until as late as possible.
This impairs the Branchs capacity to have a more even schedule and work in this area tends to be concentrated in the late April/May period.
Conclusions
While the Department has been active in implementing the Joint Review Groups recommendations, some key actions still remain to be taken four years on.
Over and above the actions recommended by the Joint Review Group, the Department has taken positive steps in improving customer service in this area through the use of modern technologies in particular.
The continued absence of a proper management information system in the Branch is a hindrance to its efficient operation.
The proposed State Examinations Commission will need to have a clearly enunciated strategy for addressing the human resource issues which presently constrain the efficiency of the operation of the examination system.
Observations of the Department of Education and Science
Following the difficulties with Leaving Certificate Art in 1995 the Department was dealing with a fluid and volatile examination landscape and chose to prioritise measures which were designed to re-build public confidence in the examination system. This approach was best suited to the unprecedented circumstances that applied at that time. On that basis new processes and measures were introduced which were not features of the examination process at the time of the review or identified by the review team as warranting implementation. Some of these flowed from reports produced by consultants in relation to the problems encountered in the 1995 Art examination and from a subsequent report by the same consultants on the Leaving Certificate Appeal system. Others were conceived by the Department itself. These included the introduction of bar-coding, alternative collection arrangements for examination material, introduction of examination appeal commissioners, automated dial-up and internet access to results and the mould breaking initiative to return marked scripts to candidates. All of these measures were implemented without additional permanent staff. Their introduction placed a heavy burden on an already heavily stretched IT resource in particular. The Examinations Branch has placed a heavy emphasis on finding IT driven solutions to problems because of the constant tension between the demands of the examination system and the availability of human resources. Some were part of an integrated approach to development. For example the huge logistical challenge posed in re-sorting scripts for return to candidates in August is enabled by use of sort codes to facilitate tracking of scripts from examination centres to the Department and to examiners throughout June and July. It is the firm view of the Departments top management that those measures with a quality assurance and customer service focus and supported by innovative IT solutions were a greater priority than some of the measures identified by the review that might have appeared to hold out the prospect of reducing the resources required. In essence, the Department chose to extract more and increase its services from within existing resources rather than maintain services with reduced resources.
The costs of the Public Examination System in Ireland are not out of line with those for broadly comparable examination systems, for example, those in Scotland and New Zealand. The review identified the human resources available to the Scottish Qualifications Authority (SQA) and compared activity volumes but its recommendation on permanent staffing fell well short of bringing the Examinations Branch to the level of permanent staffing in the SQA without any indication of how the gap that had been identified was to be otherwise covered. In the comparison with the Scottish model the review furthermore did not factor in the greater degree of devolution to schools in Scotland in relation to examination administration. For example, the entire process of appointing examination superintendents there is school based and does not usually involve teachers. Here because of the historical involvement of teachers in supervision, which they guard closely, the Department is forced into recruitment, appointment, assignment and payroll functions. Furthermore, in relation to the cost of running the examinations, it is worth pointing out that travel and subsistence costs account for €8.2m (25% approximately of overall budget) in 2001. These costs arise from supervisory duties performed by teachers who travel to out-of-town centres and the insistence on the almost exclusive use of an external visiting examiner model for oral and practical examinations. These are not features of examination systems in other countries.
The public consultation process seeking a better balance in the Junior Certificate involved countrywide public meetings, at which teachers union representatives, articulated with regularity, the continued opposition of the union to any movement to school certification.
While the Examinations Branch would have wished to be further advanced there has been some progress in the area of procedure manuals and form design. Procedural manuals have been compiled recently for the Leaving Certificate Applied operations. In the case of efficacy of forms the Branch has, for example, reviewed the use and effectiveness of its forms and, as a result, has developed generic forms for use on processes that are broadly similar. The agenda is moving swiftly along and the issues now are how, through the use of web enabled technologies, all interactions with schools, students and examiners can be improved. At time of writing for example candidates have been given Internet capacity to lodge appeal applications and to pay on-line by credit card.