Chapter 8.1 of the report of the Comptroller and Auditor General reads:
Forest Service - Development of IT Systems
Background
The concept of developing a Forest Inventory and Planning System (FIPS), was first considered by the Forest Service in 1993. Following consideration of the results of a pilot study carried out in 1994, the Forest Service, which is a Division of the Department of Communications, Marine and Natural Resources (the Department), identified the continued absence of a national forest inventory and planning system as an impediment to securing optimum economic and social benefit from the forestry sector. In its 1996 Strategic Plan, it committed to developing a comprehensive inventory and planning system to provide geographical and environmental data on the national forest estate, for management, control and planning purposes. It was stated in the Operational Programme for Agriculture, Rural Development and Forestry (OPARDF) 1994-1999 that the compilation of an inventory which would entail identifying the location of forests together with their quality, species mix and production potential would also assist in ensuring the protection of forests from excessive or destructive exploitation. It was also felt that Geographic Information System (GIS) technology was the only realistic way in which Ireland could respond to requests from the European Commission for information on forestry resources, the development of which was largely funded by the Commission itself.
Projects Undertaken
Work on developing FIPS commenced in 1997. Five component projects were identified:
Forest Inventory Project
Classification
Ground Survey
Forest Planning Project - Soils Analysis
Forestry Grants and Premiums Administration System (GPAS)
Pilot Study
Main Contract
The Classification project entailed the establishment of a Geographic Information System using satellite imagery to locate, map, classify and quantify the area under forest in the country. It was envisaged that when this project was completed, digital maps, capable of being produced and read electronically, would be available showing the areas of the country under forest, and in addition would show the broad
categories of trees in each main forest.
The Ground Survey was to focus on the collection of volumetric inventory data to help produce statistically sound national data on, inter alia, production potential and entailed a more detailed analysis in which a representative sample of forests was to be surveyed for a range of information such as standing volume and timber production.
The Soils Analysis project entailed the undertaking of major research to establish the classification and productive potential of forest soils, which would assist in guiding the location and character of future afforestation by indicating the environmental, social, or economic sensitivity of areas proposed for afforestation. With the further integration of environmental data known to bodies such as the Office of Public Works, Fishery Boards and Dúchas, it was thought that the future planning of afforestation in the State would be greatly enhanced.
The purpose of the GPAS pilot study was to test the feasibility of transferring digital spatial data between remote locations, and in particular to integrate the Forest Service payments system/forestry classification system with forestry companies own map based systems, evaluate the results of this integration, and set up linkages with several County Councils.
The purpose of the GPAS main project was to develop a computerised system to pay forestry premiums and grants. The system which was to be map driven and capable of being used in remote locations, would also be compatible with the Forest Classification system through linkages with that system. Furthermore it would allow for the updating of the inventory records of the Forest Classification system by creating digital maps relating to new forests on which grants and premiums were being paid for the first time, which could be transferred over to that system. The system would also be compatible with the Land Parcel Identification System (LPIS) of the Department of Agriculture and Food which shows land parcels in receipt of EU payments in relation to a number of land uses including forage, setaside and arable. It was envisaged that the system when completed would enable the efficient payment of forest premiums, and provide effective controls to ensure that premiums would not be paid more than once on the same area of forest, or in respect of land on which the Department of Agriculture and Food had paid EU area based premiums. It would similarly enable that Department to check, using its computer system, that it was not paying any area based premiums relating to lands which were also attracting forestry premiums. The EU has introduced Regulations requiring the two Departments to carry out such cross checks, known as cross compliance checks, with effect from 1 January 2003.
Expenditure to end 2002 on the projects totalled €9.2m. This comprised costs in relation to contracts, computer hardware and software, licences, consultancy etc. but does not include Department staff costs.
The EU, under the OPARDF 1994-1999, undertook to fund 75% of the expenditure on FIPS.
Audit Findings
Time and Cost Overruns
There were significant time and cost overruns on the four projects undertaken as detailed in Table 8.1 and Table 8.2. The Ground Survey project was not undertaken.
Table 8.1 Time Overruns
|
Project
|
Planned Time (months)
|
Actual Time (months)
|
Status
|
|
Classification
|
11
|
53
|
Completed
|
|
Soils Analysis
|
36
|
77*
|
Not Yet Completed
|
|
GPAS - Pilot
|
15
|
37
|
Terminated Before Being Completed
|
|
GPAS - Main
|
13
|
19
|
Terminated Before Being Completed
|
*Latest projected time.
Table 8.2 Cost Overruns
|
Project
|
Contract Value
|
Contract Cost to 31 December 2002
|
|
€m
|
€m
|
|
Classification
|
2.00
|
3.00
|
|
Soils Analysis
|
2.17
|
2.70
|
|
GPAS - Pilot
|
0.30
|
0.55
|
|
GPAS - Main
|
0.85
|
1.45
|
Failure to Deliver Objectives
There were serious failures in relation to the GPAS project to deliver its intended objectives, and also some failures in relation to three of the other projects as follows:
While the Classification project did succeed in mapping areas under forest planted up to mid-1997, the data was not updated as envisaged, resulting in the Forest Service not having a complete and up to date picture of the national forest estate in digital format. It had been decided to achieve continuous updating through GPAS, but this functionality was not delivered, due to limitations in that system. Updating of the system will not be completed until a more sophisticated digitising component is developed which is being undertaken as part of a project which is due for completion at the end of 2004. While the lack of up to date data limited, to a degree, the use made of the system for planning, compiling reports on Ireland's compliance with environmental obligations, and answering queries from the public and a range of public and private entities, the new system was, according to the Department, a considerable improvement on the inventory records which had existed previously.
Department papers indicated that the completion of the national Inventory project, an integral part of which was the Forest Ground Survey, was considered vital if Ireland was to meet its national and international commitments to manage the country's forests in a manner consistent with the principles of Strategic Forest Management. While it was planned to commence the Ground Survey in 1998, it was delayed because of the transfer of staff resources to other elements of FIPS. Recently, the Department decided to commence the survey on a pilot basis.
The direction of the GPAS pilot study changed when it became apparent that the existing Forest Service grant administration database was not capable of being linked with the private forestry companies' computer systems. The study was terminated in November 2000, without having achieved its objectives. Essentially, it had been overtaken by the main GPAS project, which had commenced in January 1999. While no doubt some useful information was acquired, this fell short of what was expected when the contract was placed, and did not help in the planning and execution of the main GPAS contract to the extent that was envisaged. However the project did deliver a data capture/digitising tool that was used extensively in the main GPAS project.
The contract relating to the GPAS main project had to be terminated prematurely in March 2000 following the rejection by the Government Contracts Committee of a proposed extension to the contract which was estimated to cost €1.4m. The system delivered following termination of the GPAS main project, while capable of processing approximately 15,000 grant and premium payments each year, was limited in its functionality. It had been envisaged that data in the national Forest Classification system would be updated from the GPAS payments system as new forests were planted and as applicants were paid grants and premiums on these forests for the first time. However this updating has not taken place due to limitations in a digitising component, and so the data in the Classification system is incomplete, and cross checks with the Department of Agriculture and Food and the matching of all payments with discrete areas of land cannot be done electronically. The effectiveness of what was seen as a vital and useful payments control check is, therefore, greatly weakened.
The system was not installed at remote sites to enable forest ground staff to access it, and because GPAS is not map driven, environmental constraints must be checked by either referring to hard copy maps or to a separate Departmental computer system known as MAPS.
Furthermore because GPAS does not fulfil all Forest Service data capture requirements, different types of data relating to soils, tree classifications, etc. are held in different systems, which makes access more difficult and time consuming.
Because of the failure of GPAS to deliver its intended objectives a new project has been initiated (IForIS) to develop a computer system capable of delivering most of the original objectives of GPAS.
Following a tender procedure, a vendor has been chosen to build IForIS and contract negotiations are under way. The total cost of the IForIS project, including data migration elements, is expected to cost in the region of €3.5m and to be completed by the end of 2004.
Project Management Control Weaknesses
Because of serious cost overruns and delays experienced on the GPAS project, coupled with the refusal by the Government Contracts Committee to sanction any further extensions to the contract, the Department engaged consultants to carry out a review. The consultants' report identified major project management weaknesses in the handling of the GPAS project in the following areas:
Invitation to tender
Tender Evaluation
Contract Negotiation
Contract Changes
Communications and Reporting
Quality Management
Administration
The main findings of the consultants are set out in Figure 3.
The time and cost overruns and failures to deliver objectives on some of the other projects would suggest that they also were not sufficiently well managed from the point of view of good planning, adequate quantification of required work, deployment of adequate resources to complete tasks, monitoring of progress, and taking effective action to deal with difficulties arising.
This view would appear to be supported by the fact that the Department did not endeavour to enforce penalty clauses on any of the contractors, which would suggest that it accepts that the majority of the blame for the time and cost overruns and output deficiencies was attributable to it.
Procurement Issues
In respect of the GPAS pilot study it was noted that:
Three private forestry companies approached the Department in 1997 with a view to testing the feasibility of transferring data electronically between forestry companies and the Forest Service and other interested parties.
The Department agreed to assign responsibility for managing the pilot study to one of the companies and to provide funding of €219,508.
This proposal was treated by the Department as an application for grant aid and payments in respect of this project management function totalled €239,562 over the period of the study.
As part of the study, tenders were sought from three suppliers for the provision of computer hardware, software and related services. A tender for €54,700 was accepted by the Department.
Between acceptance of the tender and the signing of the contract, the contract sum had increased to €113,000. Thereafter, additional modifications to the contract were made on four occasions. None of these was subjected to competitive tendering procedures.
In total, €303,346 was paid for computer hardware, software and related services.
Total expenditure on the pilot study amounted to €542,908.
Figure 3
Invitation to tender (ITT)
Work was not adequately quantified prior to going to tender
Definitions and quantities in the ITT were sometimes ambiguous and open to different interpretations
Time schedules specified in the ITT were unrealistic
Gross underestimation of the volume of data which had to be captured
Quality and Control procedures for data capture were not adequately defined
Lacked comprehensiveness
Tender Evaluation
Personnel who evaluated the tenders lacked experience and expertise
The scoring system was flawed
Reasons for large variations in tenders were not analysed
Insufficient attention was given to quantifying the time which would be given by the key contractor personnel to the project
Contract Negotiation
A large element of the work was carried out before an agreed contract was in place
Many of the assumptions made by the Department in setting out the objectives and scope of the project were unsafe
Contract Changes
Issues arising which gave rise to contract changes were not documented or reported other than receiving token mentions in project summary reports or Project Control Team (PCT) meetings
Contract change control procedures were not always implemented
Communications and Reporting
Information on progress, the management of changes to the project and revised schedules and deliverables did not appear to have been effectively communicated to senior management
The Department lacked suitably skilled and experienced staff and was unable to evaluate technical statements made by the contractor
Undue reliance on informal discussions and ad-hoc meetings
Quality Management
Adequate quality standards and procedures were not agreed with the contractor
Requirements for standards in data capture were not documented by either the Department or the contractor
Administration
The Department did not allocate appropriately skilled or experienced staff to the GPAS project
There was no Database Administrator in place nor were there suitable candidates for this role within the Department
A high percentage (30%) of grant applications were treated as exceptions to standard Departmental procedures which was likely to make the newly installed system overly complex and costly to develop and maintain
Significant staff resources were diverted from the project to other activities which adversely impacted on the efficient implementation of the project.
Accounting Officer's Response
The Accounting Officer stated that there were circumstances in the Department and the Forest Service in the period 1997-2000 that had either a direct or indirect impact on the FIPS projects.
The Classification project, the GPAS pilot project and the Forest Soils project had all commenced in 1997, when there was a general election, a change of Government and the Forest Service was transferred from the Department of Agriculture, Food and Forestry to the Department of Marine and Natural Resources.
The Forest Service had gone from a situation where it had the support and back-up of a substantial IT division to a Department where there were less than four full-time IT staff available.
In October 1998, the Forest Service moved to Wexford as a result of a Government decision on decentralisation. This move resulted in the loss of approximately 90% of the experienced Forest Service staff and the consequent introduction of new staff with no experience or knowledge of the Forest Service business. Furthermore, the senior manager position had remained in Dublin.
During this period, the IT Unit at the Department of Marine and Natural Resources was significantly under-resourced and under-staffed.
Following the move to Wexford in October 1998, electronic communications structures were poor, and it was late 1999 before an effective e-mail system was installed, and internet mail was available to all staff.
These factors put considerable strain on staff and management in delivering business outputs over the period and, inevitably, had an adverse impact on progressing all of these projects, especially the GPAS projects.
He offered the following responses to a number of specific questions which I put to him on the management of the project:
Time and Cost Overruns and Failure to Deliver Objectives
The main reasons for the time and cost overruns?
In relation to cost overruns the Accounting Officer stated that poor quality national geographical datasets, and limitations in the satellite imagery, resulted in additional costs of €385,000 and €613,000 respectively on the Classification project. The latter increases resulted from the fact that the satellite images only enabled 504,333 hectares out of an estimated figure of 570,000 hectares to be mapped, and so digital maps had to be created from existing printed copies for the remaining area. The purchase of additional equipment and restrictions due to the threat of Foot and Mouth Disease resulted in higher costs of €104,000 and €177,000, respectively, on the Soils Analysis Project.
He stated that there was a delay of two years in formally accepting the Classification system because of bugs in the computer software. Eliminating these bugs represented only 1% of the overall project and skewed the completion date unfavourably. No additional costs were associated with this delay. Further delays resulted from dealing with the poor quality of datasets provided to the contractor by the Department and from making up the shortfall in the area mapped through satellite imagery. Delays in receiving Department of Finance sanction to hire contract staff, movement restrictions due to the threat of Foot and Mouth Disease, and the poor quality of input datasets had caused a delay of about 2 years on the Soils Analysis project.
Why, in view of the cost and time overruns on the projects and the failures to deliver the intended outputs, the contract penalty provisions were not invoked?
The Accounting Officer stated that the invoking of penalty provisions on the Forest Classification and Soils Analysis projects would have represented a risk for the Department, in view of the quality issues with the input data provided. Moreover, the invoking of penalty provisions was considered the option of last resort in terms of resolving what were complex difficulties. Adopting an adversarial approach would have added further to costs and time and the preference was to work through the problems with the contractors and this is what happened. He stated that there were no penalty provisions in the GPAS pilot project contract and in the case of the main GPAS project the findings of the independent review indicated that the invoking of penalty provisions was not an option.
The impact which the delays in completing the projects and the failure to deliver all of the intended outputs has had on the formulation of an indicative forestry strategy, the management and administration of the Forestry aid and plantation programmes, and the ability of the Department to fulfil its EU and International obligations?
The Accounting Officer stated that its ability to undertake its functions in the areas outlined was not materially affected by the delays on the Forest Classification and Soils Analysis Projects.
The development of county-based indicative forest strategies, which could have commenced in August 2000, was delayed by the decision to devote relevant staff resources to the wrap-up phase of the GPAS main project.
While the GPAS project failed to deliver its objectives, the system did deliver a simpler Grant Payment System, which enabled the Forest Service to:
Pay 15,000 premiums per annum
Process approximately 8,000 new applications
Complete the required EU documentation on recipients of grants and premium payments, and enable payment details to be issued electronically to the Departments Accounts branch.
However he also stated that the range of management information available was limited so that, for instance, lack of information on individual plantations has affected the Departments ability to fulfil EU obligations with regard to cross compliance.
Also, in relation to environmental checks, Forest Service staff were carrying out these checks at the present time on an existing Departmental platform called MAPS (Minerals Administration Programme Support Services).
When the Department expects to have a system in place to achieve all of the original objectives set for the FIPS project?
The Accounting Officer stated that the IForIS system, which was currently being developed would, inter alia, deliver a map driven payments system that would enable the Department to process grant and premium payments in relation to over 16,000 customers and amounting to over €70m annually, in a secure database environment which would be subject to the latest national and EU auditing standards. It was expected that the project would be completed by the end of 2004.
Project Management Control Weaknesses
If the Department accepted the findings of the consultants review in relation to the management of the GPAS project?
The Accounting Officer stated that his Department accepted the review findings. In his opinion, the review which was commissioned by the Secretary General at the time, demonstrated an acknowledgement within the Department that there had been project management deficiencies but also demonstrated a willingness to identify these weaknesses, learn from the experiences and take steps to ensure that future projects were better managed.
If the time and cost overruns on three of the other projects, and the delay in commencing another, indicated that poor management control was also exercised over those projects?
The Accounting Officer stated that he would agree that the findings of the consultants in relation to the GPAS project could also apply to the GPAS pilot project.
In relation to the Classification project, he stated that while the quality of input datasets should have been assessed prior to the issue of tenders which resulted in additional costs of €385,000, his Department had no control over the classification results of the satellite imagery which prompted the additional expenditure of €613,000.
In relation to the Soils Analysis project, he acknowledged that it had taken considerably longer than expected and that this was due in part to staff recruitment and data quality issues. However, he also stated that this project was unusual in that it was fundamentally a research and development project, the methodology for which was being revised and developed throughout, and that in all other respects the project had been well managed. Over 90% of original deliverables were to hand and the financial element had been managed very tightly such that the cost overrun has been limited to €400,000 or just over 15% of the original budget.
In relation to the Ground Survey component of FIPS, the reason why this had not been completed was due to the prioritisation of limited Departmental resources. The development of GPAS and the payment of grants and premiums had been designated as a priority over the Ground Survey.
The steps taken to ensure that future projects would be better managed?
The Accounting Officer stated that there had been massive changes in the Department vis-à-vis the structures and the expertise to manage projects undertaken. The Departments Information Systems Division had increased in size from eight people in 1999 to its current level of sixty people comprising twenty management and administrative staff and forty technical staff, which included specialists in database administration, GIS development and project management. The Department had also implemented a Project Management Framework to manage the delivery of IT projects. The Departments Information Systems Steering Group was now chaired by the Secretary-General and it was through this forum that senior management approved new IT strategic programmes and projects, prioritised projects and monitored the progress of major IT investments.
He also stated that the Department reported formally to the Centre for Management and Organisation Development (CMOD) in the Department of Finance, which involved impact statements prior to instigating a new project, and final costs/impacts at the end of the project. In addition, there were discussions with CMOD on technology issues and generally where it was considered that CMOD could add experience or knowledge to resolving issues encountered.
In view of the lack of experience and expertise, which existed in his Department at the time, on the management of complex IT projects, whether advice or guidance was sought from the Department of Finance on the management of the FIPS project?
The Accounting Officer stated that in hindsight it was clear that the GPAS project would have benefited from close consultation with the Department of Finance at both the planning and implementation phases. However, this was not seen at the time because of the lack of experience of staff working on the project, until the wrap-up stage of the project in March 2000. At that stage, CMOD was approached for advice on how to achieve the best outcome in the prevailing circumstances, and a representative of CMOD attended GPAS steering committee meetings thereafter.
As neither the soils analysis nor forest classification projects involved developing complex IT systems and the necessary expertise in forestry existed in-house, it was felt that the Department of Finance did not need to be consulted.
In view of the serious errors and deficiencies found by the consultants to have existed in defining and quantifying the work required to be carried out on the GPAS contract, if consideration was given to seeking redress from the relevant service providers?
The Accounting Officer stated that one of the fundamental errors highlighted in the GPAS review was the omission of a User Requirement Document before the contract went to tender. The document prepared by the advisers was a systems design document which wasnt based on a defined set of user requirements - this was to take place later as part of the GPAS project itself. Once the Department had begun the user requirements definition process, the systems design document became largely redundant as it did not reflect the requirements of the users.
Procurement Issues
Why national and EU procurement procedures weren't complied with on the pilot study?
In regard to project management, the Accounting Officer stated that the records in this case show that the proposers presented the project as, and that it was treated as, an application for grant-aid under the OPARDF and not as a contract for services.
In regard to the provision of computer hardware, software and related services, the Accounting Officer stated that a tender process did take place to procure these services at the outset of the pilot project. However, as the focus of the pilot project changed, so did the financial extent of this element of the pilot project. Additional services were dealt with as extensions to the original contract for hardware, software and related services.
EU Aid
If the aid granted to the projects by the EU was conditional in any way on the projects being successfully completed according to plan and, if so, if it was possible that the cost and time overruns and deficiencies in outputs delivered might result in some or all of the funds having to be repaid to the EU?
The Accounting Officer did not furnish a view on whether it was likely that the EU might impose financial penalties because of failure to complete the project to plan, other than stating that the Department had not made any financial correction to the final claim for assistance because of noncompletion of the project.
He stated that €4.66m had been claimed and received in respect of the 1994-1999 OPARDF.
Chapter 8.1 draws attention to the management of the development of key IT systems for the forest service. Back around 1997 work commenced on a computerised forest inventory and planning system that was intended to modernise the management of our forestry resources and streamline grant payments. It was originally envisaged that the five component subsystems would be implemented within three years, but it did not work out that way. Members will note on page 128 of the report that only one of the five subsystems has been completed. One is ongoing, one has just about started and two others were terminated before completion. The contract costs alone for the four subsystems that went ahead total €7.7 million for what was provided as against contract values of €5.3 million for the delivery of completed systems. When one adds in related equipment and software costs, total expenditure on the project up to end 2002 amounted to €9.2 million. That does not include departmental staff costs, which in the later stages were considerable.
The Accounting Officer explains the reasons for the shortcomings in my report and, therefore, I will not repeat them. I have no doubt that he will refer to them in his opening statement. They suggest that the project was not sufficiently well managed from the point of view of good planning, quantification of the required work, deployment of resources, monitoring of progress and taking effective action to deal with the emerging difficulties.
Members will note that a consultants' report was commissioned on one of the subsystems that was aborted. A summary of their findings is on page 131 and members will note that it is critical. A new system incorporating most of the original objectives of that subsystem has been initiated and the Accounting Officer informed me that it was expected it would be completed by the end of 2004 at an estimated cost of €3.5 million. I am sure he can bring the committee up to date on how that particular new system is progressing.
Apart from project management weaknesses, there were also procurement issues surrounding one part of the project, namely, the pilot study to test the feasibility of transferring data between remote locations. This involved trying to integrate the forest service's system with forestry company systems and evaluating the results. One of the companies that approached the Department with the proposal was assigned the task of managing the pilot study and was paid nearly €0.25 million for this. There was no recourse to tendering as the Department treated the payment for the management service as grant aid to the company.
Over and above the management of this study, it was necessary to procure computer hardware, software and related services. In this case, tenders were sought from three suppliers and one for €54,700 was accepted. Over time there were several modifications to this contract, none of which was subjected to competitive tendering. The final cost of this particular contract was €303,000. This does not say much for the way the work was scoped and ultimately it led to a situation where the principles of public procurement were compromised.
