What are the benefits of evaluation?

Government rules and guidelines for the effective governance, performance and accountability of programs expands the required performance information beyond the financial. Broader qualitative and quantitative performance information, evaluation and reporting is also expected and encouraged.

While this might seem a little scary, it actually presents a raft of exciting opportunities for program managers to showcase the successes of their programs and ensure that their teams are performing to its full potential.

We here at Grosvenor have a lot of experience working with Commonwealth and state government programs and so we understand what can be achieved through excellent performance monitoring and evaluation processes.

And while the benefits and opportunities presented by government rules and guidelines will vary from program to program, there are some that are universal.

Below we’ve outlined 6 reasons why program evaluation provides a fantastic opportunity for you and your team to shine.

1. It provides clarity for the team

Creating a framework which outlines clear responsibilities for performance encourages accountability and clarity as it gives you, your program delivery team and managers a way to be very clear on their responsibilities for monitoring and evaluating performance from the outset.

2. It will tell you how well your program is really doing

Evaluation helps program managers measure how well their programs are really performing to achieve the desired outcomes. This can confirm your suspicions that things could be improved or let the success of your program shine.

3. It encourages continuous improvement

By encouraging performance monitoring and evaluation within your organisation, including the creation of robust evaluation tools, you can create an environment focused on learning and the continuous improvement and success of your program.

4. You can measure performance throughout the entire program

By doing this you can identify what is working well and what needs to change at key points to ensure the ongoing relevance and performance of your program and make certain that you are well placed to respond to environmental or policy changes. 

5. It encourages honest conversations

Evaluation enables program managers to seek genuine input and feedback from stakeholders. This helps you to understand what your stakeholders really think about your program and how it could be better tailored to meet their needs. 

6. It allows you to access valuable information from other programs

Sharing performance information and evaluations allows you to learn from the challenges and successes of others and for others to learn from you. By leveraging the outcomes of others’ activities and applying them to your own situation, you can ensure your program is robust and functioning at its best.

So, whether you know your program is a success or you think it could be improved but don’t know how, program evaluation presents a great opportunity for all program managers to have a better understanding of their program’s performance.

By taking the time and effort to commit to your government evaluation rules or guidelines, program managers can ensure that they have robust processes in place to understand how well programs are really performing and continuously improve.

How do I monitor and evaluate performance?

An integral part of understanding how well your program has performed is having a monitoring and evaluation framework.

Following the delivery of an initiative, benefits evaluation can be undertaken.

Benefits evaluation is important because:

• It helps to confirm that the benefits established and defined in the initiative's development stage are being achieved and that the implemented initiative is running smoothly.
• It measures the extent to which the initiative achieves the outcome(s) sought (as described in the Business Case) and to what extent further justified actions are required to achieve and/or enhance benefits.
• It provides information that may be used to help inform future decisions.
• It provides a level of transparency and accountability for past initiative decisions.
• It provides learnings for future planning, assessment and decision-making.

The key activities undertaken in benefits evaluation include:

• Collecting, analysing and comparing benefits data
• Identifying key findings and lessons learned
• Disseminating the findings and lessons learned
• Updating the Benefits Management Plan and Benefits Register.

An agreed Benefits Management Plan guides the implementation of the benefit evaluation process. It confirms the scope and purpose of the review, enables an organised approach to timing and conducting the evaluation, implements the roles and responsibility of stakeholders involved in the evaluation process, and guides the benefit handover process to the benefit owners.

It should be noted that benefits evaluation is broader than a post-completion review as it encompasses planning actions necessary for the full realisation of project benefits, such as undertaking complementary projects or altering land use plans. These actions are frequently outside the direct control of the project proponent, so the Benefits Management Plan has to address how the responsible parties can be influenced to ensure their actions are aligned.

A benefit evaluation report may be prepared to document findings and lessons learned as part of this process.

5.1 Timing

Benefits evaluation should be conducted after an initiative has been delivered, at periodic intervals reflective of the timing and ramp-up of benefits (depending on the scale and associated behavioural change from an initiative). This allows for evaluations as benefits are fully materialised. This means that:

• Initial potential disruptions and business processes do not affect the evaluation results.
• Statistically significant collection of measurement data can be undertaken.

Benefits evaluation can also be undertaken as part of a post-completion review and findings from the evaluation process will help inform a post-completion review. Further guidance on post-completion review can be found in F7.

5.2 Collecting, analysing and comparing benefits data

5.2.1 Data collection

The approach for benefits data collection should follow the same methodology identified during baseline measurement in benefits planning. Data collection should be targeted. Since the benefits evaluation is explicitly linked to the benefits planning process, the methods outlined below should deliberately seek data and information that elucidate whether the benefits outlined in the initial plan have been achieved.

In addition to existing information about the initiative (i.e. Business Case, Benefits Management Plan and Benefits Profiles) and reports, information collection techniques may include the following:

• Questionnaires/surveys
• Interviews (structured or semi-structured and based upon issues developed for each stakeholder)
• Observation (i.e. of the service or asset)
• Collection of operational and service outcome data from the relevant agency or service provider
• Focus groups and workshops.

All data collected before or during delivery that will be used are reviewed and a decision made about what new data are needed. This requires specifying the characteristics of the data so that what is collected is useful and comparable to other datasets that are used. This may include:

• Specifying dates and times for data to be collected
• Specifying levels of accuracy of data and collection equipment
• Specifying volumes of data or sample sizes
• Other specifications particular to a type of data.

It is important to consider the availability and reliability of data to support the evaluation of benefit performance. The time and the cost of capturing the data should be relative to the scale of the benefits and initiative.

5.2.2 Compare and analyse data

Analysis of the information gathered will involve comparing what actually happened against what was predicted (as outlined in the Benefits Management Plan and supporting Benefits Profiles). This includes analysis of:

• Outcome performance (i.e. actual vs target)
• Realisation date (i.e. ahead of schedule/on time/lag and including reasons)
• Dependencies (i.e. cross-referenced to other benefits internal and/or external to the project)
• The justified actions required to realise the benefit
• Stakeholders impacted by the benefit
• Assumptions (i.e. about probability and/or value of benefit)
• Outstanding risks to future benefit realisation and corrective actions that can be taken to mitigate the risks
• Costs associated with benefits measurement
• Effectiveness of benefits management approach.

In examining the outcomes, consideration should be given to what caused any variances between what was expected and what was achieved. The recommendation may be to modify aspects of the initiative in question (if justified) so as to increase the likelihood of achieving the expected benefit or to take the observed variances as expected due to changes in the baseline.

5.2.3 Document findings and lessons learned

Findings and lessons learned arising from the analysis of benefits data should be consolidated and reported to inform:

• The ongoing operation of the service or asset (which is the final output of the initiative)
• Future initiatives.

In addition to understanding if the intended benefits of the initiatives have been fully realised, benefits evaluation should assess:

• What was done well and contributed to achieving expected and additional benefits
• What opportunities for improvement exist.

Lessons learned may include:

• How practical problems were overcome with strategic assessments, Business Cases, funding models, procurement processes, project knowledge management, effectiveness of benefits management and risk management
• Unexpected benefits that emerged
• Experiences of stakeholders and staff that provide useful insights for future initiatives
• Improvements to a jurisdiction’s benefits management practice.

5.3 Disseminate the findings and lessons learned

For benefits evaluation to have a positive impact, the results must be communicated to practitioners and stakeholders involved in planning and evaluating the initiative, and those involved in the management or operation of the initiative. The information gathered during a benefits evaluation can be used to inform planning and project management practices. An evaluation report may be prepared to document findings.

Those required to evaluate similar options in future will be able to use the information gathered to inform their decisions. Thus the benefits evaluation process creates a feedback mechanism through which evaluation practices can be improved.

A communications plan for the findings can be agreed with the benefit owners and key stakeholders to facilitate dissemination of the findings and report.

5.4 Update Benefit Management Plan and Benefits Register

Once benefits evaluation is completed, any changes to the Benefits Management Plan and Benefits Register should be agreed with the benefit owner and the changes made as appropriate.

Benefits evaluation may involve periodic updates to the Benefits Management Plan and Benefits Register with findings shared and documented.