Introduction

Since their advent 40 years ago, performance audits in the public sector have tended to focus on systems and practices. Despite intentions to produce more audits focused on whether government programs achieve their intended results, these results audits remain uncommon.

In this article, I present an approach, based on a simple model, that can help public sector auditors to plan and conduct results audits and report their findings effectively. This approach can also help identify root causes.

The Promise of Results Auditing: Still a Work in Progress

Performance audits were originally designed to examine economy, efficiency, and effectiveness (with restrictions on effectiveness in the mandate of some audit institutions). Disappointingly, the first generation of audits in the 1980s often looked like a Management 101 textbook—too focused on systems and practices, and not enough on whether the audited activities were producing results. The consensus was to shift to a results focus (which I refer to as “results audits”), but not much seemed to change in the years that followed. Audits have been still much more about whether audited activities are well managed than whether they deliver results.

Much of the early problem was that information on results was hard to obtain and to work with for the following reasons:

• The few results databases available were often not used for management decision making.
• When a database did exist, auditors faced the daunting task of first cleaning the data before they had something that could be analyzed or used as a sample frame. Or the audit required linking two data sets that had never been matched before; missing identifiers would reduce the number of successful matches.
• If sampling was required, the large sample sizes needed to obtain a representative sample allowing extrapolation were daunting.

Fortunately, the situation has improved with the digital revolution and the advent of powerful data analytics tools and artificial intelligence applications.

But, even with available data, the complexity of auditing results remains. The analytical techniques—the kind that program evaluators and statisticians use—are often new to auditors. In recent years, we have seen regression analysis and other statistical techniques used in performance audits, but the examples are still rare. And, even where auditors have the necessary statistical training, it can still be challenging to clearly explain the basis of the audit findings in the audit report or to the Public Accounts Committee.

Auditing Results Made Simpler: Modelling the Audited Activity as a Pipeline

The complexity of auditing results begins at the scoping stage. Auditors often find it complicated and time-consuming to complete the logic or process models required to understand the relation of inputs, activities, outputs, and outcomes in any given public sector program. For example, at the CAAF we teach four modelling techniques for planning (logic models, management plan/do/check, process models, and issue trees), plus another four Root Cause Analysis techniques (5 whys, Fishbone diagrams, Pareto models, and cause mapping). The resulting model is often not easy to understand and hence rarely makes it into the audit report. With so much complexity, it is not easy to find the right audit scope or determine where evidence collection and analysis should focus.

About the Author

Neil Maxwell worked for over 30 years at the Office of the Auditor General (OAG) of Canada, concluding his career there as Assistant Auditor General from 2007 to 2015. In his final years at the OAG, he was also the Product Leader for the performance audit practice and was appointed as Commissioner of the Environment and Sustainable Development on an interim basis.

In recent years, as a CAAF Associate, Neil has contributed to course development and delivered performance audit courses in many jurisdictions. He has also contributed to several research projects, including the CAAF Discussion Paper Approaches to Audit Selection and Multi-Year Planning, for which he was the lead researcher and author.

Neil received Master of Public Administration and Bachelor of Arts (Honours, Geography) degrees from Queen’s University.

Contact the author at:

neilrjmwell@gmail.com

I often use the simpler approach of modelling the audited subject as a “pipeline.” Everyone understands a pipeline. Something flows in at one end and flows out the other, barring any leaks. Applying this to an audit, resources flow in one end, pumped by activities subject to management controls along the pipeline, and results flow out the other end. In theory, all the intended results emerge at the end of the pipeline, but more often some of the intended results leak out along the way because management controls are not designed or implemented well.

In a recent audit of supplies (not yet published), I suggested using the pipeline model. Parliament provides funds at one end, and users receive supplies when they require them at the other. The audit looked for leaks all along the pipeline, beginning at the input end (Figure 1). A significant portion of the money authorized by Parliament looked to be unspent because of approval delays. Then procurement delays seemed to reduce the quantity of supplies flowing through the pipeline. Losses from expired supplies in storage added to the leakage. The audit sought to quantify leaks along the pipeline and determine why controls at each stage failed.

Figure 1 – Finding Where the Results Pipeline Leaks in a Supplies Audit

While it’s easy to see how a pipeline analogy fits an activity that distributes supplies, this analogy can also be applied to a wide range of results audits, as detailed in Table 1.

Table 1 – Using the Pipeline Analogy in Different Types of Results Audits

Using the Model for Root Cause Analysis

Many of the examples above also show how this pipeline model can be a simple way of identifying and reporting root causes. That should not be a surprise, because causal relations are just results relations in reverse—for example, if parts shortages result in unavailable equipment, then the cause of unavailable equipment is parts shortages.

Another recent audit (not yet published) used the pipeline model to look for root causes in a hazardous waste audit. In this case, inputs start with national policies; flow through activities that communicate standards and provide training at national, regional, and facility levels; and then flow through implementation at each facility. If everything worked well, all hazardous waste would be treated so that human health and the environment were protected. Laying out the components of the results pipeline and their flow helps auditors to have a good understanding of the processes and controls involved (as required by the applicable Canadian audit standard CSAE 3001), which in turns facilitates the identification of “leaks” and their root cause.

How to Apply the Pipeline Model to an Audit of Results and Root Causes

The standard steps in audit planning, examination, and reporting are listed below, with suggestions on how to apply the pipeline model in bold.

1. Gain the required knowledge of the audited activity—always the starting point of good audit planning.
2. Identify the most significant aspects of the subject, including the inputs to the audited activity, the key sub-activities, and the main intended results. Try depicting these in a simple pipeline graphic.
3. Conduct a risk assessment to identify the results issues with greatest inherent likelihood and impact, considering economy, efficiency, and effectiveness.
4. Gather indications of actual results. What shortfalls seem to be occurring? Look at possible root causes. Show on the pipeline where results are likely leaking, and which management controls seem to be failing. This allows you to refine your risk assessment to reflect residual risk (if your organization focuses scoping on residual rather than inherent risk).
5. Determine the audit scope based on the risk assessment, along with considerations of auditability and value added. Develop the audit objective and criteria to use.
6. Identify evidence collection and analysis strategies that will provide sufficient appropriate evidence on results and the management control weaknesses that cause results shortfalls.
7. Conduct the examination by applying the evidence collection and analysis strategy to gather sufficient appropriate evidence to assess each audit criterion.
8. Draft the report incorporating the significant findings. Try telling the story using the pipeline analogy—what results are lacking, where were the leaks, and which management controls are at fault?

Conclusion

Every performance audit is complex, especially those that examine results and root causes. Faced with complexity, the auditor’s best strategy is to keep it simple. I’ve found this simple analogy of a pipeline to be a powerful way to scope performance audits, and an effective way to report their findings. I hope these tips help!