Language selection

Search

Evaluation of Performance Measurement and Reporting Programs – NAHARP and NCGAVS

Report
Office of Audit and Evaluation

The AAFC Evaluation Committee recommended this evaluation report for approval by the Deputy Minister on June 29, 2012.


Executive Summary

This evaluation examines the relevance and performance of Agriculture and Agri-Food Canada's (AAFC) Environmental Performance Measurement and Reporting programs – National Agri-Environmental Health Analysis and Reporting Program (NAHARP) and the National Carbon and Greenhouse Gas Accounting and Verification System (NCGAVS). These programs support AAFC's Strategic Outcome of "an environmentally sustainable agriculture, agri-food and agri-based products sector".

The evaluation was conducted by the Office of Audit and Evaluation (OAE) in accordance with the Treasury Board Policy, Directives and Standards on Evaluation (2009). The results are intended to inform the development of programming under the next agricultural policy framework, Growing Forward 2.

Background and Profile

NAHARP and NCGAVS aim to improve AAFC's ability to report on the agriculture sector's environmental performance by developing models to monitor and report on agriculture's impact on the environment.

NAHARP develops science-based environmental indicators specific to the agriculture and agri-food sector and analytical tools to use this information to enable and support the policy development and assessment process. AAFC began developing agri-environmental indicators (AEIs) in 1993 with the Agri-Environmental Indicator Project. NAHARP was then introduced in 2003 under the Agriculture Policy Framework (APF). Under Growing Forward, NAHARP was allocated $9.96 million (Vote 1) over four years to continue developing and reporting on AEIs.

NCGAVS develops scientific models for measuring greenhouse gas (GHG) emissions resulting from Canada's agricultural sector to support Canada's international reporting obligations under the United Nations Framework Convention on Climate Change (UNFCCC). Environment Canada is responsible for reporting to the UNFCCC, but allocates the reporting of GHG estimates on agricultural lands to AAFC through an MOU. NCGAVS was originally funded by Environmental Canada from 2000 to 2005. The program is now funded by AAFC as part of Growing Forward. NCGAVS was allocated $3.32 million (Vote 1) for four years under Growing Forward to continue measuring and reporting on GHG emissions resulting from the agricultural sector.

Methodology

The evaluation gathered quantitative and qualitative data using the following lines of evidence: a document and literature review, analysis of program performance information, key informant interviews and a survey.

Key Findings

Canada has ongoing international and national commitments to report on the impacts of agricultural practices on the environment. Agriculture interacts with the environment and therefore it is necessary that the agriculture sector develops tools to track the results of its agri-environmental policies and programs.

Further, the objectives of AAFC Performance Measurement and Reporting Programs are consistent with AAFC strategic outcomes and federal government priorities related to reporting on the health of the environment. The federal government has a history of supporting environmental reporting and AAFC has responded to this by implementing agri-environmental monitoring and reporting as part of its Growing Forward policy framework.

In terms of the achievement of outcomes, NAHARP informs decision-makers and policy-makers about conditions and trends as they relate to key agri-environmental issues. NAHARP products are primarily used as a reference tool for planning or communication purposes. However, NAHARP is achieving limited success in the achievement of outputs and outcomes.

NCGAVS is making progress towards the achievement of outcomes and meeting requirements as set out in the Memorandum of Understanding (MOU) with Environment Canada (EC).

The evaluation identifies several areas requiring attention:

  • The increased focus on place-based approaches and the resulting need for performance data at a more detailed level of scale requires AAFC to reassess its role in agri-environmental monitoring and reporting.
  • There is limited data on the achievement of outcomes for both the NAHARP and NCGAVS programs.
  • Pressures on the NCGAVS program related to internal analysis and reporting requirements has led the program to re-assess its working priorities with EC.
  • Assessing the efficiency and economy of the NCGAVS and NAHARP programs is difficult in the absence of a system to track how much time AAFC scientists spend on activities associated with different programs.

Recommendations

The evaluation identifies the following three recommendations:

  1. The Agri-Environment Services Branch, in consultation with the Research Branch (the new Science and Technology Branch as of July 1, 2012) should work with the provinces and territories to identify approaches for developing and reporting on agri-environmental indicators that are aligned with the needs of both federal and provincial agri-environmental policies and programs.
  2. The Agri-Environment Services Branch, in consultation with the Research Branch (the new Science and Technology Branch as of July 1, 2012) should review AAFC's role related to reporting on greenhouse gas (GHG) emissions for the agriculture sector to ensure alignment with AAFC and the Government of Canada's mandate and priorities.
  3. The Agri-Environment Services Branch, in consultation with the Research Branch (the new Science and Technology Branch as of July 1, 2012) should develop a system to track, within a reasonable approximation, the relative effort that scientists invest in activities associated with the different programs from which they receive funding.

List of Acronyms

AAFC
Agriculture and Agri-Food Canada
ADM
Assistant Deputy Minister
AEI
Agri-Environmental Indicator
APF
Agriculture Policy Framework
AESB
Agri-Environmental Services Branch
CCAF
Climate Change Action Fund
CFS
Canadian Forest Service
CRAM
Canadian Regional Agriculture Model
DPR
Departmental Performance Report
EC
Environment Canada
G&C
Grants and Contributions
GHG
Greenhouse Gas
NAHARP
National Agri-Environmental Health Analysis and Reporting Program
NCGAVS
National Carbon and Greenhouse Gas Accounting and Verification System
NRCAN
Natural Resources Canada
OAE
Office of Audit and Evaluation
OECD
Organization for Economic Cooperation and Development
PAA
Program Activity Architecture
PMS
Performance Measurement Strategy
RB
Research Branch
RPP
Report on Plans and Priorities
RIAS
Regulatory Impact Assessment Statement
TB
Treasury Board
TBS
Treasury Board Secretariat
UNFCCC
United Nations Framework Convention on Climate Change

1.0 Introduction

The Office of Audit and Evaluation (OAE) of Agriculture and Agri-Food Canada (AAFC) conducted an evaluation of the National Agri-Environmental Health Analysis and Reporting Program (NAHARP) and the National Carbon and Greenhouse Gas Accounting and Verification System (NCGAVS). These programs support AAFC's Strategic Outcome of "an environmentally sustainable agriculture, agri-food and agri-based products sector". This evaluation was a requirement under AAFC's Five-Year Departmental Evaluation Plan. With the Growing Forward Framework AgreementFootnote 1 expiring at the end of 2012/13, the evaluation will help to inform planning for the next phase of policy and program development.

As NAHARP and NCGAVS have common objectives related to reporting on the state of the agri-environment and are delivered by the same division (Agri-Environmental Programs and Science Coordination Division) within the Agri-Environment Services Branch (AESB), the two programs were evaluated together.

1.1 Evaluation Scope

In accordance with the Treasury Board Directive on the Evaluation Function, the evaluation examined the programs' relevance and performance. Specifically, the evaluation examined: continued need for the programs; alignment with government priorities, departmental strategic outcomes, and federal roles and responsibilities; achievement of intended outcomes; and the extent to which the programs demonstrate efficiency and economy.

The evaluation was national in scope, covering the period from the programs' continuation under Growing Forward in 2008/09 to 2011/12. There was greater emphasis placed on assessing NAHARP, as it has significantly higher materiality than NCGAVS and also requires more extensive data collection to assess its outcomes. In addition, the evaluation placed less emphasis on assessing efficiency and economy as there is limited data on the actual costs and expenditures of the programs, as well as the time spent by AAFC scientists on specific program activities. The evaluation focused on the relevance of the programs and on their performance in terms of the achievement of outcomes. Finally, as a national program, the evaluation assessed both the perspectives of the federal government and also provincial stakeholders.

1.2 Evaluation Approach

All evaluation activities were completed by OAE. Employing a summative mixed-methods design Footnote 2, incorporating both qualitative and quantitative information, the evaluation used multiple lines of evidence to address the evaluation issues and questions. Qualitative data was used to provide context around quantitative data and also provide more descriptive examples of outcomes.

1.3 Methodology

The evaluation included several lines of evidence.Footnote 3

  • A document and literature review was completed to gain a comprehensive understanding of the programs and their forerunners, and to gather information relevant to the evaluation questions. The review examined: foundational documents, such as the Treasury Board Submission, to gain a solid understanding of the design and operation of both programs; reports produced by the programs to gain a sense of what the programs have accomplished; and finally, branch/departmental level reports to gain a better understanding of how the programs align with broader branch and department level objectives. A very brief review of the literature was also conducted to understand the broader context of agri-environmental indicators and lessons learned from other jurisdictions.
  • An analysis of program performance information was completed to assess the outputs related to the programs. Both programs report on their indicators as outlined in the performance measurement strategy (PMS). These reports were used to assess the achievement of outputs and outcomes. The programs also report on research related activities such as the number of peer-reviewed papers produced, conferences attended, etc. This information was used to gain a better understanding of the activities undertaken by the program.
  • Interviews were undertaken in 22 interview sessions with a total of 25 key informants, in order to gather perspectives on the programs from key stakeholder groups. Interviewees included: five AAFC program staff (four at National Headquarters and one at the Winnipeg Office); six AAFC staff that use NAHARP/NCGAVS data (from Agri-Environment Services Branch (AESB) and Research Branch (RB)); six staff of other federal departments (Environment Canada and Natural Resources Canada), seven provincial stakeholders and one private sector association. Interviewees were selected to include government officials familiar with program delivery and management, as well as federal, provincial and private sector stakeholders who use NAHARP/NCGAVS research.
  • A survey was undertaken of NAHARP stakeholders. The purpose of the survey was to gain a better understanding of how NAHARP research is used. The survey was sent to those who are included on the NAHARP Indicator Report distribution list and those who attended the NAHARP workshop series. These two groups were chosen to be included in the survey as they represented the most comprehensive lists of stakeholders available. The survey was sent to 373 recipients (the entire population); 157 completed the survey, which resulted in a response rate of 42%.

1.4 Methodological Considerations

There are several considerations or limitations to note when reading the evaluation. First, there was limited performance data on outcomes because the programs did not regularly collect this type of information as there was no clear policy requirement at the time of program implementation to establish performance measurement strategies for non-grants and contribution (G&C) programs.Footnote 4 This limitation was mitigated by conducting a survey to obtain data on how NAHARP research is used.

There were also difficulties in identifying potential users of both NAHARP and NCGAVS research/data as it is not always possible to know if someone has used the research. This limitation was mitigated by using stakeholder lists that were made available by the program in order to conduct the survey and interviews. However, these lists are not comprehensive and thus do not represent the entire population of NAHARP/NCGAVS stakeholders. Thus, limitations remain due to the biases associated with using stakeholder lists provided by the programs.

There were challenges in assessing the efficiency and economy of both the NCGAVS and NAHARP programs due to the way in which program activities and expenditures are tracked. While expenditures for non-pay operating including indirect, enabling, accommodation and employee benefit costs are tracked separately, salary costs are tracked at the individual research centre level and are difficult to trace back to specific initiatives due to the nature of the expense item and limited coding used. While scientists provide planned estimates of the time allocated to specific research activities over the life of the program, these estimates are at a very granular level of detail and not aggregated to link to the overall program activities. As a result, it is not possible to develop a picture of how program resources are allocated or expended over the life of the NAHARP and NCGAVS programs.

OAE attempted to mitigate this limitation by conducting a costing exercise to estimate the salary costs incurred and the activities undertaken by each program. However, response to the costing exercise was not broadly representative (40%) and therefore the results could not be used to make reliable statements about program expenditures. Thus, limitations remain in assessing program efficiency and economy in any authoritative way.

2.0 Profile Of The Programs

2.1 Background

AAFC committed to supporting the following three strategic outcomes for the current five-year agricultural policy framework, Growing Forward:

  • A competitive and innovative sector;
  • A sector that contributes to society's priorities; and,
  • A sector that is proactive in managing risk.

NCGAVS and NAHARP are part of the Growing Forward suite of initiatives designed to promote environmentally responsible agriculture. Both programs use science-based computer models to assess the impact of agriculture on the environment. They share the expected outcome of "enhancing the capacity of the agriculture and agri-food industry to encourage sound environmental decision making."

Under AAFC's Program Activity Architecture (PAA), these programs both fall under the same Sub-Activity "Agri-Environmental Sustainability Assessment" and contribute to the Program Activity "Environmental Knowledge, Technology, Information and Measurement" which, in turn, contributes to AAFC's Strategic Outcome - "An Environmentally Sustainable Agriculture, Agri-food and Agri-based Products Sector".

AAFC began to develop a set of science-based environmental indicators specific to the agriculture and agri-food sector in 1993 as part of the Agri-Environmental Indicator Project. This project was initiated in response to a need for information to assess the impacts of agricultural policies and programs on the environment both nationally and internationally. The indicators were designed to help determine how environmental conditions within agriculture were changing over time, and what factors caused those changes.

Prior to the Agriculture Policy Framework (APF), AAFC's work in this area mostly involved the development of science-based models. However, when NAHARP was officially established under APF in 2003, the program included the following objectives:

  • Establish departmental capacity to evaluate and regularly report on the general state and trends of agriculture's interactions with the environment in key priority areas; and
  • Develop analytical tools to use this information to enable and support the policy development and assessment process.

NCGAVS develops scientific models for estimating greenhouse gas (GHG) emissions stemming from Canada's agricultural sector and supports Canada's international reporting obligations under the United Nations Framework Convention on Climate Change (UNFCCC). Environment Canada (EC) is responsible for reporting annually on Canada's national GHG inventory to the UNFCCC, but allocates responsibility for GHG estimates on agricultural lands to AAFC through a memorandum of understanding (MOU).

NCGAVS was developed in accordance with the guidelines of the Intergovernmental Panel on Climate Change. From 2000-2005, the program was originally funded by EC through the Climate Change Action Plan. During that time it focused on developing an inventory system to estimate the agriculture sectors' GHG emissions and removals. Since 2006, NCGAVS has been funded by AAFC and its scope has been broadened to include: offset system supportFootnote 5, the development of scenarios to assess future GHG impacts on the agricultural sector and assessment of GHG emissions from the broader agri-food system. NCGAVS research provides information for Canadian negotiators participating in international climate change negotiations for negotiating standards associated with estimating GHG emissions. NAHARP soil organic matter and GHG emissions indicators are based on NCGAVS research and inventory analysis.

2.2 Design and Delivery - NAHARP

NAHARP develops science-based models that predict the interaction between agricultural practices and the environment. The program includes three components.

The primary activity of NAHARP is the development of agri-environmental indicators (AEIs) to measure key environmental conditions, risks, and changes resulting from agriculture and management practices used by producers. AEIs are science-based indicators used to identify trends with respect to soil, water, air, biodiversity and environmental farm management. NAHARP AEIs are grouped into six themesFootnote 6:

  • Risk to Water Quality
  • Atmospheric Emissions
  • Land Vulnerability
  • Landscape Ecology
  • Life Cycle Analysis
  • Land Use, Land Management and Climatic Data

The program is also undertaking uncertainty assessment and validation activities for each indicator. NAHARP aims to develop five new indicators during the Growing Forward period, in addition to the 20 indicators it had developed previously.

Census of Agriculture data, which is reported every five years, is then used to regularly report on the state of the agri-environment through NAHARPs primary publication, The Environmental Sustainability of Canadian Agriculture: Agri-environmental Indicator Report Series. The Indicator Report is published every five years (based on the cycle of the Agriculture Census). The last Indicator Report was published in May 2010.

The second activity under NAHARP involves working with AAFC's Strategic Policy Branch (SPB) to integrate economic models with agri-environmental indicator models to try to understand how changes to agricultural policies and programs will impact future economic and environmental outcomes of the sector. This is primarily accomplished through integrating AEI models with the Canadian Regional Agriculture Model (CRAM)Footnote 7.

The third activity under NAHARP, agri-environmental valuation, attempts to quantify the economic costs and benefits of environmental impacts from agriculture, to both farmers and society.

Previous to the 2012 Federal Budget, NAHARP was managed by the Agri-Environmental Programs and Science Coordination Division in the Agri-Environment Services Branch (AESB). However, the NAHARP program will be transferred with other AESB programs to the newly created Science and Technology Branch in July 2012. The new Branch was created to better integrate research and science while eliminating duplication in management, policy development, planning and administration.

There is one full-time Program Manager assigned to the NAHARP program. However, the program is primarily delivered through AAFC scientists in locations across Canada. Research Branch scientists develop and assess the agri-environmental indicators and staff from Strategic Policy Branch provide the economic modelling expertise. There are a total of 53 scientists that work on NAHARP, but also divide their time among a number of other AAFC programs. When the capacity to conduct work within AAFC does not exist, the program seeks outside expertise through competitive contracting.

2.3 Design and Delivery - NCGAVS

The primary function of NCGAVS is the development and maintenance of an agricultural GHG emissions inventory for Canada. The Government of Canada is required to report on its GHG emissions as per the UNFCCC (1992) and the Copenhagen Accord (2009). EC is the department responsible for reporting on behalf of Canada, but allocates the responsibility for GHG estimates on agricultural lands to AAFC through an MOU that was signed in 2004.

NCGAVS is guided by the Inter-Agency Advisory Committee on Climate Change which includes EC, Natural Resources Canada (NRCan) and AAFC. The annual GHG inventory falls under the guidance of the Interdepartmental Monitoring Accounting and Reporting System (MARS) Steering Committee and the MARS Agricultural Working Group. AAFC conducts an annual GHG inventory on agricultural lands and reports the findings to EC.

Similar to NAHARP, NCGAVS also works closely with SPB on economic and policy modelling activities involving the development of scenarios to assess future GHG impacts on the agricultural sector and assessment of GHG emissions from the broader agri-food system.

Research Branch (RB) managed the program between 2000 and 2008. AESB is thus reliant on close collaboration with RB in administering the NCGAVS program, wherein much of the expertise to develop and validate the programs' GHG inventory and associated analysis resides. Similar to NAHARP, from 2009 to the 2012 Federal Budget AESB managed NCGAVS, but the program is now being transferred over to the newly created Science and Technology branch. The program has three part-time staff – a Lead Scientist, a Program Manager and an Inventory Specialist.

Program Resources

Table 1 presents the AAFC budget allocations for the NAHARP and NCGAVS programs for the fiscal years of 2009/10 to 2012/13. All funding is Vote 1 (Operating).

Table 1: AAFC NAHARP and NCGAVS Budgeted costs for 2009/10 to 2012/13
($ millions)*
Year 2009/10 2010/11 2011/12 2012/13 Total
NAHARP 2.49 2.49 2.49 2.49 9.96
NCGAVS .83 .83 .83 .83 3.32
Total 3.32 3.32 3.32 3.32 13.28

3.0 Evaluation Findings

3.1 Relevance

The overarching need for agri-environmental reporting has not changed.

This section discusses the continued need for agri-environmental performance measurement and reporting programs given agriculture's interactions with the environment and the resulting need to track the performance of agri-environmental programs and policies.

Agriculture's Interaction with the Environment

Agricultural practices interact with the environment and therefore play a role in maintaining the health of environmental systems such as soil, air and water. Soil erosion can degrade soil quality while nutrient and pesticide runoff can pollute water supplies. Transformation of undisturbed land to crop production can diminish habitat for wildlife. This affects the health of Canadians through the air we breathe, the water we drink and the food we eat, and in terms of economic efficiency, the productive capacity of Canadian farmers and processors. Environmentally responsible agriculture is important to ensure the long-term sustainability and profitability of the agricultural sector.

The Development of AEIs

AEIs have developed as part of a long history of establishing socio-economic indicators to meet demands by public and policy decision makers for better information on economic performance and social and environmental trends. During the 1990's, most Organization for Economic Cooperation and Development (OECD) countries introduced a variety of agri-environmental policies and programs. At the same time, these countries began to develop indicators to help assess the effectiveness and efficiency of these policies and programs and to fulfill reporting requirements related to international environmental agreements.Footnote 8

The growing expectations around accountability and value for money as actualized through program review processes, auditor general reports, and evidence-based policy and program development have provided further pressure for objective evidence linking public investment to outcomes. There is also a growing market demand to objectively quantify the environmental impacts of agricultural products. The Canadian public, and international consumers of Canadian agricultural products, are increasingly demanding that the sector find the proper balance between increasing production and maintaining environmental sustainability.

The importance of developing AEIs has been recognized both nationally and internationally. The development of environmental indicators began with the publication of the Brundtland Report (1987) Our Common Future and the 1992 United Nations "Earth Summit" at Rio de Janeiro (United Nations, 1993), leading to the completion of the UN Commission on Sustainable Development Indicators (UNCSD, 2001). At the 1989 summit in Paris, G-7 countries committed to developing better environmental indicators to support decision-making. Finally, at the 1989 G-7 Economic Summit, member countries agreed on the priority of environmental indicators and tasked the OECD with development of an international set of environmental indicators.

Nationally, there have been a number of reports that have recommended the development of AEIs. The following are a few examples:

  • In 1990, the Federal-Provincial Agriculture Committee on Environmental Sustainability, which was endorsed by all Ministers, called for the development of "indicators to monitor and assess the state of natural resources and environmental quality in relation to agriculture".
  • The Science Council of Canada, in its 1992 report "Sustainable Agriculture: The Research Challenge", called for the development of physical and biological indicators against which alternative agricultural practices can be assessed.
  • In 1992, the Canadian Agricultural Services Coordinating Committee recommended that increased efforts be made within the sector to develop a set of environmental indicators that would adequately reflect the state or sustainability of the agricultural environment.
  • In its 1992 report "The Path to Sustainable Agriculture", the House of Commons Standing Committee on Agriculture recognized the importance of monitoring Canadian agriculture's progress towards sustainability.
  • In its 1993 audit of the environmental sustainability project of AAFC, the Office of the Auditor General of Canada identified the development of agri-environmental indicators as a key priority for the department.

International Reporting Requirements

NCGAVS allows Canada to fulfill its reporting requirements related to the UNFCCC (1992) and the Copenhagen Accord (2009). At the 1992 Earth Summit Canada agreed to the UNFCCC, which came into force in 1994. The 1997 Kyoto Protocol built on the UNFCCC by setting a legal requirement for industrialised countries to reduce their emissions. The Copenhagen Accord (2009) then replaced the Kyoto Accord with a new set of targets that are intended to be met by 2020. As a signatory to the above agreements, the Canadian government has committed to reporting on GHG emissions on a yearly basis.

EC is the department responsible for reporting GHG emissions under the UNFCCC. EC then allocated the responsibility for GHG estimates on agricultural lands to AAFC through an MOU that requires AAFC to develop the NCGAVS system and to provide annual reports on emissions and removal of emissions through carbon sinks. AAFC was given this responsibility as it has the necessary scientific expertise related to agriculture. EC has a similar MOU with NRCan and the Canadian Forest Service (CFS) which provides a GHG inventory for forest lands. Under the Chairmanship of EC, a Federal Steering Committee coordinates the work plans and activities of EC, AAFC, CFS and other federal departments in the area of GHG accounting.

In conclusion, Canada has ongoing international and national commitments to report on the impacts of agricultural practices on the environment. Agriculture interacts with the environment and therefore it is necessary that the agriculture sector has tools to track the results of its agri-environmental policies and programs.

NAHARP and NCGAVS align with federal government priorities for environmental reporting and AAFC strategic outcomes for an environmentally sustainable sector.

The evaluation assessed the alignment of NAHARP and NCGAVS with federal priorities and with AAFC strategic outcomes.

Since the early 1990s, as agri-environmental issues began to take on a more prominent role in agriculture policy, there has been continued federal leadership in the reporting of agri-environmental performance. The federal Green Plan (1990) identified environmental information as being key to sound decision-making and committed the federal government to developing and reporting, on a periodic basis, a comprehensive set of indicators that measure Canada's progress in achieving its environmental goals.

The Climate Change Action Fund (CCAF), established in 1998 as a $150M initiative over three years, further committed the federal government to help Canada meet its commitments under the Kyoto Protocol to measure and reduce GHG emissions. More recently, Budget 2010 provided $18.4 million over two years to Environment Canada for the Canadian Environmental Sustainability Indicators program to sustain the Government's annual reporting on environmental indicators.

AAFC has committed to monitoring and reporting on the agri-environmental performance of its programs and of the sector in general. NCGAVS and NAHARP align with AAFC's strategic outcome of "an environmentally sustainable agriculture, agri-food and agri-products sector," the objective of which is an agriculture and agri-food sector that uses environmental resources in a manner that ensures their sustainability for present and future generations.

NAHARP and NCGAVS also contribute to the environment component of Growing Forward which focuses on two priorities - water and climate change. The aim is to reduce agriculture's negative impact on water quality and increase water use efficiency. Climate change is expected to result in changes to growing conditions that may change the way agriculture interacts with the environment. AAFC's environment programs, including NAHARP and NCGAVS, are intended to promote practices that will help to increase the sector's ability to cope with these changing conditions.

The Saint Andrews Statement, an early statement that describes the essential elements that Ministers will look for in the next policy framework, Growing Forward 2, note a move toward more targeted, collaborative and result-oriented approaches to addressing environmental challenges through:

  • better integrated and targeted science efforts to meet existing and new challenges; and,
  • collaboration among governments, industry and academia to increase sustainability and profitability for the sector.

Finally, NAHARP AEIs are also used as performance measures in AAFC's Report on Plans and Priorities (RPP) and the Departmental Performance Report (DPR). Grouped into thematic areas (soil, water, etc.), there are five performance categories: low is unacceptable, and high is desired. The goal is for all the metrics to be in the high range by 2030. Currently, the soil quality, water quality and air quality indexes are in the "good" range, whereas the biodiversity quality index is in the "average" range.

In conclusion, the objectives of AAFC Performance Measurement and Reporting Programs are consistent with AAFC strategic outcomes and federal government priorities related to reporting on the health of the environment. The federal government has a history of supporting environmental reporting and AAFC has responded to this by including agri-environmental monitoring and reporting as part of its Growing Forward policy framework.

While the NAHARP and NCGAVS programs align with historic federal roles and responsibilities related to environmental reporting, there is a need to reassess AAFC's role in supporting agri-environmental performance monitoring and reporting given the emerging need for more place-based information.

With a strong history of agriculture research, AAFC has had the necessary expertise to develop the scientific models that form the basis of the NAHARP and NCGAVS programs. The development, monitoring and reporting of AEIs requires in-depth scientific expertise on a broad spectrum of biological processes in order to capture how agriculture interacts with the environment. The NAHARP program alone utilizes the expertise of over 50 RB scientists. Throughout their history, the programs have been able to draw on input from RB's many specialised experts to address the wide array of issues and technical expertise required for comprehensive reporting on AEIs. Further, Canada's commitments to UNFCCC (1992) and the Copenhagen Accord (2009) require a country-wide effort thus necessitating a federal role related to NCGAVS.

However, the evaluation found that in order for NAHARP and NCGAVS to have the greatest influence on agri-environmental outcomes there is a need for greater involvement of the provinces. Currently, there is no formal relationship between the NAHARP and NCGAVS programs and the provinces. The provinces are users of the information generated by the programs, and for NAHARP where there is some degree of consulting the provinces to ensure the accuracy of findings, the relationship is informal and on an ad-hoc basis. However, most interviewees mentioned the need for AAFC to have greater collaboration with the provinces, citing three reasons: (1) to continue to maintain the accuracy of data; (2) to access data collected by the provinces; and (3) to influence policy at the provincial level.

As AAFC's environment programs begin to support place-based approaches and therefore conduct analysis at a greater variation of geographic scales, it will be important to obtain input from provincial counterparts to support the continued accuracy of data. Under APF and Growing Forward, AAFC's agri-environmental programs have followed a universal, population-based approach customized at the provincial level. This has meant that success of these programs could largely be measured at the federal or provincial levels. However, it has been found that these approaches may not address agri-environmental issues that manifest at broader watershed or landscape scales.Footnote 9 Place-based approaches are viewed as addressing this issue.

A place-based approach is defined as a 'collaborative means to address complex socio-economic issues through interventions defined at a specific geographical scale'. Place-based approaches provide a geographic lens for identifying, examining and addressing priority agri-environmental issues at the appropriate scales. Although the watershed is one of the most common scales, others include airsheds, social communities, bioregions, etc. Working at the appropriate scale and targeting efforts at specific places are thought to assist all levels of governments in focusing work on the landscape portions that will yield the greatest environmental and economic benefits.Footnote 10

The emphasis on place-based approaches means that reporting AEIs at the national or provincial level is no longer sufficient. As described above, the geographic focus could be at the level of the watershed, bio-region, social community, etc, not just nationally or provincially. Place-based approaches require a much greater degree of flexibility in the geographic scale related to monitoring and reporting of AEIs.

It is important to work with provincial counterparts as the programs begin to conduct analysis at finer resolutionsFootnote 11. The provinces in some cases have a more detailed understanding of the impact of agriculture on the environment within their own borders. For example, one province noted that the pesticide section in the indicator report is rated poor for their province. However, from their technical staff's perspective, a lot of improvements have been made in that area so the indicator report makes it look worse than it actually is. If the province's input was considered more thoroughly or if they were contacted more often, they feel that the data for their province would be more accurate.

Another example is how the Indicator Report shows pockets on Vancouver Island as agriculture land when in fact there is no agriculture. Cowichan Valley land is not agricultural land, but it keeps showing up as such in the report. It has been pointed out by the provinces that the Indicator Report is not relevant to their needs when there are these types of errors. Greater consultations with the provinces would help to identify and correct these problems.

If both NAHARP and NCGAVS are going to move towards reporting on more detailed geographic areas, they will have to rely more heavily on provincial data. The provinces collect, or have the ability to collect data at smaller geographic units than what is currently available through NAHARP as reported through the Agricultural Census. Further, as agriculture and the environment are shared jurisdictions, provincial governments have significant influence on agri-environmental outcomes. Therefore to have the greatest influence on agri-environmental policies and programs it is necessary to have provincial involvement to ensure that information is useful for provincial policy makers. One provincial stakeholder noted that they don't use NAHARP information very often,

The reason is because it is quite high level. It is really more targeted at a national policy discussion rather than a provincial level discussion, though we certainly do refer to it. We do refer to it but we don't use it as a primary reference tool for policy decisions. If we're not using this at a provincial level, what value is this at a federal policy level?

Increased provincial involvement in NAHARP and NCGAVS is important to both ensure the continued accuracy of data and so that the programs are able to influence policy at the provincial level.

In conclusion, the increased focus on place-based approaches and the resulting need for performance data at a more detailed level of scale requires AAFC to reassess its role in agri-environmental monitoring and reporting. The fact that the provinces have significant responsibility for agriculture and the environment makes it even more imperative that the provinces have a greater role in both the NCGAVS and NAHARP programs.

Recommendation #1:

The Agri-Environment Services Branch, in consultation with the Research Branch (the new Science and Technology Branch as of July 1, 2012) should work with the provinces and territories to identify approaches for developing and reporting on agri-environmental indicators that are aligned with the needs of both federal and provincial agri-environmental policies and programs.

Management Response and Action Plan:

Agree. A Sustainable Science and Technology Advancement Initiative (SSTA) for the Growing Forward 2 policy framework to ensure the integration of all stakeholders along the science research continuum will be developed. The SSTA will include the identification of new Agri-Environmental Indicator (AEI) development and reporting models and approaches.

(Target: March 31, 2013 / Responsibility: Director General, Cross Sectorial Strategic Direction, S&T Branch)

The Branch will also implement a pilot project that explores ways to utilize Earth Observation Technology to increase the Frequency and Scalability of AEI's. This pilot also aims to incorporate key stakeholder concerns to better position NAHARP and AEI activities in the Growing Forward 2 policy framework.

(Target: March 31, 2013 / Responsibility: Director General, Cross Sectorial Strategic Direction, S&T Branch)

3.2 Performance – Effectiveness

There is limited data on the achievement of outcomes for both the NAHARP and NCGAVS programs.

AAFC's Growing Forward programs were launched in 2008-09. At that time, there was clear guidance from Treasury Board Secretariat (TBS) on the requirement to develop Performance Measurement Strategies (PMS') for all grant and contribution programs. Section 6.5.2 of the Treasury Board Policy on Transfer Payments (2008) noted that Deputy Heads were responsible for ensuring that performance measurement strategies were established at the time of program design, and that they were maintained and updated throughout a program's life cycle, to effectively support an evaluation or review of relevance and effectiveness of each transfer payment program.

The TBS guidance relating to the development of performance measures for non-grant and contribution programs was less clear. At the time Growing Forward programs were launched in 2008, TBS had not yet implemented the revised TB Policy on Evaluation. As a result, there was no clear requirement to develop performance measurement strategies for AAFC's Vote 1 (Operating) programs. Notwithstanding this lack of clear guidance, AESB developed PMS' for both the NAHARP and NCGAVS programs. While AESB should be recognized for this good practice, the evaluation did find some weaknesses with the articulation of performance measures for both the NAHARP and NCGAVS programs.

The following three weaknesses in program performance measurement were noted:

  1. Outcomes were not well defined at the immediate and intermediate levels, and there were no end-outcomes for either program.
  2. Indicators for both programs were primarily based on activities and outputs (e.g. indicator development and reporting) as opposed to measures of the achievement of program outcomes (e.g., awareness and use of agri-environmental indicators).
  3. Data for some of the performance measures was not being collected as the program did not have a system to track the increase in availability and accessibility of NAHARP data over time.

Going forward for Growing Forward 2 programs, AAFC program officials have indicated that steps will be taken to strengthen performance measures for all programs. In January 2012, the department implemented a new template for performance measurement strategies. This template includes specific guidance on how to develop robust performance measures and processes.

Furthermore, TBS has now provided guidance on the requirement to develop performance measurement strategies for non-grant and contribution programs. A revised TB Policy on Evaluation (together with its associated Directive and Standards) was implemented in April 2009. Section 6.2.1 of the Directive notes that, among other things, "program managers are responsible for developing and implementing ongoing performance measurement strategies for their programs, and ensuring that credible and reliable performance data are being collected to effectively support evaluation". Section 6.1.4 a) notes that "heads of evaluation are responsible for reviewing and providing advice on the performance measurement strategies for all new and ongoing direct program spending, including all ongoing programs of grants and contributions, to ensure that they effectively support an evaluation of relevance and performance". In short, the TB Policy on Evaluation extends the requirement for the development of performance measurement strategies beyond grant and contribution programs to include all new and ongoing direct program spending.Footnote 12

In conclusion, weak performance measures limit the ability to demonstrate the achievement of outcomes for both the NAHARP and the NCGAVS programs. Going forward, for Growing Forward 2 programs, AAFC will develop suitable performance measures for all new and direct program spending, including ongoing programs of grants and contributions.

NAHARP is achieving limited success in meeting its outputs.

NAHARP has reported on outputs to November 2011 and thus has a little less than a year and a half of reporting remaining as the program ends in April 2013.

NAHARP is on track to achieve its targets related to the development of agri-environmental indictors, but is significantly behind in achieving its outputs related to linking AEIs to economic models (see Table 2). In terms of the development of AEIs, NAHARP planned to develop five indicators throughout the Growing Forward period. However, the program reports that two indicators will likely be ready for the fourth report, but it is uncertain if the other three indicators will be finalized by that time.

Table 2: NAHARP Outputs, Performance Indicators, Targets and Results (reporting as of November 2011)
Outputs Performance Indicators Targets Results
Development of Agri-Environmental Indicators (AEIs) Proportion of indicators that can report on national results 90% 80% of indicators (20 out of 25) are able to report on national results
Linking AEI's to Economic Models Proportion of indicators linked to economic models 90% 12% of indicators (3 of 25) have been linked to economic models.

Linkages have been made with the Residual Soil Nitrogen indicator, the indicator of Risk to Water Contamination by Nitrogen and the Agricultural Greenhouse Gas Emissions indicator

In terms of linking AEIs to economic models, linkages have been made with the Residual Soil Nitrogen indicator, the Indicator of Risk to Water Contamination by Nitrogen and the Agricultural Greenhouse Gas Emissions indicator. SPB is currently in the process of assessing the feasibility of linking the Wildlife Habitat Capacity indicator and the Integrated Soil Erosion indicator.

However, it has been reported by the program that changes to the CRAM model have resulted in difficulties making linkages with other indicator models.Footnote 13 There will therefore be significant delays in linking CRAM with other indicator models and the ability to conduct analysis to support policy development.

In addition, outputs related to agri-environmental valuation that were originally intended to be completed (as outlined in the approved program description) were removed from the PMS in 2008 as they are no longer included in active work due to a lack of program resources.

NAHARP is on track to achieve its expected immediate outcome related to reporting on agri-environmental indicators. However reliance on the Agriculture Census, which reports every five years, results in untimely and therefore less useful information.

NAHARP is currently on track to achieve its intended immediate outcome (see Table 3).

Table 3: NAHARP Immediate Outcome, Performance Indicator, Target and Results (reporting as of November 2011)
Immediate Outcome Performance Indicator Targets Results
Agri-environmental indicators (AEIs) are available to assess and report on the sector's environmental and economic sustainability Regular reporting on the environmental sustainability of Canadian agriculture. Full reporting (every five years) Environmental Sustainability of Canadian Agriculture: Agri-Environmental Indicator Report Series – Report #3 was released in print September 2010.

The Environmental Sustainability of Canadian Agriculture: Agri-Environmental Indicator Report Series is the main deliverable of the NAHARP program. The third report was released in May 2010 and published on-line in September 2010. The fourth report is expected to be published by December 2013. The fourth report will be published after GF due to delays in receiving Census of Agriculture data from Statistics Canada.

Perceived Quality of the Indicator Report

A survey was undertaken of recipients of the Indicator Report. Overall, responses to the survey questions about the quality of the report were highly favourable. Of those who rated its overall quality, a strong majority (91%) either agreed or strongly agreed that it was "good" (see Chart 1). A similar proportion of respondents agreed that the report was easy to read (91%) and that the indicators are relevant to governments and other stakeholders in the agricultural sector (92%). A large majority (83%) also indicated that the report was "timely, presenting data that was sufficiently recent", while a weaker majority (66%) agreed that the suite of indicators presented in the report "captures all significant dimensions of the agri-environment." As discussed further on in this report, this may be due to the desire to have information on more detailed geographic regions.

Chart 1: Perceived Quality of the Indicator Report
Description of this image follows

Source: AAFC, 2012

Long Description

The agri-environmental indicators covered in the report are relevant to government and other stakeholders in the agriculture and agri-food sector (n=92), 92% strongly agree/agree, 8% strongly disagree/disagree.

The overall quality of the report is good (n=92), 91% strongly agree/agree, 9% strongly disagree/disagree.

Overall, the report was easy to read (n=93), 91% strongly agree/agree, 9% strongly disagree/disagree.

The meaning of the agri-environmental indicator values presented in the report were easy for me to understand (n=91), 90% strongly agree/agree, 10% strongly disagree/disagree.

The report was timely, presenting data that was sufficiently recent (n=86), 83% strongly agree/agree, 17% strongly disagree/disagree.

The suite of agri-environmental indicators presented in the report captures all significant dimensions of the agri-environment (n=82), 66% strongly agree/agree, 34% strongly disagree/disagree.

The meaning of the agri-environmental indicator values presented in the report could be easily understood by a non-scientific audience (n=76), 55% strongly agree/agree, 45% strongly disagree/disagree.


Reliance on the Agriculture Census

NAHARP's reliance on the Agriculture Census, which reports every five years, limits data utility to decision-making. Most of the data used for the NAHARP models come from the Agriculture Census. This means that, although NAHARP is a very good reference tool for understanding long-term trends, which generally take 15-20 years to notice, it gets out of date very quickly in terms of policy needs. As the policy development process requires timely, regularly updated information, there is a misalignment between policy needs and NAHARP reporting.

Provincial governments also require a higher frequency of reporting as they generally work on annual budgeting and performance measures. The program is currently working on options to supplement the Agriculture Census that would allow reporting on a more frequent basis, such as using earth observation (satellite imagery). The program also intends to increasingly rely on the provinces for data as it begins to look at more detailed geographic regions.

It is not possible to assess progress of the NAHARP against its intermediate outcome as the program does not track the availability and accessibility of data to decision makers over time.

Although the NAHARP program has a performance indicator and target for its intermediate outcome, it does not have a system to track the results (see Table 4).

Table 4: NAHARP Intermediate Outcome, Performance Indicator, Target and Results
Intermediate Outcome (reporting as of November 2011) Performance Indicator Targets Results
Enhanced capacity by the agriculture and agri-food industry to encourage sound environmental decision making both within government and within industry Level of availability and accessibility of results for agricultural decision makers Increase in availability over time The program does not have a system to track the increase in availability and accessibility of results over time

NAHARP indicators are made available through a variety of methods including a web-mapping application, a mailing list that includes over 400 stakeholders, presentations to the provinces and NAHARP science and results appear in journals and presentations.

NAHARP adopted a passive communication strategy. According to the interviews and survey, communication about the NAHARP program and NAHARP products is largely through word-of-mouth between colleagues or through conferences, publications, etc. However, the survey also found that the majority (84%) of respondents are from a government setting, with other groups only marginally represented (see Figure 2). Most (63%) Canadian respondents work for the federal government, while 21% work for a provincial government. There are few representatives of NGOs (8%) and academia (3%), and only one respondent represents the private sector (1%) (see Chart 2).

Chart 2: Distribution of Survey Respondents (n=118) by Type of Employer.
Description of this image follows

Source: AAFC, 2012

Long Description

Survey Distribution (by type of employer)
62% Federal Government
21% Provincial Government
8% NGO
5% Other
3% Academia
1% Private Sector


A few stakeholders suggested a more direct, proactive approach specifically targeting the agri-environmental policy community in the provinces, the private sector and even individual producers. One such positive example was the cross-Canada workshop series for provincial stakeholders that was conducted by the NAHARP program in the summer of 2011. According to the survey results, provincial participants found this very useful. Of the participants that responded to the survey, 72% reported that the workshop had increased their knowledge either "very much" or "somewhat;" 26% said "a little;" and only 4% indicated that they had learned nothing new at the workshop. As well, three-quarters (76%) reported that their questions at the workshop had been answered satisfactorily.

Availability of AEIs for Federal Policy Analysis

One output of the NAHARP program that is very important for the availability and accessibility of results for decision makers is that AEIs are centrally programmed and accessible for policy modeling. This ensures that NAHARP results are available in a centralized database that is accessible and useable for policy analysis.

It is expected that only two or three indicators will be located in a central database and accessible for policy modeling by the end of Growing Forward (March 2013), severely limiting the availability of results for decision makers. According to program management, this output has been delayed due to resource issues within the Agri-Geomatics program. Work is currently being done to import two indicators in a central place that is accessible by a number of users and the program is working with Agri-Geomatics to develop a work plan to import other indicators.

In conclusion, NAHARP is achieving limited success in the achievement of outputs and outcomes.

NAHARP is primarily used as a reference tool for priority setting.

The following section outlines how NAHARP informs decision-makers and policy-makers about conditions and trends as they relate to key issues, and their ecological, economic and health-related significance.

AEIs as a Reference Tool for Priority Setting

The survey and interviews found that NAHARP indicators are primarily being used as a reference tool for priority setting, mostly at the federal level, but also, on more limited basis at a provincial level.

At the federal management level it was noted:

Think of it as a reference book. Policy development is very fluid, very organic - it's not a mechanical process. NAHARP needs to exist to formulate thinking based on something objective and solid. NAHARP indicators are a great way to simplify communication about policy. Having that reference information is important for situating what we do in the right context. It's like a dictionary, you're always going back to check it and eventually you get to know it.

At the provincial management level it was noted:

Officials use the reports as an outside source of information to validate conclusions. And they also use it to understand what are the challenges and priorities within agriculture in Canada. They use it for policy and program planning work. So people use the report - it's a great reference; it's a good picture of broader themes.

There are a number of specific examples of how both federal and provincial policy and program staff use the Indicator Report. For example, they use the Indicator Report as a basis for making recommendations on future programming options. It was noted that this report was the foundation for the measuring and monitoring and documenting on whether governments are making environmental improvements.

At the federal level, AAFC officials worked with the provinces and non-governmental agencies to do an environmental scan for the purposes of agricultural policy development. NAHARP research was also used to support federal-provincial discussions around priorities for 2010/11. Finally, NAHARP research was used for deciding branch priorities/issues in Saskatchewan and where AAFC resources are needed. Initially planning was around "agricultural intensity", but trends from NAHARP were incorporated to indicate where things are getting incrementally worse/better. NAHARP allowed them to allocate AAFC resources according to environmental trends.

In terms of the survey results, 61 out of 157 (39%) people surveyed reported having used NAHARP research. At a more defined level, 34 (9%) used NAHARP research for conducting other research, 38 (10%) used NAHARP for policy discussions or decision making and 31 (8%) used NAHARP for an activity other than research or policy discussion.

However, an analysis of the details of individual responses suggests that NAHARP is largely being used as a reference/ communication tool or for priority setting/resource planning. Specific comments include that it is being used for research (13 of 59 responses), as a reference tool (10 of 59 responses), to facilitate communications with and awareness building among farmers (10 of 59 responses), to inform policy discussions (9 of 59 responses) and to inform priority setting and resource planning (6 of 59 responses) (see Chart 3).

Chart 3: How NAHARP Research is Used by Survey Respondents.
Description of this image follows

Source: AAFC, 2012

Long Description

Question Q10: Did you use NAHARP research to inform: (A) The conduct of research you were involved in?; (B) Policy discussions or decisions?; (C) An Activity other than research or policy discussions? (n=58).

Research – 13 respondents
Reference tool/background information/ general discussion – 10 respondents
Communication/ awareness building/knowledge transfer/ facilitating discussions/ communicating to producers and producer groups – 10 respondents
Policy – 9 respondents
Priority setting/ resource planning – 6 respondents
Evaluating/implementing BMPs – 3 respondents
Miscellaneous – 8 respondents


The NAHARP indicator report is by far the most often used or referenced NAHARP research product, with 93% of respondents staying that they have used the report to support their work. Respondents also used NAHARP related peer-reviewed journal articles (32%) and conference or workshop proceedings (18%) (see Chart 4).

Chart 4: NAHARP Research Products that Survey Respondents Used or Referenced to Inform their Work.
Description of this image follows

Source: AAFC, 2012

Long Description

Percentage of Responses (n=68)
NAHARP Indicator Report – 93%
Conference or workshop proceedings presenting NAHARP research - 18%
Reviewed journals – 32%
Other – 13%

Finally, the vast majority (92%) of respondents agreed that "the agri-environment indicators covered in the report are relevant to governments and other stakeholders in the agriculture and agri-food sector."


In conclusion, NAHARP informs decision-makers and policy-makers about conditions and trends as they relate to key agri-environmental issues. Both the interviews and survey concluded that NAHARP products are primarily used as a reference tool for planning or communication purposes.

AEIs are not at a point where they are an effective tool for policy monitoring and evaluation because they do not provide analysis at a sufficient level of detail.

AAFC's AEIs are currently "coarse", providing analysis at the national level, and thus are not able to make linkages between policies and agri-environmental outcomes at the local /regional level. For example, the 2007 Chapter Evaluation of the APF Environment Priority found that "while the main goals of the program can be viewed as micro (farm level) and macro (regional or even country-wide), the performance measurement is national in scope. No process exists for aggregating (rolling up) changes in environmental risk at the farm level to the regional and national level or disaggregating (splitting out) regional/national changes to the farm level."

This is consistent with how the OECD (2008) views the purpose of AEIs: the use of AEIs as a reference tool provides a foundation for their use as a tool in policy monitoring and evaluation and is not an end-point in and of itself. According to the OECD (2008), "Most of the initial effort involving AEIs has been to identify appropriate indicators, and establish data sets to track the state and trends of environmental conditions in agriculture."[…] "This body of work has provided the foundation to begin using AEIs as a tool in policy monitoring and evaluation to better understand the linkages between policies and agri-environmental outcomes (OECD, 2008; p.555)"

For example, the 2007 Chapter Evaluation of the APF Environment Priority was one of the main reports evaluating the performance of AAFCs environmental programs. However, NAHARP data was not used. The reported states, "data [NAHARP data] for 2006, which could provide mid-term measures of program impacts, will not be available in 2008-2009 and post-APF data for 2011 will not be available until 2013. Thus, this evaluation could not use NAHARP information to comment on the impact of the Environment Chapter".

AESB management was clear that in order to be effective AEIs must be able to inform decision making, rather than being solely used as a reference tool. Management's view is that,

It was okay in the past to look at the regional and national level, but now we need to look at more detailed geographic areas. Right now the data is coarse. To make the data more relevant we need to go down to a more detailed geographic area. It is an issue of scaleability. The monitoring objective is still important: what are the long term trends and what needs to be done.

If it stays static - inward-focused, Census-focused, for policy – it will not survive. It's too expensive. We need to become more collaborative with the provinces to build it ground-up and make it applicable at the working level.

A number of provincial stakeholders also stated that at a national level, the report does not meet their needs. Overall, they did not feel that NAHARP was applicable to provincial policy needs because it is a national report and does not include enough detail at the regional, provincial or watershed level.

The survey results came to similar conclusions. When asked if there is information on Canadian agri-environmental indicators that is not currently available that would be useful, the most important was more detail at the Provincial, Regional and Watershed level/Finer scale (10 of 45 respondents) (see Chart 5). Some examples of Misc. category include: pesticide use and its impacts, areas of land use change and agro-ecology.

Chart 5: Information on Canadian Agri-environmental Indicators that is not Currently Available that Would be Useful
Description of this image follows

Source: AAFC, 2012

Long Description

Question: Is there any information on Canadian agri-environmental indicators that is not currently available that would be useful to your organization (n=45)

More detail at the Provincial, Regional and Watershed Level/ Finer Scale – 10 respondents
Soil Health/Phosphorus – 7 respondents
Climate change/air quality – 5 respondents
Methodologies, assumptions, access to data sets – 5 respondents
Biodiversity – 4 respondents
Water issues – 3 respondents
More timely information – 2 respondents
Livestock issues – 2 respondents
Miscellaneous – 14 respondents


The survey also found that that only 6% of respondents report their primary profession as policy analysis, which is surprising considering that one of NAHARP's main objectives is to inform policy (see Chart 6).

Chart 6: Distribution of Survey Respondents (n=129) by Profession.
Description of this image follows

Source: AAFC, 2012

Long Description

Distribution of Survey Respondents (by profession)

48% Scientific Research and/or Training
32% Program Delivery/Management
7% Senior Management
7% Other
6% Policy Analysis.


There are some examples of NAHARP research being used to support policy work at AAFC mostly involving the use of modeling. These include:

  • environmental assessments for business risk management programs and trade negotiations;
  • supporting development of a bioeconomy strategy for AAFC and GOC (2009 and ongoing);
  • and providing analysis to inform GFII.

There is also one example of the private sector using NAHARP research. Pulse Canada used NAHARP research for marketing to show the positive environmental practices of their growers. They conducted a project that looked at 4 indicators (GHG, soil, energy, land use) that were developed in the US and they were able to adapt it for Canada using NAHARP data. They were able to demonstrate, through using NAHARP research, the environmental benefits of their members' products.

In conclusion, in order to move beyond reporting on overall conditions and trends towards facilitating analysis of issues in support of policy and program evaluation and development, AEIs need to provide analysis on a more detailed scale. This is particularly important for provincial governments as national reporting is not adequate for informing provincial policy processes.

NCGAVS is making progress towards achieving its outputs and immediate and intermediate outcomes.

NCGAVS Outputs

NCGAVS is required to report annually to EC on the agriculture sector's impact on GHG emissions. As of November 2011, AAFC has provided three annual report inventories and three annual quality control and quality assurance reports to EC as per requirements outlined in the MOU (see Table 5).

Table 5: NCGAVS Outputs, Performance Indicators, Targets and Results (reporting as of November 2011)
Outputs Performance Indicator Target Result
Development of annual report on carbon and greenhouse gas emissions and removals from Canada's agricultural land Annual inventory and report of greenhouse gas emissions and removal for agriculture including uncertainties that incorporates required improvements to data and methodologies 5 3 inventories delivered to Environment Canada
(Nov 16, 2009; Nov 15, 2010; Nov 15, 2011)
Development of improvements to greenhouse gas inventory methods and data Annual quality control and quality assurance reports documenting how improvement requirements are being met 5 3 quality control documentation reports for annual inventory delivered to Environment Canada

(Jan 8, 2010; Nov 19, 2010; Nov 15, 2011)
Technical analytical support for international climate change negotiations Participating in international climate change negotiations, as called by Conference of Parties 10 15 occasions where expert advice was provided at UNFCCC negotiations

NCGAVS and Environment Canada

Originally NCGAVS was funded by EC, thus ensuring that the program's mandate was entirely focused on international reporting requirements. However, now that NCGAVS is being funded by AAFC, the program has to respond to AAFC internal priorities in addition to international reporting requirements led by EC. For example, the NCGAVS program is responsible for the development of scenarios to assess future GHG impacts on the agricultural sector and assessment of GHG emissions from the broader agri-food system. AAFC's current focus on placed-based strategies will only increase the pressure on the NCGAVS program to produce more localized data to support this type of programming.

This increased internal pressure on the NCGAVS program has led to less synergy between AAFC and EC (as related to NCGAVS) and also an awareness of the potential impact this could have on AAFC's ability to accurately report in a timely manner to Environment Canada. The issue is essentially one of resources: NCGAVS' internal work is taking resources away from its international reporting requirements and EC has expressed concern that this will affect the quality of the inventory reporting.

In response, in December 2011, AAFC, EC and CFS held a meeting to reassess their working priorities for the "post Kyoto" period, through to 2020. According to interviewees, this meeting helped to re-define the relationship between the three departments to ensure that international reporting requirements will continue to be met in a timely and accurate manner. However, it was recognized by several interviewees that this will continue to be an issue and that it would be best addressed by a re-commitment of senior management to NCGAVS' reporting requirements to EC.

NCGAVS Immediate Outcome

NCGAVS is on track to achieve expected immediate outcomes (see Table 6).

Table 6: NCGAVS Immediate Outcomes, Performance Indicators, Targets and Results
Immediate Outcomes
(reporting as of November 2011)
Performance Indicator Target Result
The sector is provided annually with a transparent and internationally accepted estimate of greenhouse gas emissions and removals from agriculture, with uncertainty and carbon intensity estimates Number of requirements met for methodological and data improvements resulting from regular formal international and inter-departmental review 20 14 methodological and data improvements (e.g. improved estimation of the distribution of land use types at the local level)
Improved capacity building regarding Government of Canada negotiators on international climate change negotiations Number of reports on greenhouse gas emissions for agriculture and agri-food sector including updated greenhouse gas intensities for agricultural products 5 5 scientific publications detailing the greenhouse gas emissions intensity from Canadian agricultural activities
Number of consultations and analysis provided in support of climate change negotiators 10 4 responses to questions on various potential changes to accounting rules related to the UNFCCC
Performance in Environment Canada inventory reviews and UNFCCC in-country  and international reviews 6 3 Environment Canada reviews of LULUCF inventory (May 2009; Feb 2010; Feb 2011)

Overall, NCGAVS quality reviews have been positive. For example, an international team of experts that were commissioned by the program to review the CanAGMARS system for quality assurance "commended the Canadian government for the development of an advanced system for conducting inventories of carbon stock changes in agricultural lands". However, the review team "found some limitations in the current system that were mostly related to the activity data, and may create significant bias in the current C stock change estimates.Footnote 14 […] The review team also recognizes that improvements are not always necessary and moreover require sufficient resources". The program is currently working on the issues identified by the review.

NCGAVS Intermediate Outcome

Although the NCGAVS program has a performance indicator and target for its intermediate outcome, it does not have a system to track the results (see Table 7).

Table 7: NCGAVS Intermediate Outcomes, Performance Indicators, Targets and Results
Intermediate Outcome
(reporting as of November 2011)
Performance Indicator Target Result
Enhanced capacity by the agriculture and agri-food industry to encourage sound environmental decision making both within government and within industry Level of availability and accessibility of results for agricultural decision makers Increase in availability over time The program does not have a system to track the increase in availability and accessibility of results over time

NCGAVS is achieving its primary objectives related to international reporting requirements, but there are also examples of NCGAVS research being used to support decision-making at AAFC, at other federal departments, and with the private sector.

AAFC's CRAM model uses NCGAVS data to estimate 20-year projections involving various scenarios to determine the impact on GHG emissions. For example, NCGAVS has worked with SPB in supporting the development of a bioeconomy strategy for AAFC and the Regulatory Impact Assessment Statement (RIAS) on biofuels for Environment Canada's 2010/11 biofuel mandate regulations. There is also an example of NCGAVS data being used in the private sector. McCain's measures and shares publicly their GHG footprint and uses NCGAVS research in order to conduct lifecycle analysis.

In addition, Canada was accepted as co-lead of the inventory and measurements cross-cutting theme within the 33-country Global Research Alliance on Agricultural Greenhouse Gases, which is a direct result of the NCGAVS program. Canada will chair the Alliance Council during FY 2012-13.

In conclusion, NCGAVS is making progress towards the achievement of outcomes and meeting requirements as set out in the MOU with EC. However, pressures on the NCGAVS program related to internal analysis and reporting requirements has led the program to re-assess its working priorities with EC.

Recommendation #2:

The Agri-Environment Services Branch, in consultation with the Research Branch (the new Science and Technology Branch as of July 1, 2012) should review AAFC's role related to reporting on Green House Gas (GHG) emissions for the agriculture sector to ensure alignment with AAFC and the Government of Canada's mandate and priorities.

Management Response and Action Plan:

Agree. The Branch will create an MOU between AAFC and Environment Canada to ensure the role of monitoring related to GHG emissions reporting for the agriculture and agri-food sector is aligned with the Departmental mandate and priorities.

(Target: March 31, 2013 / Responsibility: Director General, Cross Sectorial Strategic Direction, S&T Branch)

The Branch will also develop the Sustainable Science and Technology Advancement Initiative (SSTA) for the Growing Forward 2 policy framework to ensure that AAFC-S&T Branch continues to meet formal international obligations for monitoring and reporting GHG emissions for the agriculture and agri-food sector.

(Target: March 31, 2013 / Responsibility: Director General, Cross Sectorial Strategic Direction, S&T Branch)

3.3 Performance - Efficiency And Economy

Assessing the efficiency and economy of the NCGAVS and NAHARP programs is difficult in the absence of a system to track how much time AAFC scientists spend on activities associated with different programs.

Both NCGAVS and NAHARP are funded through Vote 1 (Operating) funds. The NCGAVS program was allocated $3.32 million and NAHARP $9.96 million over four years. Funds are allocated to cover a variety of costs including: Non-Pay Operating (NPO), indirect, enabling, EBP, accommodation and salary costs (please refer to Table 8a-8d for a detailed breakdown of program costs).

In terms of the allocation of funding for the NCGAVS program, the majority of funds ($2.89 million or 87%) were allocated to support direct program costs. The remainder of funds ($430,110 or 13%) were allocated to cover the costs associated with program administration and enabling costs (e.g., audit, evaluation, human resources, finance, communications)

For the NAHARP program, the majority of funds ($8.9 million or 90%) were allocated to support direct program costs. The remainder of funds ($1.05 million or 10%) were allocated to cover the costs associated with program administration and enabling costs.

Table 8a: Resources by program from 2009/10 to 2012/13*
Total Full-Time Equivalent (FTE) NAHARP NCGAVS
* These amounts are notional distributions of the forecast program delivery requirements to 2012-13. Actual program expenditures were not available at the time of the evaluation.
Total 13 4
Table 8b: Resources by program from 2009/10 to 2012/13*
Direct Program Costs NAHARP NCGAVS
* These amounts are notional distributions of the forecast program delivery requirements to 2012-13. Actual program expenditures were not available at the time of the evaluation.
Full-Time Equivalent (FTE) 11 4
Salary 3,100,000 1,000,000
EBP 620,000 200,000
Accommodation 403,000 130,000
NPO 4,780,000 1,560,000
Subtotal 8,903,000 2,890,000
Table 8c: Resources by program from 2009/10 to 2012/13*
Admin and Delivery NAHARP NCGAVS
* These amounts are notional distributions of the forecast program delivery requirements to 2012-13. Actual program expenditures were not available at the time of the evaluation.
Full-Time Equivalent (FTE) 1 0
Salary 200,000 100,000
EBP 40,000 20,000
Accommodation 26,000 13,000
NPO 200,000 100,000
Subtotal 466,000 233,000
Table 8d: Resources by program from 2009/10 to 2012/13*
Enablers NAHARP NCGAVS
* These amounts are notional distributions of the forecast program delivery requirements to 2012-13. Actual program expenditures were not available at the time of the evaluation.
Full-Time Equivalent (FTE) 1 0
Salary 293,271 97,757
EBP 58,653 19,551
Accommodation 38,124 12,708
NPO 201,282 67,094
Subtotal 591,330 197,110
Total 9,960,330 3,320,110

It is difficult to assess the efficiency and economy of both the NCGAVS and NAHARP programs. Both NCGAVS and NAHARP are funded through Vote 1 (Operating) funds and AESB and RB do not have a system in place for tracking how AAFC scientists divide their time between the various programs they support. As a result, it is not possible to determine how much time scientists have worked on either the NAHARP or NCGAVS programs. AESB officials note that there are three challenges in trying to determine the allocation of time spent by AAFC scientists on specific programs:

First, there is a "common elements" problem. Some portion of a researcher's time will be spent undertaking activities that are associated with keeping a lab up and running. This could include developing or refining lab methods, ordering supplies, or building specialized equipment. These activities support multiple programs.

Second, there is a "project overlap" problem. Research scientists often receive funding from a variety of program sources, as well as from external funding sources. The research supported through this collective funding (e.g., collection and analysis of lab samples) often supports multiple program objectives. As a result, it is difficult to allocate funding to specific program-related activities.

Third, there is a "validation" problem. A scientist often spends more than a typical work day (e.g., 8 hours) undertaking project-related work. However, there is no formal way to capture this time without being required to pay corresponding overtime (in the case of unionized employees). In addition, given the aforementioned problems, from a scientists' perspective any attempt to allocate activities to specific programs would be at best an estimate that could not be validated through any formal time-tracking process.

As part of this evaluation, a costing exercise was undertaken to try to estimate staff time spent on the NAHARP and NCGAVS programs. A total of 62 individuals were surveyedFootnote 15, however due to poor participation (40%), the results were determined not to be statistically valid. Through the costing exercise it became evident that, for a variety of reasons, the scientists working on NAHARP and NCGAVS feel that they are not able to report how much time they have spent on each program. For example, one scientist noted,

It is very difficult to collect meaningful estimates of the hours that have been spent on specific science programs, since much of the work is "related to" but not strictly "entirely billable to" a single program, and much of the work is toward goals of more than one program at once.

Another scientist noted:

All my research is so related to these programs [NAHARP and NCGAVS] that it is practically impossible to separate the allocation.

In terms of program economy, does the investment of $9.96 million in the NAHARP program to develop three indicator models and to support regular public reporting on 20 agri-environmental indicators represent value for money? Answering the question of program economy for science-based programs is challenging given that immediate and intermediate outcomes are typically associated with knowledge creation and transfer, and the achievement of end outcomes can take seven to ten years or longer to be realized, depending upon the objectives of the program. AESB officials noted that finding the appropriate measures to assess the performance of research programs is a common challenge across all science-based departments.

In conclusion, notwithstanding the challenges associated with measuring the performance of science-based programs, some attempt should be made to track how scientists' time is allocated to various activities, in order to support future assessments of program efficiency and economy.

Recommendation #3:

The Agri-Environment Services Branch, in consultation with the Research Branch (the new Science and Technology Branch as of July 1, 2012) should develop a system to track, within a reasonable approximation, the relative effort that scientists invest in activities associated with the different programs from which they receive funding.

Management Response and Action Plan:

Agree. The Branch will collaborate with Corporate Management Branch and the Programs Branch to study and present options on how to capture the efforts of AAFC scientists related to program activities.

(Target: March 31, 2013 / Responsibility: Director General, Cross Sectorial Strategic Direction, S&T Branch)

4.0 Conclusions and Recommendations

4.1 Conclusions

Canada has ongoing international and national commitments to report on the impacts of agricultural practices on the environment. Agriculture interacts with the environment and therefore it is necessary that the agriculture sector has tools to track the results of its agri-environmental policies and programs.

The objectives of AAFC Performance Measurement and Reporting Programs are consistent with AAFC strategic outcomes and federal government priorities related to reporting on the health of the environment. The federal government has a history of supporting environmental reporting and AAFC has responded to this by including agri-environmental monitoring and reporting as part of its Growing Forward policy framework.

The increased focus on place-based approaches and the resulting need for performance data at a more detailed level of scale requires AAFC to reassess its role in agri-environmental monitoring and reporting. The fact that the provinces have significant responsibility for agriculture and the environment makes it even more imperative that the provinces have a greater role in both the and NAHARP programs.

Weak performance measures limit the ability to demonstrate the achievement of outcomes for both the NAHARP and the NCGAVS programs. Going forward, for Growing Forward 2 programs, AAFC will develop suitable performance measures for all new and direct program spending, including ongoing programs of grants and contributions.

NAHARP informs decision-makers and policy-makers about conditions and trends as they relate to key agri-environmental issues. Both the interviews and survey concluded that NAHARP products are primarily used as a reference tool for planning or communication purposes.

In order to move beyond reporting on overall conditions and trends towards facilitating analysis of issues in support of policy and program evaluation and development, AEIs need to provide analysis on a more detailed scale. This is particularly important for provincial governments as national reporting is not adequate for informing provincial policy processes.

NCGAVS is making progress towards the achievement of outcomes and meeting requirements as set out in the MOU with EC. However, pressures on the NCGAVS program related to internal analysis and reporting requirements has led the program to re-assess its working priorities with EC.

Assessing the efficiency and economy of the NCGAVS and NAHARP programs is difficult in the absence of a system to track how much time AAFC scientists spend on activities associated with different programs. Some attempt should be made to track how scientists' time is allocated to various activities, in order to support future assessments of program efficiency and economy.

4.2 Recommendations

The evaluation identifies the following three recommendations:

  1. The Agri-Environment Services Branch, in consultation with the Research Branch (the new Science and Technology Branch as of July 1, 2012) should work with the provinces and territories to identify approaches for developing and reporting on agri-environmental indicators that are aligned with the needs of both federal and provincial agri-environmental policies and programs.
  2. The Agri-Environment Services Branch, in consultation with the Research Branch (the new Science and Technology Branch as of July 1, 2012) should review AAFC's role related to reporting on Green House Gas (GHG) emissions for the agriculture sector to ensure alignment with AAFC and the Government of Canada's mandate and priorities.
  3. The Agri-Environment Services Branch, in consultation with the Research Branch (the new Science and Technology Branch as of July 1, 2012) should develop a system to track, within a reasonable approximation, the relative effort that scientists invest in activities associated with the different programs from which they receive funding.

Annex A: List of NAHARP Indicators

The indicators that report nationally are:

  1. Greenhouse gas emissions
  2. Particulate Matter emissions
  3. Ammonia emissions from agriculture
  4. Indicator of Risk of Water Quality by Nitrogen (IROWC-N)
    1. a. This indicator is supported by another, Residual Soil Nitrogen
  5. Indicator of Risk of Water Quality by Phosphorus (IROWC-P)
  6. Indicator of Risk of Water Quality by Pesticides (IROWC-Pest)
  7. Indicator of Risk of Water Quality by Pathogens (IROWC-Coliform)
  8. Soil Cover
  9. Soil organic matter
  10. Contamination by Trace Elements
  11. Integrated soil erosion, supported by three sub-indicators
    1. Risk of water erosion
    2. Risk of tillage erosion
    3. Risk of wind erosion (prairies only)
  12. Wildlife Habitat Capacity on Farmland
  13. Energy Use and Greenhouse Gas emissions in the Food and Beverage Industry
  14. Water Use in the Food and Beverage Industry
  15. Packaging use in the Food and Beverage Industry
  16. Soil Salinity Indicator (One indicator reports for the prairies only since it is not a problem in other areas)

Under Growing Forward, five indicators are under development:

  1. Odour emissions from agriculture (this should be ready for the next report)
  2. Risk of land degradation (this should be ready for next report)
  3. Integrated water quality indicator (will not be ready by next report)
  4. Riparian Health (may be ready for next report, but probably not nationally)
  5. AgroForestry (may be ready for next report, but probably not nationally)

Annex B: Works Cited

Boag, Gemma (2011), "Place-Based Approaches for Agri-Environmental Policy in Canada," prepared by AESB Policy Research Division.

OECD, 2008, "Using Agri-environmental Indicators for Policy Analysis," OECD Publishing, retrieved from http://www.iisd.org/pdf/2006/measure_use_indicators.pdf, April 2012.

Report a problem on this page
Please select all that apply:
Date modified: