Language selection

Search

Evaluation of the AgriInnovation Program – Stream B: Research, Development and Knowledge Transfer

May 1, 2017

List of abbreviations

AAFC
Agriculture and Agri-Food Canada
ADM
Assistant Deputy Minister
AIP
AgriInnovation Program
R&D
Research and Development

Executive summary

Purpose of the Evaluation

This Evaluation examined the relevance and performance of Stream B: Research, Development and Knowledge Transfer of the AgriInnovation Program (AIP). The evaluation covers April 1, 2013 to March 31, 2016, and was conducted as part of Agriculture and Agri-Food Canada's (AAFC) 2014-15 to 2018-19 Departmental Evaluation Plan, in accordance with the Treasury Board of Canada's Policy on Results.

Background

The AIP is a five-year Growing Forward 2Footnote 1 program designed to accelerate the pace of innovation by supporting research and development (R&D) activities in agri-innovation and facilitating the demonstration, commercialization and adoption of innovative products, technologies, processes, practices and services. The program funds activities under three streams: Stream A: Research Accelerating Innovation (usually ten years or more from commercialization); Stream B: Research, Development, and Knowledge Transfer (usually two to ten years from commercialization); and Stream C: Enabling Commercialization and Adoption (usually two years or less from commercialization). Similar programming was implemented from 2008-09 to 2012-13 under the first Growing Forward framework.

Stream B includes two components:

As of March 31, 2016, $268.1 million in AAFC funding was budgeted for Stream B activities from 2013-14 to 2017-18.

Methodology

The evaluation used the following quantitative and qualitative lines of evidence:

Key findings and conclusions

Relevance

There is a continued need for Stream B programming to develop innovations to address threats to the profitability and competitiveness of the agriculture and agri-food sector. The sector faces numerous threats such as variable input and output prices, extreme weather, and new pests and diseases. There is also a need to capitalize on new opportunities for value-added products due to increasing global demand for food with specific attributes and non-food products. Investments in agricultural R&D contribute to improved productivity and generate a substantial return on investment in the long-term.

Limited funding, weak linkages between agricultural R&D performers, and capacity challenges constrain the sector's ability to meet its innovation needs. The federal government continues to play a lead role in funding and maintaining the capacity to undertake agricultural R&D and knowledge transfer to complement and leverage limited investment, foster innovation in small and emerging agricultural industries and encourage industry to invest in research with long-term public good benefits (for example, soil, water, and air quality; biodiversity). There is an ongoing need for collaboration among federal and provincial governments, universities, industry and other organizations in conducting agricultural R&D in Canada. R&D collaboration has many benefits such as leveraging investment, enhancing access to highly qualified personnel, infrastructure and equipment, and generating research excellence. AAFC collaboration rates are increasing at a higher rate than other federal government departments involved in scientific research. This increase coincides with the introduction of the cluster funding model under Growing Forward and its continued support under Growing Forward 2 through Stream B, which was designed to enhance collaboration.

Program-supported R&D activities can also provide opportunities for sector highly qualified personnel to gain research experience, which in turn supports the sector's future scientific and innovation capacity. This addresses several challenges that may constrain the ability to meet future highly qualified personnel needs, such as the high number of expected scientist retirements across the sector, limited networks and mentorship opportunities for newer scientists, declining enrolment rates in agricultural programs, and competition from other sectors.

Performance (effectiveness)

The evaluation found that three years into the five-year program, Stream B has made significant progress towards the achievement of its intended outputs and outcomes. As of March 31, 2016, Stream B has committed $220.5 million over the five-year period of 2013-14 to 2017-18 to 14 Agri-science clusters, 86 Agri-science projects and 20 AAFC-led Knowledge Transfer projects. In total, 60 Collaborative Research and Development Agreements have been signed, allowing for support in the form of collaborative assistance from AAFC research scientists and experts in Industry-led R&D activities. Program activities span a variety of sectors and portfolios, with the largest proportion of Industry-led R&D funding committed to the oilseeds sector portfolio and the largest proportion of AAFC-led Knowledge Transfer funding committed to agro-ecosystem productivity and health.

Private sector investments in Stream B activities have more than doubled compared to private sector investments in similar programs under Growing Forward, due to increased AAFC investments, improved cost-share ratios, and increased industry awareness and interest in the program. The level of R&D collaboration has increased. New research networks have been established and existing networks have been strengthened and expanded across provinces and different types of organizations. Though most Agri-science cluster and project activities were not complete at the time of the evaluation, considerable progress had been made in developing and identifying new products, processes, and technologies for industry commercialization or adoption. Several examples of innovations resulting from Stream B activities were identified, some of which arose from activities that were initiated under previous programming and tracked over time. All technology, knowledge and information transfer targets have been exceeded. The program has enhanced R&D capacity in the agriculture and agri-food sector by providing research experience for individuals completing an MSc or PhD.

The program has made some progress towards its intended end outcome that the sector produces, adopts, and commercializes innovative products, processes, practices, services and technologies. It is too early to assess the full impact of Stream B activities since the program is not yet complete and it can take several years to further develop, adopt and commercialize products and practices resulting from Stream B supported R&D. A cost-benefit analysis conducted by Programs Branch in 2015-16 of two successful innovations identified high rates of return from long-term innovation investments by AAFC, provincial governments and industry in agricultural R&D activities across policy frameworksFootnote 2.

Program activities have resulted in significant economic impacts. Industry-led R&D expenditures of $120.2 million from April 1, 2013 to March 31, 2016 resulted in an estimated $289.9 million in spending across the Canadian economy during the same time period: $2.4 in total facilitative economic impacts for every dollar of AAFC funding. According to interviews, approximately 80%, or $232.0 million, of the total facilitative economic impact is estimated to be directly attributable to the Industry-led R&D component of Stream B (that is, it would not have occurred in the absence of AAFC assistance).

Total Facilitative Economic Impact from Stream B Industry-led R&D Expenditures, 2013-14 to 2015-16 (Estimated)
Description of this image follows.
Description of above image

The table below illustrates Figure 8 as follows:

AAFC funding $120.2 million
Leveraged funding $74.5 million
Indirect economic impact $95.2 million
Total facilitative economic impact over 3 years $289.9 million

Performance (efficiency and economy)

Stream B has achieved program outcomes more efficiently than similar programming implemented under Growing Forward in terms of higher levels of private sector investment leveraged per dollar of AAFC investment and lower AAFC costs per innovation developed.

The major factors contributing to the effective delivery of Stream B activities include collaboration across R&D funders and performers, cost-shared funding mechanisms, and continuous improvement of program delivery. Industry-led R&D funding mechanisms facilitate collaboration across a variety of R&D funders and performers, which contributes to improved capacity and research excellence and facilitates more coordinated and efficient R&D funding. The evaluation found some evidence that opportunities or mechanisms could be explored to facilitate increased collaboration between AAFC and other federal and provincial government organizations. AAFC's matching funding encourages industry stakeholders to fund larger scale and longer-term R&D activities that have the potential to yield substantial returns. The ongoing efforts to refine program delivery have improved the efficiency of delivery processes over time.

The main challenges that constrained the effective delivery of stream B activities include complex and time consuming application review, approval, and claims reporting processes due to the various collaborators involved and types of agreements required and limited ability to respond to emerging R&D needs and sectors. Some interviewees felt the clusters model does not allow sufficient flexibility to respond to emerging R&D needs, because it is difficult to reallocate funds between activities or towards new activities in a timely manner. Smaller, emerging sectors struggle to access Stream B funding due to capacity constraints.

Program spending as a percentage of the available budget authority increased from 49% in 2013-14 to 93% in 2015-16. The primary factors contributing to under-spending in 2013-14 were a large backlog in applications and delays in processing these applications at the outset of Growing Forward 2.

Mechanisms in place to track the performance of Stream B program activities have been successful in collecting a significant amount of performance data. However, there are limitations with respect to its consistency and accuracy. For example, the clarity of definitions and categories could be improved. There are also opportunities to improve the utility of performance data to industry and AAFC stakeholders in order to increase its use.

Recommendations

The key recommendations resulting from the evaluation are:

1.0 Introduction

1.1 Purpose of the report

This report represents the findings of the evaluation of the AgriInnovation Program (AIP) Stream B: Research, Development and Knowledge Transfer. Stream B provides non-repayable funding and collaborative assistance to support industry-led research and development (usually about two to ten years from commercialization), and knowledge transfer led by Agriculture and Agri-Food Canada (AAFC). The evaluation was conducted as part of AAFC's 2014-15 to 2018-19 Departmental Evaluation Plan by the department's Office of Audit and Evaluation.

2.0 Program profile

2.1 Program context

In April 2013, the Government of Canada launched the five-year, multilateral Growing Forward 2 policy framework for Canada's agriculture and agri-food sector. Growing Forward 2 is a $3 billion dollar investment by federal, provincial and territorial governments and the foundation for government agricultural programs and services.Footnote 3 By focusing on innovation, competitiveness and market development, Growing Forward 2 provides Canadian producers and processors with tools and resources that help them to innovate and capitalize on emerging market opportunities. The AIP is a five-year program (2013-14 to 2017-18) under Growing Forward 2 that resides within AAFC's Program Alignment Architecture Program Activity 2.1 Science, Innovation, Adoption and Sustainability and is delivered through three streams (Table 1).

Table 1: Description of AgriInnovation Program Streams
AIP Stream ( Program Alignment Architecture Sub-program) Description Budget
A: Research Accelerating Innovation (2.1.2) Discovery research stage (usually ten years or more from commercialization) $150.1 million
B: Research, Development and Knowledge Transfer (2.1.3) Pre-commercialization/adoption and knowledge transfer stage (usually two to ten years from commercialization) $268.1 million
C: Enabling Commercialization and Adoption (2.1.4) Commercialization/adoption stage (usually two years or less from commercialization) $137.3 million
Total $555.5 million

The AIP is designed to accelerate the pace of agri-innovation at each stage of the innovation continuum by supporting research, development, and knowledge transfer activities and facilitating the demonstration, commercialization and/or adoption of innovative products, technologies, processes, practices and services. The aim is to enhance the economic growth, productivity, competitiveness, adaptability and sustainability of the Canadian agriculture, agri-food and agri-based products sector and assist in capturing opportunities for the sector in domestic and international markets.

The AIP Stream B: Research Development and Knowledge Transfer (Stream B) is a continuation of similar programming implemented under the first Growing Forward framework from 2008-09 to 2012-13, consisting of Agri-science clusters and the Developing Innovative Agri-products Program, supported activities at the pre-commercialization stage of the innovation continuum.

2.2 Overview of Stream B: Research, development and knowledge transfer

This section provides a description of the Stream B objectives and activities, program delivery and resources. A logic model, details on program governance, and description of stakeholders for Stream B are in Annex A.

Program objectives and activities

The objective of Stream B is to accelerate the pace of innovation in the sector. The program includes two components: industry-led research and development (R&D) and AAFC-led knowledge transfer related to innovative products, practices, processes and systems in the agriculture, agri-food and agri-based products sector. The ultimate goal of the program is to increase market opportunities, strategic research planning and development, and foster innovation, industry investment and leadership.

Industry-led research and development

The objective of industry-led R&D is to encourage private sector investments and facilitate greater academia-industry-government and international collaborations to implement applied science and technology development projects that lead to new market opportunities for innovative agricultural products, practices, processes and/or services. This component also aims to accelerate the pre-commercialization development of new products, practices and processes by supporting the required applied science, technology development and knowledge transfer activities. Industry-led R&D is delivered through:

Examples of the types of activities funded under this component include:

Eligible recipients for Agri-science clusters and projects include non-profit organizations and private sector companies. Non-profit organizations are eligible to receive up to 75% of total eligible costs and for-profit organizations are eligible to receive up to 50% of total eligible costs. Eligible cost categories under the Industry‐led R&D component include administrative expenditures, salaries/benefits, contracted services, travel, capital/asset costs, and other expenditures directly associated with Agri-science cluster and project implementation. In-kind contributions can be counted toward recipient contributions, but the value of such contributions cannot normally exceed 10% of the total eligible costs of the Agri-science cluster or project. The Industry-led R&D component clusters' and projects' activities can be undertaken by the recipient and/or contracted to a research partner, or undertaken by AAFC researchers.

For work undertaken by the recipient or contracted to a research partner external to AAFC, AAFC provides non-repayable contributions funded through Vote 10 (grants and contributions) funds. Approved requests for contribution funding are formalized through a Contribution Agreement between AAFC and the recipient that details how contribution funding will be expended by the recipient. Contribution Agreements are managed through AAFC's Programs Branch. For research, development and knowledge transfer activities undertaken by AAFC researchers, AAFC Vote 1 funding (that is, internal operating funding) is used to support salaries and operating costs associated with AAFC scientists collaborating on R&D activities. Approved requests for AAFC research collaboration are formalized through a Collaborative Research and Development Agreement, an agreement managed by the Science and Technology Branch outlining research to be conducted by AAFC researchers.

AAFC-led knowledge transfer

The objective of this component is to facilitate the federally-led transfer of innovative ideas, tools, and practices resulting from R&D conducted by AAFC science professionals. Regionally relevant and commodity specific knowledge transfer approaches are informed by advice from industry users, such as farms and businesses, to ensure knowledge will be transferred according to local circumstances and needs, thereby enhancing sector competitiveness, profitability, sustainability and adaptability.

This component supports activities that facilitate the demonstration and transfer of new or improved products, processes or technologies developed by AAFC scientists. Only AAFC science and technology and knowledge transfer professionals are eligible as funding recipients (through Vote 1 funding). Projects are normally supported for one to three years in duration. Examples of Knowledge Transfer activities conducted by AAFC's Science and Technology Branch include, but are not limited to:

2.3 Purpose of the report

The total budget for Stream B activities from 2013-14 to 2017-18 is $268.1 million (Table 2). Of this total budget, $260.5 million is available for the Industry-led R&D component, mostly in the form of Vote 10 contribution funding (66%). The budget for the AAFC-led Knowledge Transfer component of Stream B is $7.6 million over the five-year period.

Table 2: Stream B – Research, development and knowledge transfer budget ($ millions)
Vote/total 2013-14 2014-15 2015-16 2016-17 2017-18 Total
Industry-led R&D Vote 1 19.1 16.7 16.6 17.1 19.6 89
Vote 10 26 31.1 40.9 36.8 36.8 171.5
Total 45.1 47.7 57.5 53.8 56.3 260.5
AAFC-led knowledge transfer Vote 1 1.5 1.5 1.5 1.5 1.5 7.6
Total 1.5 1.5 1.5 1.5 1.5 7.6
Total Stream B Vote 1 20.6 18.2 18.1 18.6 21.1 96.6
Vote 10 26 31.1 40.9 36.8 36.8 171.5
Total Stream B 46.6 49.3 59 55.3 57.9 268.1

3.0 Evaluation methodology

3.1 Evaluation scope

The evaluation of Stream B was conducted in accordance with the Treasury Board of Canada's Policy on Results. The evaluation covers the period from April 1, 2013 to March 31, 2016, though the program is ongoing until March 31, 2018. The evaluation was conducted midway through the program in order to inform the development of innovation programming under the next agricultural policy framework. The evaluation was designed to address the issues of relevance (continued need) and performance (the extent to which the program's activities resulted in the achievement of expected outcomes, and how efficiently and economically the program was delivered).Footnote 5

The following lines of evidence were used to address the evaluation questions:

Annex B provides more detailed descriptions of each of the specific lines of evidence and evaluation questions.

3.2 Evaluation limitations

The most significant limitations of the evaluation as well as strategies that were used to address those limitations include:

4.0 Evaluation findings

4.1 Relevance – Continued need for the Program

The agriculture and agri-food sector faces ongoing competitiveness and productivity challenges, low levels of investment in agricultural R&D, research capacity limitations and collaboration challenges. These issues are limiting the extent that resources can be leveraged to produce relevant, high-impact research.

Competitiveness and Productivity Challenges

According to studies conducted by AAFC and the Organization for Economic Co-operation and Development, there is a continued need for investment in agricultural R&D to address ongoing competitiveness challenges in the Canadian agriculture and agri-food sector. Threats such as variable input and output prices, influenced by world market and exchange rate shifts, threaten the profitability and global competitiveness of Canadian agriculture and agri-food industries. Climate change is increasing the severity and frequency of extreme weather and bringing new pests and diseases to Canadian crops and livestock.Footnote 6 Global demand for agricultural products is expanding due to growing populations, higher incomes, urbanization and changing diets. There is also an increasing demand for food with specific attributes (for example, food safety, nutrition, environmental stewardship, and animal welfare) and non-food products such as biodiesel.Footnote 7 To remain competitive, Canadian producers need to compete on costs and value-added attributes.Footnote 8

Agri-science cluster leads interviewed for this evaluation recognize the importance of innovation in addressing these competitiveness challenges, both in terms of addressing threats and seizing new, value-added opportunities. The primary objectives of the Stream B Agri-science cluster activities are to increase sector productivity (for example, through improved agronomic and feed practices, genetic traits, and technology) and expand market demand (for example, by developing new uses for agricultural products or developing premium products that consider animal welfare, sustainability, food safety and quality, and health benefits).

Evidence from the literature suggests that investments in agricultural R&D contribute to improved productivity and generate a substantial return on investment in the long-term. A recent study that examined agricultural Total Factor Productivity in Canada, Australia, and the United States found that investment in R&D capacity is a key contributor to long-term productivity growth.Footnote 9 The study assessed Total Factor Productivity impacts using a dynamic panel regression analysis. After controlling for climate conditions and services from public infrastructure, the study found that coefficients associated with R&D knowledge stocks are positive and significant at the one percent level, with consistent results using different scenarios. The results imply that a one per cent increase in the R&D knowledge stock tends to raise the agricultural Total Factor Productivity level by more than 0.3 per cent.

Investment in agricultural research and development

The federal government continues to play a lead role in funding agricultural R&D and Knowledge Transfer to complement and leverage limited investment, foster innovation in small and emerging agricultural industries and encourage industry to invest in research with long-term public good benefits.

There are limited public and private sector resources to invest in agricultural R&D in Canada. Total public agricultural R&D expenditures have been trending downwards in real terms since 2010-11 with $650 million in public sector investments in 2015-16, about 70% of which is from the federal government.Footnote 10 Similar trends have been reported in other jurisdictions, such as the United States and United Kingdom. Canada is likely to be impacted more by this trend, since federal and provincial governments generally represent the largest source of funding for agricultural R&D in Canada.Footnote 11 Canada continues to rank low compared to other countries with respect to private sector investment in R&D. According to a recent Organization for Economic Co-operation and Development study on Canadian agricultural innovation policies, Canada does not perform well in terms of business expenditures on R&D (Business Enterprise R&D Spending) and patenting compared to other Organization for Economic Co-operation and Development countries. Footnote 12 Though private sector investments in agricultural R&D have grown steadily since the 1980s (particularly in crop varieties and food processing) reaching approximately $226 million in 2011, several factors constrain private sector R&D investment such as narrow profit margins and lower intellectual property protection for plant breeding compared to other Organization for Economic Co-operation and Development countries.Footnote 13

Private sector investment in agricultural R&D varies by sector size, profitability, province, and mechanisms in place to raise industry investment. Commodity-based sectors, such as beef, canola and pulses, often raise industry funding for marketing and research from mandatory levies or check-off systems. Investments are managed by non-profit industry associations, which tend to have more capacity to fund agricultural R&D since structured systems are in place. Other smaller and emerging sectors, such as bioproducts, tend to raise funds from individual industry partners, with specific goals to commercialize technologies. For Agri-science clusters in these industries, it is more difficult to raise private investment due to the higher number of partners and agreements required.

The level of industry investment in agricultural R&D also varies in terms of the focus of the activities and timing of the expected return on investment. Industry stakeholder interviewees reported that in the absence of AAFC financial assistance they would likely continue to fund some productivity-related research expected to generate short-term returns, but less likely to fund R&D related to market expansion, as it tends to be more costly and take a longer time to yield returns. Market expansion activities also tend to be associated with more public good benefits such as agricultural and agri-food products that consider animal welfare, sustainability, food safety and quality, and health benefits.

Agricultural research and development capacity

There is an ongoing need to leverage agricultural R&D capacity, including highly qualified personnel, in Canada. The Canadian agricultural research and innovation system is complex and involves a range of federal and provincial government, university, industry, and other stakeholders. A recent study conducted by AAFC on Canadian agricultural innovation capacity estimates that agricultural R&D is performed at over 300 research facilities across Canada, each with slightly different roles and areas of focus:Footnote 14

An estimated 3,580 highly qualified personnelFootnote 18 conduct agricultural R&D at these facilities, of which 380 are AAFC scientists.Footnote 19 Highly qualified personnel are recruited through various mechanisms such as Canadian universities (for example, 28,332 students were enrolled in agricultural and natural resources programs in Canadian colleges and universities in 2012) and international recruitment, exchanges and partnerships.Footnote 20

Several challenges may constrain the ability to meet future highly qualified personnel needs to maintain the sector's R&D capacity. It is estimated that more than 60% of agricultural sciences highly qualified personnel in Canada will retire over the next five to twenty years.Footnote 21 Similar trends have been reported in the United States and Australia. The disciplines facing the greatest potential shortages include both traditional areas such as plant, soil and animal science, agronomy, biosystematics/taxonomy, and entomology as well as emerging areas such as genetics and bioinformatics. The potential consequence of a lack of highly qualified personnel in agricultural sciences in the future is that the sector will not have the capacity to meet its R&D needs, which could impact its competitiveness. There are also challenges to integrating newer scientists due to limited opportunities to train with more experienced scientists and lack of industry awareness of newer scientists. The supply of new highly qualified personnel is further constrained by competition from other sectors, a lack of awareness of agricultural science careers on the part of students, and declining enrolment rates in agricultural programs (for example, 1.42% of all students enrolled in Canadian colleges and universities were enrolled in agricultural and natural resources programs in 2012, compared to 1.71% in 2002).Footnote 22

The Industry-led R&D component addresses the issue of limited sector R&D capacity by providing the opportunity for firms that are interested in innovation but lack the highly qualified personnel and the facilities to undertake research to work in collaboration with AAFC scientists and other scientists from universities, the private sector or other R&D organizations. Stream B-funded R&D activities can also provide opportunities for university graduate students to gain research experience.

Collaboration and knowledge transfer

There is a need to facilitate agricultural R&D collaboration since it leverages R&D investment and capacity and generates research excellence. Collaboration optimizes the capacity available for agricultural R&D by leveraging resources across Canada to support national goals and priorities. According to the Organization for Economic Co-operation and Development, collaboration is needed due to increasing demands for more specialized or capital-intensive areas of science and due to the fact that few organizations have sufficient funds or highly qualified personnel capacity to address all of these needs without working with external collaborators.Footnote 23

R&D collaboration between different organizations contributes to improved research excellence and tends to result in increased knowledge and technology transfer. A study of collaboration among thirteen federal government departments found that compared to overall publications, publications with co-authors tended to result in more highly cited papers, a measure of a paper's contribution to the field of research. AAFC's highly cited paper rates were higher among co-publications involving external organizations compared to overall publications (1.36 versus 1.24), particularly among co-publications with international organizations (1.65), independent R&D facilities (1.61), private sector organizations (1.47), and academic institutions (1.41).Footnote 24

The collaboration study also found that AAFC collaboration rates are increasing at a higher rate than other federal government departments involved in scientific research. AAFC showed steady year-over-year increases, from a low of 63.8% publications involving external collaborators in 2000 to a high of 87.4% in 2014 (Figure 1). This growth was driven by increased collaboration with partners outside the federal government. The trend coincides with the introduction of the cluster funding model under Growing Forward and its continued support under Growing Forward 2 through Stream B, which is designed to support and strengthen collaboration. International collaboration rates have shown steady growth since 2000, and are approaching the 50% mark in recent years (for the thirteen federal departments and agencies, overall). AAFC and Environment Canada are leading the way in increases to international collaboration.Footnote 25

Figure 1: R&D Collaboration Trends among Thirteen Canadian Federal Departments, 2000-2014
Description of this image follows.
Description of above image

The table below illustrates Figure 1 as follows:

Federal organization Number of co-publications Collaboration rate Minimum Maximum
Total – 13 Federal Canadian federal departments 49,554 79.6% 69.9% 87.9%
PHAC – Public Health Agency of Canada 2,601 94.2% 79.0% 100%
TC – Transport Canada 52 86.7% 33.3% 100%
ECCC – Environment and Climate change Canada 7,975 86.6% 76.1% 91.0%
DFO –Fisheries and Oceans Canada 5,108 86.6% 72.3% 92.7%
AECL – Atomic Energy Canada Ltd. 1,036 81.9% 66.7% 90.8%
CSA – Canadian Space Agency 342 79.9% 62.5% 94.7%
NRCan – Natural Resources Canada 7,049 79.9% 73.2% 87.4%
NRC Canada – National Research Council 13,190 79.2% 69.6% 91.2%
DRDC – Defence Research and Development Canada 1,601 78.5% 68.2% 86.3%
HC – Health Canada 2,248 76.4% 70.5% 82.9%
AAFC – Agriculture and Agri-Food Canada 10,221 75.5% 63.8% 87.4%
CFIA – Canadian Food Inspection Agency 1,004 74.6% 58.6% 83.3%
CRC – Communications Research Centre Canada 445 69.2% 50.0% 90.7%

Note: The collaboration rate for each organization is coloured relative to the weighted average across all 13 federal bodies. The colouring scheme ranges from dark red (lowest value of all the federal bodies) to dark green (highest value of all the federal bodies) with white in the middle (equal to the federal average). To appreciate the fluctuations of the sparklines the maximum value was set differently for each department, hence the height of the bars should not be compared between departments.

Source: Science-Metrix. 2016. Study on the Scientific Collaboration between the Canadian Government and Sectors of the National Science and Innovation System. Prepared for the Federal Science and Technology Secretariat; computed by Science-Metrix using the Web of Science (Thomson Reuters).

Figure 2 shows the external R&D collaboration patterns between Canadian federal departments and other organizations (academic, private, and other). The thickness of the lines represents the frequency in which the departments collaborate with particular types of external organizations. AAFC collaborates most frequently with academic institutions. Among the thirteen federal government departments reviewed, AAFC, National Research Council Canada, and Environment Canada are the most central actors in the collaboration network (represented by the size of their representative circles).

Figure 2: External R&D Collaboration Patterns between Canadian Federal Departments and Other Organizations, 2000-2014
Description of this image follows.
Description of above image

The table below illustrates Figure 2 as follows:

Department or agency Number of collaborations with academic partners Number of collaborations with private sector Number of collaborations with other sector Total collaborations by department or agency
Agriculture and Agri-Food Canada - AAFC 6,530 812 2,553 9,895
Atomic Energy of Canada Limited - AECL 302 92 136 530
Canadian Food Inspection Agency - CFIA 391 63 276 730
Communications Research Centre of Canada - CRC 283 53 25 361
Canadian Space Agency - CSA 224 57 91 372
Department of Fisheries and Oceans - DFO 2,989 374 1,813 5,176
Defence Research and Development Canada - DRDC 968 236 184 1,388
Environment and Climate Change Canada - EC 5,048 578 2,234 7,860
Health Canada - HC 929 57 254 1,240
National Research Council - NRC 9,402 1,177 2,908 13,487
Natural Resources Canada - NRCan 4,473 762 2,085 7,320
Public Health Agency of Canada - PHAC 1,609 135 880 2,624
Transport Canada - TC 27 10 5 42
Total collaborations by type 33,175 4,406 13,444 51,025

Source: Science-Metrix. 2016. Study on the Scientific Collaboration between the Canadian Government and Sectors of the National Science and Innovation System. Prepared for the Federal Science and Technology Secretariat.

Despite increasing collaboration rates, the Organization for Economic Co-operation and Development study on Canadian agricultural innovation policies identified major collaboration and knowledge transfer gaps: "No other country among a group of high R&D spenders, displays such a large divergence between human resources and research infrastructure, and firm R&D and patenting activity. The imbalance between world class academic research and lacklustre business R&D has led policy makers to re-examine the linkages between academia and business."Footnote 26 Another study found that Canadian academic institutions involved in agricultural R&D have limited interest in collaborating with industry partners, while industry partners see these institutions as critical to meeting their R&D needs.Footnote 27 Stream B addresses these needs by facilitating collaboration and knowledge transfer to maximize the use of limited resources and facilitate uptake of research results. The Industry-led R&D Agri-science clusters model addresses these constraints by encouraging connections between industry, academia and AAFC to further sector goals. The AAFC-led Knowledge Transfer component complements these activities by facilitating the transfer of AAFC-led R&D results to sector stakeholders.

4.2 Performance – Effectiveness

The following section examines the performance of Stream B in terms of the extent to which it has achieved its intended outputs and outcomes.

4.2.1 Activities

As of March 31, 2016, $220.5 million in AAFC funding has been committed to Stream B activities taking place from 2013-14 to 2017-18. A total of $217.0 million has been committed to Industry-led R&D activities, the largest share of which is dedicated to the oilseeds, cereals and pulses, and horticulture sub-sectors (Figure 3). A total of $3.5 million has been committed to AAFC-led Knowledge Transfer projects taking place from 2013-14 to 2017-18, primarily in the area of agro-ecosystem productivity and health (Figure 4).

Figure 3: Funding Committed for Industry-led Stream B Activities 2013-14 to 2017-18 as of March 31, 2016
Description of this image follows.
Description of above image

The table below illustrates Figure 3 as follows:

Program component Funding committed in millions
Oilseeds 42.4
Cereals and pulses 37.5
Horticulture 31.3
Dairy, pork, poultry, other livestock 27.4
Beef and forages 13.2
Bioproducts 12.4
Agri-food 9.5
Agro-ecosystem productivity and health 6.4
Organic 6.0
Greenhouse 3.4
Other 27.6
Figure 4: Funding Committed for AAFC-led Knowledge Transfer Stream B Activities 2013-14 to 2017-18 as of March 31, 2016Footnote 28
Description of this image follows.
Description of above image

The table below illustrates Figure 4 as follows:

Program component Funding committed in millions
Agro-ecosystem productivity and health 2.0
Biodiversity and bioresources 0.6
Horticulture 0.2
Beef and forages 0.2
Cereals and pulses 0.2
Bioproducts 0.1
Agri-food 0.1
Other 0.1

Source: AAFC Investment Analysis data, June 2016

4.2.2 Outputs

Stream B has achieved the targets of 14 Agri-science clusters formed in priority areas for the sector and 20 Knowledge Transfer project proposals approved (Table 3). More than two-thirds of the remaining targets have been achieved: 60 of the targeted 70 Collaborative Research and Development Agreements have been signed, 86 of the targeted 120 Agri-science projects have been approved, and $217.0 million of the budgeted $260.5 million in AAFC investment has been committed.

Table 3: Achievement of Stream B Outputs
Program outputs Indicator Growing Forward results Target for March 31, 2018 Achieved as of March 31, 2016
Industry-led R&D Collaborative Research and Development Agreements # of Collaborative Research and development Agreements 62 70 60
Agri-science Clusters # of clusters formed in priority areas for the sector 10 14 14
Agri-science Projects to develop new products, process or technologies # of projects to develop new products, process or technologies 81 120 86
Funding provided by AAFC AAFC Investment $171.0 million $260.5 million $217.0 million
(Committed)
AAFC-led knowledge transfer Project Proposals # of Knowledge Transfer proposals approved 14 20 20
Source: Stream B Program Performance and Risk Management Strategy and AAFC Databases

Since Agri-science projects are accepted through a continuous intake process, it is expected that these numbers will increase over the remaining two years of the program. Stream B has already surpassed Growing Forward program results with respect to most outputs. Four additional Agri-science clusters have been formed, more industry-led Agri-science and AAFC-led Knowledge Transfer projects have been approved, and AAFC investment in Industry-led R&D activities has increased by 27%.

4.2.3 Outputs

Outcome: The sector has access to government and private R&D capacity and funding to support research, development and the application of knowledge and technologies.

Stream B has facilitated increased private sector investment in Industry-led R&D. As of June 2015, Stream B has resulted in $97.2 million in private sector investment (Table 4), more than double the $44.7 million in private investment for similar activities under Growing Forward and 90% of the level of private investment in Stream B activities targeted for March 31, 2018. The increase in private investment is attributable to increased AAFC investments, modified cost-shared ratios, and increased industry awareness and interest in the program. Approximately $4.0 million in other government investment has also been raised toward Stream B activities, exceeding Growing Forward results but below the amount of $9.0 million targeted for March 31, 2018.

Table 4: Achievement of Stream B Outcomes – R&D Funding Leveraged - Industry-led R&D
Program outcome Indicator Growing Forward results Target for
March 31, 2018
Achieved as of June 2015
The sector has access to government and private R&D capacity and funding to support research, development and the application of knowledge and technologies Dollar value of private sector investments and other government investments Private investment:
$44.7 million
Other
Government investment:
$3.7 million
Private investment:
$108 million
Other
Government investment:
$9.0 million
Private investment:
$97.2 million
Other
Government investment:
$4.0 million
Source: Stream B Program Performance and Risk Management Strategy and AAFC Investment Analysis Data, June 2015

Various factors influence the level of private sector investment in Industry-led R&D activities. Stream B guidelines require that activities led by private companies meet a minimum 50% industry matching requirement, whereas activities led by not-for-profit organizations require 25% matching. According to an investment analysis prepared by AAFC's Innovation and Growth Policy Division (Figure 5), a higher proportion of industry investment was leveraged by Agri-science projects (36% of budgeted project costs) compared to Agri-science clusters (28%). Figure 6 shows that those focused on developing new machinery and computer-related equipment leveraged a higher percentage of industry investment compared to other sectors and cross-cutting areas. This is expected since Agri-science projects and machinery and computer-related equipment R&D activities are more likely to involve private sector companies. It was also reported by AAFC staff that during the fall of 2013, the policy was changed to give preference to applicants that raised 50% in matching funding due to the high level of demand for the program. This impacted the proportion of private sector funding from Agri-science projects more than Agri-science clusters since most clusters were approved during the first intake.

Figure 5: Source of Industry-led R&D Funding by Project Type
Description of this image follows.
Description of above image

The table below illustrates Figure 5 as follows:

Project type Source Percentage
Clusters Vote 10 49
Industry 28
Vote 1 23
Projects Vote 10 44
Industry 36
Vote 1 17
Other 3

Source: AAFC Investment Analysis Data, June 2015

Figure 6: Source of Industry-led R&D Funding by Sector and Cross-cutting Area
Description of this image follows.
Description of above image

The table below illustrates Figure 6 as follows:

Sector and cross-cutting area Source Percentage
Machinery and computer related Vote 1 9
Vote 10 41
Other government department 2
Industry 48
Bioproducts Vote 1 7
Vote 10 51
Other government department 4
Industry 38
Agro-ecosystem productivity and health Vote 1 15
Vote 10 44
Other government department 4
Industry 37
Oilseeds Vote 1 20
Vote 10 47
Industry 32
Agri-food Vote 1 24
Vote 10 43
Other government department 1
Industry 32
Horticulture Vote 1 21
Vote 10 45
Other government department 2
Industry 32
Dairy, pork, poultry, other livestock Vote 1 9
Vote 10 59
Other government department 1
Industry 31
Cereal & pulses Vote 1 38
Vote 10 31
Other government department 2
Industry 28
Organic Vote 1 25
Vote 10 50
Industry 25
Forages & beef Vote 1 34
Vote 10 44
Industry 22

Source: AAFC Investment Analysis Data, June 2015

Stream B has helped to improve the efficiency of private sector investments in agricultural R&D. Many of the Agri-science cluster leads interviewed reported that the clusters have facilitated a more efficient use of research investments by pooling provincial industry resources to support sector research priorities on a national scale. For example, the Swine Agri-science Cluster successfully introduced a national research check-offFootnote 29 of 2.5 cents per head in 2013 through a memorandum of understanding signed with seven provinces and attracted 40 industry partners in the call for proposals. The proportion of the Canadian beef cattle producer national check-off dedicated to research increased from 6% in 2010-11Footnote 30 to 16% in 2015-16.Footnote 31 According to Agri-science cluster lead interviewees, the increase in industry investment in research is due, in part, to the cluster's ability to articulate national research priorities and demonstrate impacts as a result of the cluster. Some Agri-science clusters are seeking to leverage further private sector investment from new sources such as seed companies (Field Crops Agri-science Cluster) and the food industry (Pulse Agri-science Cluster).

Outcome: The sector increases its collaboration and partnership for research and development

Collaboration between AAFC scientists and other scientists appears to be increasing. As of October 2016, 42% of Agri-science projects and clusters involved both Vote 1 and Vote 10 activities, exceeding the Growing Forward results of 40% but less than the 50% target expected by March 31, 2018 (Table 5). AAFC's rate of scientific collaboration with external organizations (that is, co-publication as a percentage of total publications) has been increasing over time.Footnote 32 Interviews with AAFC representatives suggest that this is due to Agri-science cluster programming.

Table 5: Achievement of Stream B Outcomes – Research Collaborations - Industry-led R&D
Program outcome Indicator Growing Forward results Target for March 31, 2018 Achieved as of October 2016
The sector increases its collaboration and partnership for research and development % of projects involving both Vote 1 and Vote 10 activities 40% 50% 42%
Source: Stream B Program Performance and Risk Management Strategy and AAFC Databases

Agri-science cluster lead interviewees suggest that Stream B activities have strengthened research networks and the capacity to leverage investment in R&D. New research networks have been established, and networks that had been developed during Growing Forward were strengthened and expanded across provinces and different types of organizations. Agricultural scientists are connecting with researchers they had not engaged with before. For example, Beef Agri-science Cluster activities led to the development of an integrated network of specialists focusing on antimicrobial resistance in beef production including veterinarians, feedlot researchers and other federal government researchers from the Public Health Agency of Canada.

An estimated 88% of all Industry-led R&D activities involved one or more collaborator type in addition to the recipient. About one fifth of activities involved two or more collaborator types, with 27 activities involving larger research teams with four or more collaborator types. The most frequent collaborators on Agri-science cluster or project activities were AAFC research centres and post-secondary institutions (Figure 7).

Collaborator types also varied by sector and cross-cutting area. For example, there was more frequent involvement of AAFC Research Centres in cereals and pulses activities and more frequent involvement of post-secondary institutions in dairy, pork, poultry, and other livestock activities (Figure 8).

Figure 7: Number of Industry-led R&D Activities per Collaborator TypeFootnote 33
Description of this image follows.
Description of above image

The table below illustrates Figure 7 as follows:

Collaborator type Number of activities
AAFC research centres 366
Post-secondary institutions 342
Other for-profit corporations 65
Supply chain organizations 54
Other research centres 50
Provincial government departments 43
Producers 31
Other not for-profit corporations 29
Provincial research centres 28
Processors 13
Other federal government departments 9

Source: AAFC Investment Analysis data, June 2015

Figure 8: Industry-led R&D Collaborator Types by Sector and Cross-cutting Area
Description of this image follows.
Description of above image

The table below illustrates Figure 8 as follows:

Sector and cross-cutting area Collaborator type Percentage
Cereals and pulses AAFC research centres 62
Post-secondary institutions 27
Other provincial or federal governments 3
For-profit companies 8
Other research centres or not-for-profit 0
Machinery and computer related AAFC research centres 47
Post-secondary institutions 22
Other provincial or federal governments 3
For-profit companies 3
Other research centres or not-for-profit 25
Organic AAFC research centres 40
Post-secondary institutions 45
Other provincial or federal governments 4
Oilseeds AAFC research centres 38
Post-secondary institutions 33
Other provincial or federal governments 3
For-profit companies 22
Other research centres or not-for-profit 6
Horticulture AAFC research centres 30
Post-secondary institutions 28
Other provincial or federal governments 12
For-profit companies 16
Other research centres or not-for-profit 14
Agro-ecosystem productivity and health AAFC research centres 27
Post-secondary institutions 28
Other provincial or federal governments 21
For-profit companies 18
Other research centres or not-for-profit 7
Agri-food AAFC research centres 25
Post-secondary institutions 36
Other provincial or federal governments 5
For-profit companies 23
Other research centres or not-for-profit 11
Bioproducts AAFC research centres 24
Post-secondary institutions 24
Other provincial or federal governments 3
For-profit companies 47
Other research centres or not-for-profit 3
Dairy, pork, poultry, other livestock AAFC research centres 21
Post-secondary institutions 58
Other provincial or federal governments 2
For-profit companies 8
Other research centres or not-for-profit 11

Source: AAFC Investment Analysis data, June 2015; for-profit companies includes supply chain organizations, producers, processors, and other for-profit corporations.

The capacity built through Stream B activities has leveraged further follow-on projects, leading to additional R&D impacts. For instance, a Field Crops Agri-science Cluster-funded soybean breeding systems research collaboration between a Quebec not-for-profit, two universities and an AAFC research centre has led to a follow-on project with $8 million in funding from Genome Canada and various other provincial, industry and private sector organizations.Footnote 34 Agri-science cluster lead interviewees note that the increased collaboration in research has also resulted in better value for money since a broader range of expertise and capacity is leveraged per dollar of investment. Collaboration with industry has helped to better link research to commercial priorities, which makes the research more relevant for industry applications.

Outcome: AAFC-led Innovations Developed and Identified for Transfer: Technology, knowledge and information covering the full range of AAFC research, development and technology is developed for stakeholders, target groups and the sector

AAFC-led Knowledge Transfer projects resulted in 8 new/improved products identified and made available for transfer to the sector, 71 scientific publications (for example, articles or papers, conference proceedings and abstracts), and 4 technology transfer publications (Table 6). Examples of AAFC-led Knowledge Transfer activities are provided in Text Box 1.

Table 6: Achievement of Stream B outcomes – AAFC-led innovations developed and identified for transfer - AAFC-led Knowledge Transfer
Program outcome Indicator Growing Forward results Target for
March 31, 2018
Achieved as of March 31, 2016
Technology, knowledge and information covering the full range of AAFC research, development and technology is developed for stakeholders, target groups and the sector # of existing technologies identified and made available for transfer to the sector N/A N/A 8 new/
improved products
Knowledge Transfer Instruments # of knowledge transfer instruments developed (for example, brochures, flyers and trade journals, etc.) N/A N/A 71 scientific publications
4 technology transfer publications

N/A: Not available

Source: Stream B Program Performance and Risk Management Strategy and AAFC Databases

Text Box 1. Examples of AAFC-led Knowledge Transfer Activities
  • Helping First Nations to better understand ancestral varieties and growing methods related to the Three Sisters: maize, squash, and beans for potential food applications (for example, maize flour). This project resulted in the development of flyers, translated in English, French and First Nations languages. Three First Nations signed memorandums of understanding with AAFC agreeing to participate in the project.
  • Enhancing knowledge of beneficial management practices related to wastewater from confined livestock sites among producers and regulators to minimize health and environment risks. The project resulted in the development of a factsheet and presentations to six different producer organizations and groups.
  • Improving access to weather and climate decision support tools, which can help agricultural producers to better manage climate-related risks and optimize pesticide, nitrogen and irrigation water use. The project resulted in publications such as an AgWeather Atlantic factsheet, guide, and YouTube video.
  • Transferring research on water and nutrient management to the horticultural sector, helping growers to better respond to regulatory requirements. This project aimed to utilize AAFC research to quantify and communicate the environmental risks for different horticultural production systems in Ontario and resulted in presentations to thirteen grower and regulatory organizations.

4.2.4 Intermediate outcomes

Outcome: Development of Innovations: The sector develops or advances knowledge and technologies for industry uptake for commercialization or adoption

Stream B activities have resulted in many new discoveries, products and technologies. As of March 31, 2016, Industry-led R&D activities have exceeded both Growing Forward results and Stream B targets with respect to the number of discoveries and technologies generated. In total, 870 new discoveries and technologies including new Intellectual Property items, new or improved products, processes, practices, varieties and knowledge have been developed, greater than the 540 targeted by March 31, 2018 (Table 7). A total of 844 peer reviewed publications have been published, achieving 84% of the 1,000 targeted by program close. As R&D progresses in each year of the program, the results are higher year over year. Therefore, it is expected that these targets will be met. The Achievement database indicates that 35 products developed have commercial potential.

Table 7: Achievement of Stream B Outcomes – Development of Innovations - Industry-led R&D
Program outcome Indicator Growing Forward results Target for
March 31, 2018
Achieved as of March 31, 2016
The sector develops or advances knowledge and technologies for industry uptake for commercialization or adoption # of discoveries, technologies generated (for example, new Intellectual Property items, new or improved products, processes, practices, varieties and knowledge) 540 540 870
# of peer reviewed scientific publications 1,000 1,000 844
# of products that have commercial potential N/A N/A 35
N/A: Not available

There are some challenges with respect to how innovations are categorized and counted. For instance, some scientists may define a step towards developing an innovation as a type of innovation, while others may use a more stringent definition. It was found that performance measures could be refined under a new policy framework to reflect only final results (for example, new varieties) and not those representing a step on the way (for example, gene sequences). The clarity of definitions and categories could also be improved.

Several examples of innovations resulting from Stream B activities were identified (Text Box 2), some of which were continuations of activities initiated under Growing Forward or other programming predating Growing Forward 2. Innovations are often a result of many years of research, development and scale-up work. For most Stream B activities, it is too early to have seen an impact in the sector due to the long-term nature of the R&D work undertaken under Stream B.

Text Box 2. Examples of Innovations and Discoveries Resulting from Stream B Activities

New Uses for Agricultural Products

  • Alternatives to conventional fibreglass vehicle parts using flax and hemp fibers. Recent investments are focused on efforts to develop a process to test natural fibres for viability in bio-composite products. These applications will open up a new revenue stream for farmers and provide more environmentally sustainable product options in the long-term.
  • Process to convert mushroom waste into nutrient-rich fertilizers, providing a new revenue stream for the 2.5 billion kilograms of waste discarded by mushroom growers in North America each year.
  • Camelina-based eco-friendly alternative to castor oil. Camelina is a drought tolerant oilseed crop that is being developed as an eco-friendly alternative to castor oil. Samples have been sent to a large multinational company for possible use in polyurethane coating applications.

Improved product attributes

  • Defining the environmental 'hoofprint' of beef production. Beef is often portrayed as environmentally unfriendly due to concerns about Greenhouse Gases and other waste. This project identified that producing 1 kg of beef today creates 15% less GHGs than in 1981 due to improved production practices.
  • Environmentally friendly water extraction system in nutraceutical processing. Growing Forward and Growing Forward 2 investments support the development of a more environmentally friendly water extraction system, which consumes less energy and whose by-products are non-toxic. Four new extract products were developed through this new system for use in nutraceuticals.
  • Tools which will improve meat quality and animal welfare in swine production. Research is being conducted into ways of identifying hormones associated with an unpleasant smell when meat is cooked, in order to remove affected pigs from the breeding stock and improve animal welfare practices.

Technologies and products that improve productivity

  • Drone-based technology to quickly assess soil yield in corn fields. Four new products have been developed including a quick, inexpensive and reliable test that identifies high and low yielding soil areas in corn fields by employing drones equipped with multispectral sensors.
  • Wireless sensors to assess crop disease prediction and prevention. Growing Forward and Growing Forward 2 investments support the development and testing of new wireless sensors that virtually collect data needed for crop disease prediction and prevention for producers in real-time, saving in labour costs, crop losses, and excess use of pesticides.
  • Bio-based honeybee crop control system. A honeybee dispenser system has been developed to enable crop control through pollination without the adverse consequences of spraying chemicals. The project has demonstrated that the system functions effectively on large field scale canola crops.

Outcomes: Technology, Knowledge and Information Transferred:

  1. The sector transfers research findings and methods to target groups/project stakeholders;
  2. Technology, knowledge and information is transferred to intended users, farms and firms, thereby enhancing sector competitiveness and profitability

All technology, knowledge and information transfer targets have been exceeded and have surpassed Growing Forward results (Table 8). As of March 31, 2016, Stream B Industry-led R&D activities have resulted in 2,208 knowledge transfer instruments (for example, brochures, flyers, trade journals, etc.) and 2,645 knowledge transfer events (for example, presentations at industry tradeshows and conferences), tripling the 700 instruments and 930 events targeted. There were a reported 1,065,437 participants in these knowledge transfer events, exceeding the target of 50,000, among which 25,535 indicated that they intend to adopt the knowledge and technologies presented at these events, surpassing Growing Forward results and the Stream B target of 1,000. In light of the higher than expected outcomes, performance measures related to knowledge transfer could be refined under a new policy framework to ensure they accurately capture activities where knowledge is transferred to a target audience that could adopt the innovations.

Table 8: Achievement of Stream B Outcomes – Technology, knowledge and information transferred
Program output/outcome Indicator Growing Forward results Target for March 31, 2018 Achieved as of March 31, 2016
Industry-led R&D Knowledge Transfer Instruments # of knowledge transfer instruments developed (for example, brochures, flyers and trade journals, etc.) 700 700 2,208
Knowledge Transfer Events # of knowledge transfer events organized through the sector 930 930 2,645
# of individuals participating in knowledge transfer events 50,000 50,000 1,065,437
The sector transfers research findings and methods to target groups/project stakeholders # of individuals participating in knowledge transfer events intending to adopt new knowledge and technologies 1,000 1,000 25,535
# of media reports 480 480 623
AAFC-led knowledge transfer Technology, knowledge and information is transferred to intended users, farms and firms, thereby enhancing sector competitiveness and profitability # of knowledge transfer events organized by AAFC N/A N/A 101 knowledge and expertise contributions
# of individuals participating in knowledge transfer events organized by AAFC N/A N/A Data Not Available

N/A: Not available

Source: Stream B Program Performance and Risk Management Strategy and AAFC Databases

According to interviews with Agri-science cluster leads, different factors impact the ease with which knowledge and technology is transferred. For example, knowledge transfer systems are in place for Intellectual Property-related innovations through breeding programs or projects involving private sector partners. Though a major role for Agri-science clusters is to undertake knowledge transfer, capacity to support broader-based knowledge transfer activities varies by cluster. In some cases, an Agri-science cluster lead organization may be a provincial industry association with limited mandate or capacity to undertake widespread industry knowledge transfer. Since much of the R&D activities are contracted to universities, it can be challenging for some Agri-science clusters to take ownership of the research results and effectively translate and disseminate findings to industry stakeholders. Some Agri-science clusters are employing innovative techniques to disseminate knowledge such as a mobile-friendly interactive research hub, webinars, and conferences with closed-door sessions to discuss confidential technologies.

AAFC-led Knowledge Transfer projects resulted in 101 knowledge and expertise contributions, most frequently consisting of speaking or presentation activities and acting as a scientific or technical authority.

Outcome: The sector has additional capacity to develop knowledge and technologies

The program has made progress towards its target of newly trained highly qualified personnel in the sector. As indicated in Table 9, as of March 31, 2016, 103 persons completed a MSc or PhD during program activities, about 40% of the 260 targeted by March 2018. Since the number of graduates per year is increasing year over year, the program is on track to meet its target by March 31, 2018.

Table 9: Achievement of Stream B Outcomes – Improved Sector Innovation Capacity - Industry-led R&D
Program outcome Indicator Growing Forward results Target for
March 31, 2018
Achieved as of March 31, 2016
The sector has additional capacity to develop knowledge and technologies # of newly trained, highly qualified personnel in the sector 260 260 103 persons who completed a MSc or PhD during project/activity
Source: Stream B Program Performance and Risk Management Strategy and AAFC Databases

Agri-science cluster leads advise that the program has helped to build sector highly qualified personnel capacity in various areas. For example, during a Field Crops Agri-science Cluster activity, an AAFC scientist who is one of the few Canadian specialists in corn inbred development was able to train a graduate student researcher at the University of Manitoba. A Swine Agri-science Cluster activity to develop effective feeding strategies for piglets involved nine graduate students and resulted in two participating students completing their PhD and one participant completing their MSc.

4.2.5 End outcome

Outcome: The sector produces, adopts, and commercializes innovative products, processes, practices, services and technologies

The program has made some progress towards the achievement of its intended end outcome. As of March 31, 2016, a total of five completed Agri-science projects were reported to have resulted in 20 commercial products. One innovation developed has been adopted by sector stakeholders (Table 10). It is too early to expect that Stream B supported R&D activities have realized all impacts associated with the end outcome because the program is not yet complete. Recipients are only asked to report on end outcomes in their final performance report. As of March 31, 2016 the program had received final performance reports for 11 completed projectsFootnote 35. Due to the nature of the R&D funded and the location of these activities along the innovation continuum, it can take two to ten years following activity completion for sector stakeholders to further develop, adopt or commercialize products or practices resulting from Stream B innovations. There are also some limitations to using Total Factor Productivity Growth as an indicator since it is also impacted by many factors beyond the scope of the program, such as investments by other programs or private sector companies in agricultural R&D. Data was not available for this indicator at the time of the evaluation.

Table 10: Achievement of Stream B Outcomes – Sector Production, Commercialization and Adoption of Innovations - Industry-led R&D
Program outcome Indicator Growing Forward results Target for
March 31, 2018
Achieved as of March 31, 2016
The sector produces, adopts, and commercializes innovative products, processes, practices, services and technologies Number of projects that resulted in a commercial product N/A N/A 5
Products being used/adopted by the sector N/A 5 case studies that demonstrate how products are used 1 innovation adopted by the sector;
2 case studies: Swath Grazing &
Midge-tolerant wheat
case studies (see Text Box 3)
Total Factor Productivity Growth 100 108 Data Not Available
N/A: Not available

A cost-benefit analysis conducted by Programs Branch in 2015-16 of two successful innovations provided some evidence of high rates of return from long-term innovation investments. The analysis showed that the midge-tolerant wheat and swath grazing innovations benefited from AAFC, provincial government, and industry investments in R&D activities since the mid-1990s and are expected to result in benefit/cost ratios of 37:1 and 170:1 respectively by 2025 (see Text Box 3).Footnote 36

Text Box 3. Case Study Examples of Return on Investment from AAFC R&D Investments
  • Midge-tolerant wheat – In 1997, research funded by AAFC, provinces, producer groups and industry began to develop midge tolerant wheat varieties. By 2010, the first blend was made commercially available. Since then, uptake has expanded rapidly. Benefits include: reduced yield losses; improved flexibility in crop rotation and seeding dates; lower use of insecticides; and higher producer revenues. Investments of $12.2 million over more than twenty years are expected to result in benefits of $456 million by 2025 (benefit/cost ratio of 37:1).
  • Swath grazing – Funding from AAFC, provinces and industry has been directed towards swath grazing research since 1995. Increases in the adoption of swath grazing have resulted in reduced labour, machinery and fuel expenses. Investments of $7.7 million over more than 20 years are expected to result in benefits of approximately $1.3 billion by 2025 (benefit/cost ratio of 170:1).

4.2.6 Economic impacts

Program activities have resulted in significant economic impacts. An economic impact assessment conducted as part of the evaluation found that AAFC funding is estimated to have facilitated $289.9 million in spending across the Canadian economy from 2013-14 to 2015-16 (Figure 9). The total facilitative economic impact represents the combined AAFC investment, leveraged funding from private sector and other government sources, and indirect output impacts (that is, the impact on sectors that supply goods and services to the sectors directly impacted by the R&D activities). When facilitative impacts are compared to the AAFC investment of $120.2 million during this time period, there is a ratio of $2.4 in total facilitative economic impacts generated for every dollar of AAFC investment.

Figure 9: Total Facilitative Economic Impact from Stream B Industry-led R&D Expenditures, 2013-14 to 2015-16 (Estimated)
Description of this image follows.
Description of above image

The table below illustrates Figure 9 as follows:

AAFC funding 120.2 million
Leveraged funding 74.5 million
Indirect economic impact 95.2 million
Total facilitative economic impact over 3 years 289.9 million

The attributable economic impacts represent the portion of economic impacts which are attributable to the program (that is, would not have occurred in the absence of AAFC financial assistance). Based on interviews with Agri-science cluster leads, approximately 80% of the total facilitated economic impact is estimated to be directly attributable to AAFC financial assistance.Footnote 37 Table 11 outlines what this spending means in terms of Gross Domestic Product, wages and salaries, and jobs from 2013-14 to 2015-16. As an illustration, of the $175.6 million in Gross Domestic Product expenditures that were facilitated by Stream B activities from 2013-14 to 2015-16, $140.5 million in Gross Domestic Product expenditures would not have occurred in the absence of AAFC financial assistance.

Table 11: Total economic impacts facilitated by and attributable to Stream B industry-led R&D expenditures, 2013-14 to 2015-16
Economic impacts Facilitative economic impacts Attributable economic impacts
Spending (millions) $289.9 $232.0
Gross Domestic Product (millions) $175.6 $140.5
Wages and Salaries (millions) $106.4 $85.1
Average annual employment (# of jobs[1]) 570 456

[1] Includes full-time, part-time jobs, and temporary jobs.

Source: Economic Impact Assessment

4.3 Performance – Efficiency and economy

4.3.1 Effectiveness of Industry-led R&D Stream B Design and DeliveryFootnote 38

Contributing factors to effective program design and delivery

The major factors contributing to the effective delivery of Stream B activities are the program funding mechanisms, which facilitate collaboration across various R&D funders and performers and encourage industry to support longer-term innovation activities. In addition, the continuous improvement approach employed by the program has led to increasingly efficient delivery over time.

Collaboration across R&D funders and performers

The funding mechanisms for the Industry-led R&D component of Stream B facilitate collaboration across R&D performers by encouraging researchers to work collaboratively across different regions and types of organizations (for example, industry, government, and universities). Almost 90% of all Industry-led R&D activities involved one or more collaborator type, most frequently involving AAFC research centres and post-secondary institutions. Research collaboration generates a variety of benefits, such as improved R&D capacity in areas that are relevant to agriculture and agri-food industries and strengthened research networks, which can leverage further investments for follow-on projects, leading to improved research excellence.Footnote 39 Collaboration leads to improved research results by bringing together different perspectives, experience, skills and knowledge and fostering cross-disciplinary interactions. Collaboration encourages skills and knowledge transfer, develops the new knowledge systems and social relationships required to maximize value and stimulate creativity. Collaboration with different research institutions leverages research investments, provides access to different sources of highly qualified personnel, and makes more efficient use of available resources.

Agri-science clusters and projects bring together provincial industry organizations to coordinate research funding at a national level. National Agri-science clusters have allowed agriculture and agri-food industry organizations to fund R&D projects across Canada that support common goals and priorities. This coordination optimizes the use of limited industry funding as it reduces the level of duplication in funding. Industry stakeholders also benefit from knowledge transfer across a broader range of projects.

The evaluation found some evidence that opportunities or mechanisms could be explored to facilitate increased collaboration between AAFC and other federal and provincial government organizations. According to Agri-science cluster lead interviewees, in some instances, federal and provincial government funding rules related to stacking and the use of Growing Forward 2 funding constrained collaboration. For instance, it was reported that an Agri-science cluster activity involving National Research Council of Canada and AAFC researchers did not move forward because funding could not flow from one federal department to another. A study of collaboration among thirteen federal government departments found that three of the four most productive organizations (National Research Council Canada, AAFC, and Natural Resources Canada) are also among the least engaged in inter-departmental collaborations.Footnote 40

Cost-shared funding

Cost-shared funding mechanisms, which provide from 50% to 75% in AAFC matching funding for industry-led R&D, encourage industry stakeholders to fund larger scale and longer-term R&D activities. According to Agri-science cluster lead interviewees, in the absence of AAFC financial assistance, it is likely that industry stakeholders would continue to fund some provincial productivity-related research, but it is much less likely that they would fund national R&D related to market expansion. R&D activities that focus only on productivity improvements may not be sufficient to maintain the competitiveness of the sector due to its vulnerability to external threats such as shifts in input and output prices. Longer-term R&D activities tend to involve market expansion activities such as developing new uses for agricultural products or developing value-added products, which require long-term investments in R&D but have the potential to yield significant long-term returns to the Canadian agriculture and agri-food sector by capitalizing on expanding Canadian and international demand for food and agri-based products with particular attributes.Footnote 41

Continuous program improvement

The ongoing efforts to refine the program under Growing Forward 2 have improved the efficiency of Stream B delivery processes. AAFC senior management seek feedback from funding recipients through annual meetings, regular communications, and in-person visits. In October 2016, a consultation session was held to discuss program design and delivery and included representatives of all 14 Agri-science clusters, five universities, AAFC representatives across 5 Branches, and the Treasury Board Secretariat. AAFC senior managers have responded to program delivery challenges reported through consultations by undertaking leanFootnote 42 exercises to enhance the efficiency of application and claims processes. Various refinements have been made to program delivery such as identifying the optimal number of project files per program officer (about 6 to 8 files per officer), moving from a staggered to a continuous intake of Agri-science project applications (though this process presents some challenges in terms of ensuring adequate resource allocation), accepting electronic signatures, and accepting a per diem allocation for travel expenditures as part of claims processing.

Stakeholder engagement and continuous improvement activities have led to improved performance against service standards and client satisfaction. The percentage of approval/rejection letters being sent within 100 business days of full receipt of complete applicant packages increased from 11% in Q3 2013-14 to 100% in Q4 2015-16, though it is important to note that 236 applications for clusters and projects were received in 2013-14, while 25 applications were received for projects in 2015-16. Recent reports indicate that service standards are being exceeded, which could warrant a review of the current targets (for example, 97% of transactions met the standard of payment sent within 30 days of receipt of a completed claim in Q2 2016-17, above the target of 80%). The Agri-science cluster leads interviewed were satisfied with most aspects of the program design and delivery, with many indicating that the processes have improved over time.

Design and delivery constraints

The main challenges associated with Stream B program design and delivery include the complex application process, delays in processing applications, requirements for claims processing, and limited ability to respond to emerging needs and sectors.

Complex application process and delays in processing applications

Though improvements have been made over time, the application and approval process for Industry-led R&D activities remains complex and time consuming. Agri-science cluster lead interviewees were least satisfied with the application process compared to other program design and delivery elements. Various factors contribute to the administrative complexity of the application process such as the number of collaborators involved and background documents required (for example, letters of approval from the managers of every proposed university collaborator) and the requirement for separate funding agreements for Vote 1 and Vote 10 elements.

Some steps in the review process are duplicative. For example, proposed Agri-science cluster activities are required to undergo technical peer reviews both by Agri-science cluster leads prior to the application (for example, by industry-led science advisory committees) as well as by AAFC scientists during the application review process. In some cases, industry-led science advisory committees are comprised of world leading scientific experts and technical advisors who review potential activities before they are approved by the Agri-science cluster lead organization to be included as part of the application to AAFC. Once the application is submitted, each of the activities undergoes subsequent reviews by AAFC scientists. Agri-science clusters include, on average, 35 activities, and each of these activities needs to be reviewed by different AAFC scientists. Some activities require more than one reviewer if they involve different areas of scientific expertise (for example, climate change and genomics). According to AAFC program delivery representatives, part of the AAFC internal review challenge is finding an AAFC scientist to review a particular activity who is both knowledgeable of the subject matter and not involved in the proposed research activity. Screening mechanisms could be refined so that fewer Agri-science clusters or projects are required to go through the full technical review process.

The complexity of the application process combined with the unexpectedly high number of applications at the outset of the program (292 Agri-science projects and 23 Agri-science cluster applications by Q3 2013-14) led to delays resulting in less than half of the available budget authority being spent in the first year of the program. AAFC representatives reported that the major factors contributing to delays included insufficient staff resources to manage high program demand, changes in program leveraging (during the second intake in October 2013, applicants were asked to re-submit with 50% cost matching ratios), and requirements for program applicants to provide audited financial statements. Some not-for-profit organizations withdrew their applications because the latter requirement was too costly. The application process limited the extent to which applicants could communicate with AAFC and modify proposals based on feedback from AAFC. Due to the amount of time that lapsed between completion of similar programming in Growing Forward and the commencement of Stream B program activities in Growing Forward 2, it was difficult for Agri-science clusters and projects to maintain research capacity since some specialists funded by the program moved on to other projects.

Requirements for claims processing

The level of documentation required for processing claims, including a recipient cover letter, financial update, and transaction listing is a burden to both recipients and research collaborators. Some recipients have required a level of detail from their partners beyond what is required for AAFC to process claims, which can strain the R&D collaborations facilitated by the program. AAFC has raised this issue with clusters recipients, to ensure that partners are no longer asked to provide more detail than is required by the program. Some Agri-science cluster lead interviewees expressed concern that documentation requirements could deter potential research collaborators from participating in Agri-science clusters and could impact the program's ability to leverage agricultural R&D highly qualified personnel and infrastructure capacity. A few AAFC representatives noted that the program's consistency in interpreting program guidelines with respect to the eligibility of expenses, sampling processes, and acceptable tolerance levels for discrepancies could be improved. These issues have been examined as part of the continuous improvement activities of the Innovation Programs Directorate, including through the establishment of a departmental, Director-level claims working group that reviews claims processes across a variety of AAFC programs in order to streamline processes and improve consistency.

Limited ability to respond to emerging needs and sectors

Several Agri-science cluster lead interviewees indicated their belief that the program does not allow sufficient flexibility to respond to emerging priorities or opportunities identified by private sector partners, since most activities are determined at the beginning of the Agri-science cluster or project and are relatively fixed. Reallocating funds between activities or towards new activities can take six months to one year. A few Agri-science cluster leads indicated that part of the challenge is due to the Vote 1 and Vote 10 funding structure which requires two separate funding agreements (collaborative research and development agreements and contribution agreements) and requires increased administration in terms of making amendments. AAFC representatives noted that the lack of flexibility may be a matter of perception among recipients about what is possible (that is, reallocating funding between activities) and what is not possible (that is, reallocating funding between years) within the program and that greater education of program guidelines could resolve misperceptions. It could also be possible to use the Agri-science projects rather than the clusters to address emerging issues.

According to Agri-science cluster lead interviewees and AAFC program delivery representatives, the program also does not adequately serve the needs of emerging agricultural sectors (for example, bioproducts), which tend to be more reliant on individual private sector partners and do not have access to industry check-off resources to fund sector-wide R&D activities. Larger, more established sectors are better equipped to raise industry funds and more likely to receive public funding due to the quality of their applications. Smaller, emerging sectors struggle to coordinate funding and have limited capacity to administer national research programs such as those administered through the Agri-science clusters.

4.3.2 Economy and efficiency

Most of the available budget authority for the first three years of the Industry-led R&D Stream B program has been spent and program spending has increased over time.Footnote 43 A total of $120.2 million in AAFC funding was spent on Stream B Industry-led R&D activities from 2013-14 to 2015-16, representing 80% of the available budget authority during this period (Table 12). Program spending as a percentage of the available budget authority increased from 49% in 2013-14 to 93% in 2015-16. AAFC program delivery representatives indicated that factors contributing to under-spending in 2013-14 were the large backlog in applications and subsequent delays in processing those applications. Program spending against authorities improved in following years as a result of continuous improvement activities implemented to address the backlog.

Table 12: Stream B Industry-led R&D Expenditures and Available Budget Authority
R&D expenditures and available budget authority 2013-14 2014-15 2015-16 Total
Available Budget Authority (millions) $45.1 $47.7 $57.5 $150.4
Actual Expenditures (millions) $22.1 $44.6 $53.5 $120.2
% of Budget Expended 49% 93% 93% 80%
Source: AAFC databases

To examine the efficiency of the delivery of Stream B activities, efficiency ratios were developed that compare program outcomes with program costs and commitments. These efficiency ratios were compared with overall program results for similar programs implemented during Growing Forward. Stream B, under Growing Forward 2, has achieved program outcomes more efficiently than similar programs implemented under Growing Forward (Table 13). AAFC commitments under Growing Forward 2 have leveraged $0.45 in private sector investment per $1 of AAFC investment, compared to $0.26 per $1 of AAFC investment under Growing Forward. Fewer program expenditures were required per innovation developed compared to previous programs ($138,161 versus $316,667).

Table 13: Efficiency in Achieving Program Outcomes: Comparison between Growing Forward and Industry-led R&D Stream B AgriInnovation Programming
Indicator Growing Forward results Growing Forward 2 results as of March 31, 2016
Private Sector Investment Committed $44.7 million $97.2 million
AAFC Funding Committed $171.0 million $217.0 million
Leverage for every $1 AAFC Committed $0.26 $0.45
Number of Innovations Developed 540 870
AAFC Funding Expended $171.0 million $120.2 million
AAFC Cost Per Innovation $316,667 $138,161
Source: AAFC databases and AIP Stream B Program Performance Measurement and Risk Management Strategy

4.3.3 Performance measurement and reporting systems

Several mechanisms are in place to track the performance of Stream B program activities. Agri-science cluster and project recipients are required to prepare annual performance reports that describe progress against targeted milestones and outcomes such as the number of new or improved products, peer reviewed publications, and individuals attending information events. The final performance reports submitted following project/cluster completion capture additional information on the commercial potential, commercialization, and adoption of the new or improved innovations. The AAFC Programs Branch tracks success stories and commercialization and adoption impacts in an achievement database spanning multiple policy frameworks. These systems have been successful in collecting a significant amount of data and information on program outputs, and identifying potential case studies and success stories. Nevertheless, some limitations were identified with respect to the performance measurement data, such as challenges aggregating data due to differences in the interpretation and categorization of innovation impacts by recipients.

There are opportunities to improve the utility of performance data in order to facilitate communication of program results across a broader range of stakeholders and enhance knowledge transfer and adoption among industry stakeholders. Different types of indicators that capture the significance of impacts and help tell success stories in a manner that is accessible and understandable could be considered for inclusion in the program's performance measurement strategy. Other indicators that measure the long-term returns of innovation program investments could also be considered. While further analysis is required to assess their applicability, advantages and disadvantages, some potential measures of program impacts are as follows:

5.0 Evaluation conclusions

5.1 Conclusions

The key conclusions resulting from the evaluation are as follows:

Relevance

There is a continued need for Stream B programming to develop innovations to address threats to the profitability and competitiveness of the agriculture and agri-food sector. The sector faces numerous threats such as variable input and output prices, extreme weather, and new pests and diseases. There is also a need to capitalize on new opportunities for value-added products due to increasing global demand for food with specific attributes and non-food products. Investments in agricultural R&D contribute to improved productivity and generate a substantial return on investment in the long-term.

Limited funding, weak linkages between agricultural R&D performers, and capacity challenges constrain the sector's ability to meet its innovation needs. The federal government continues to play a lead role in funding and maintaining the capacity to undertake agricultural R&D and knowledge transfer to complement and leverage limited investment, foster innovation in small and emerging agricultural industries and encourage industry to invest in research with long-term public good benefits (for example, soil, water, and air quality; biodiversity). There is an ongoing need for collaboration among governments, universities, industry and other organizations in conducting agricultural R&D in Canada. R&D collaboration has many benefits such as leveraging investment, enhancing access to highly qualified personnel, infrastructure and equipment, and generating research excellence. AAFC collaboration rates are increasing at a higher rate than other federal government departments involved in scientific research. This increase coincides with the introduction of the cluster funding model under Growing Forward and its continued support under Growing Forward 2 through Stream B, which was designed to support collaboration.

Program-supported R&D activities can also provide opportunities for highly qualified personnel in the agriculture and agri-food sector to gain research experience, which in turn, supports the sector's future scientific and innovation capacity. This addresses several challenges that may constrain the ability to meet future highly qualified personnel needs, such as the high number of expected scientist retirements across the sector, limited networks and mentorship opportunities for newer scientists, declining enrolment rates in agricultural programs, and competition from other sectors.

Performance (effectiveness)

Three years into the five-year program, Stream B has made significant progress towards the achievement of its intended outputs and outcomes. As of March 31, 2016, Stream B has committed $220.5 million over the five-year period of 2013-14 to 2017-18 to 14 Agri-science clusters, 86 Agri-science projects and 20 AAFC-led Knowledge Transfer projects. In total, 60 Collaborative Research and Development Agreements have been signed, allowing for support in the form of collaborative assistance from AAFC research scientists and experts in Industry-led R&D activities. Program activities span a variety of sectors and portfolios, with the largest proportion of Industry-led R&D funding committed to the oilseeds sector portfolio and the largest proportion of AAFC-led Knowledge Transfer funding committed to agro-ecosystem productivity and health.

Private sector investments in Stream B activities have more than doubled compared to private sector investments in similar programs under Growing Forward, due to increased AAFC investments, modified cost-shared ratios, and increased industry awareness and interest in the program. The level of R&D collaboration has increased. New research networks have been established and existing networks have been strengthened and expanded across provinces and different types of organizations. Though most Agri-science cluster and project activities were not complete at the time of the evaluation, considerable progress had been made in developing and identifying new products, processes, and technologies for industry commercialization or adoption. Several examples of innovations resulting from Stream B activities were identified, some of which arose from activities that were initiated under previous programming and tracked over time. All technology, knowledge and information transfer targets have been exceeded. The program has enhanced R&D capacity in the agriculture and agri-food sector by providing research experience for individuals completing an MSc or PhD.

The program has made some progress towards its end outcome that the sector produces, adopts, and commercializes innovative products, processes, practices, services and technologies. It is too early to assess the full impact of Stream B activities since the program is not yet complete, most projects and all clusters are underway and have not yet submitted final performance reports, and it can take several years to develop, adopt and commercialize products and practices resulting from Stream B supported R&D. A cost-benefit analysis conducted by Programs Branch in 2015-16 of two successful innovations identified high rates of return from long-term innovation investments by AAFC, provincial governments and industry in agricultural R&D activities across policy frameworks.

Program activities have resulted in significant economic impacts. Industry-led R&D expenditures of $120.2 million from April 1, 2013 to March 31, 2016 resulted in an estimated $289.9 million in spending across the Canadian economy during the same time period, or $2.4 in total facilitative economic impacts for every dollar of AAFC funding. According to interviews, approximately 80%, or $232.0 million, of the total facilitated economic impact is estimated to be directly attributable to the Industry-led R&D component of Stream B (that is, it would not have occurred in the absence of AAFC assistance).

Performance (efficiency and economy)

The major factors contributing to the effective delivery of Stream B activities include collaboration across R&D funders and performers, cost-shared funding mechanisms, and continuous improvement of program delivery based on lessons learned. Industry-led R&D funding mechanisms facilitate collaboration across a variety of R&D funders and performers, which contributes to improved capacity and research excellence and facilitates more coordinated and efficient R&D funding. The evaluation found some evidence that opportunities or mechanisms could be explored to facilitate increased collaboration between AAFC and other federal and provincial government organizations. AAFC's matching funding encourages industry stakeholders to fund larger scale and longer-term R&D activities that have the potential to yield substantial returns. The ongoing efforts to refine program delivery have improved the efficiency of delivery processes over time. The consistency in the type of programming offered under Growing Forward and Growing Forward 2 has also facilitated the application of lessons learned.

The main challenges that have constrained the effectiveness and efficiency of program design and delivery include the complex application and claims reporting processes and limited ability to respond to emerging R&D needs and sectors. Application review and approval processes are complex and time consuming due to the various collaborators involved and types of agreements required, among other requirements. The time it takes to complete these processes risks delaying the start of Agri-science projects and clusters which may result in highly qualified personnel and other research capacity lost to other R&D work. Some recipients have required a level of detail from their partners beyond what is required for AAFC to process claims, which can strain the R&D collaborations facilitated by the program. The program does not allow sufficient flexibility to respond to emerging R&D needs, because it is difficult to reallocate funds between activities or towards new activities in a timely manner. Smaller, emerging sectors struggle to access Agri-science cluster and project funding due to capacity constraints.

Program spending as a percentage of the available budget authority increased from 49% in 2013-14 to 93% in 2015-16. The primary factors contributing to under-spending in 2013-14 were a large backlog in applications and delays in processing these applications at the outset of Growing Forward 2.

Stream B has achieved program outcomes more efficiently than similar programming implemented under Growing Forward in terms of higher levels of private sector investment leveraged per dollar of AAFC investment and lower AAFC costs per innovation developed.

Several mechanisms are in place to track the performance of Stream B program activities. These systems have been successful in collecting a significant amount of data and information on program outputs. However, there are limitations with respect to the consistency and accuracy of the performance data. For example, the clarity of definitions and categories could be improved. There are also opportunities to improve the utility of the performance data in order to increase its use among industry and AAFC stakeholders.

6.0 Issues and recommendations

Annex A: Program logic model, governance, and stakeholders

The following logic model describes the linkages between program activities, outputs and intended outcomes.

Figure 1: Logic Model for Stream B: Research, Development and Knowledge Transfer

Industry-led R&D

AAFC-led Knowledge Transfer

Program Governance

Stream B is the shared responsibility of the ADM, Programs Branch and the ADM, Science and Technology Branch. Application intake, assessment, and performance monitoring for the Industry-led R&D component, and fiscal management of Vote 10 funds, is administered by the Innovation Programs Directorate, Programs Branch. When an applicant requests that AAFC undertake certain activities on its behalf and the activities are approved, the fiscal management of Vote 1 funds is performed by Science and Technology Quality Assurance and Programs Division, Science and Technology Branch. Applications under this component undergo a risk assessment and technical reviews by the Market and Industry Services Branch and Science and Technology Branch, and a Project Recommendation Form is completed and approved by a Program Manager and the Director of the Industry Development Division of the Innovation Programs Directorate, applications are brought to the Director General Innovation Committee. This Committee consists of DGs from across AAFC and is co-chaired by the DG Partnerships and Planning Directorate, Science and Technology Branch and DG Innovation Programs Directorate, Programs Branch. Projects recommended for approval by the DG Innovation Committee are signed off at the appropriate level according to departmental delegations. Sign-off is sought from the responsible Director, DG and ADM. The Deputy Minister is responsible for recommending projects to the Minister.

The AAFC-led Knowledge Transfer component of Stream B is managed by the Science and Technology Quality Assurance and Programs Division. Applicants under the AAFC-led Knowledge Transfer component of Stream B submit a letter of intent, followed by a full proposal. Recommendations are reviewed by the Science and Technology Branch Portfolio Committee and the Science and Technology Branch Management Committee, and brought for decision to the Science and Technology Branch Executive Committee.

Overall reporting through the annual Departmental Performance Report is coordinated by the Program Design and Performance Division, Innovation Programs Directorate, Programs Branch.

Stakeholders

Under the Industry-led R&D component, internal stakeholders include AIP personnel who: conduct technical reviews of Industry-led R&D applications and reports; develop Collaborative Research and Development Agreements; and conduct Industry-led R&D activities as outlined in Collaborative Research and Development Agreements, and AAFC Programs Branch staff who administer the Industry-led R&D component. Under the AAFC-led Knowledge Transfer component, internal stakeholders include AAFC science, technology, and knowledge transfer personnel who undertake the Knowledge Transfer projects and AIP staff who administer the Knowledge Transfer component.

For both the Industry-led R&D and AAFC-led Knowledge Transfer components, external stakeholders include industry organizations, private sector companies, universities, federal government organizations (for example, National Research Council of Canada), provincial government organizations, and independent research organizations.

Annex B: Evaluation methodology

This Annex provides more detailed descriptions of the evaluation questions and issues and each of the specific lines of evidence.

Evaluation Questions – Relevance - Continue Need for the Program

  1. What is the state of innovation capacity within AAFC and in Canada?
    1. What are the baseline and projected resources available to drive research and innovation in the sector, including highly qualified personnel, infrastructure, and investment (and what are the sources)?
    2. What are the baseline and projected levels and quality of research collaboration across levels of government, academia, and industry?
  2. What are the baseline and projected productivity levels in primary agriculture? How do investments in research and innovation influence productivity?
  3. How does AAFC and Canada compare to similar countries with respect to agricultural innovation capacity, collaboration, and productivity?

Evaluation Questions – Performance - Effectiveness

  1. What has the program achieved in terms of activities and outputs, compared to baselines and against program targets?
  2. What has the program achieved in terms of levels and quality of research collaborations across levels of government, academia, and industry?
  3. To what extent has research funding been leveraged from industry?
  4. What innovations have been developed as a result of work supported by program funding?
  5. To what extent has technology, knowledge, and information been transferred and adopted?
  6. What has been achieved with respect to improved sector innovation capacity and productivity?
  7. How is investment in innovation and research expected to result in direct and indirect economic impacts? What proportion of the impacts are attributable to program investments?

Evaluation Questions – Demonstration of Efficiency and Economy

  1. How efficiently and economically is the program being delivered?

1) File and Operational Data Review

An administrative file review was undertaken to address evaluation issues related to program performance. The methodology included a review of internal data including an analysis of: federal, provincial, and territorial program investments prepared by AAFC's Innovation and Growth Policy Division, Strategic Policy Branch; financial data on planned program spending and expenditures; Industry-led R&D results data, through the Innovation Programs Directorate's Access tracking system and Achievements database; the 2015-2016 AAFC Science and Technology Branch Knowledge Transfer Program Report; and detailed Agri-science cluster and project files; as well as operational efficiency and service standards data.

2) Document Review

An extensive document review was undertaken to provide evidence with respect to the continued need for the program and the performance of the program. The document review was a critical source of information regarding the state of agriculture and agri-food R&D activities in Canada, drawing from existing research and analysis such as:

The document review also provided evidence with respect to program activities and impacts using program foundational documents (for example, Stream B Program Performance Measurement and Risk Management Strategy and Applicant Guide); program reports and analysis (for example, Innovation Investment Analysis presentation to Federal, Provincial, and Territorial Innovation Working Group, February 2016; and Science and Technology Branch highly qualified personnel data and analysis); and success stories (for example, AAFC's Cost-Benefit Analysis of Two Successful Innovation Cases).Footnote 50

3) Interviews

A total of 17 interviews with agri-science cluster leads and AAFC representatives were conducted in September and October 2016 to obtain input regarding the relevance and performance of program activities (Table 4).

Target Group Number Completed Description of the Target Group
Agri-science Cluster Leads 10 10 representatives of 6 Agri-science cluster organizations were interviewed. The roles of representatives interviewed included executive directors, advisors, managers, and science directors. Their responsibilities consisted of overseeing sector strategy development, setting-up and managing the Agri-science cluster, reviewing and recommending project proposals, and reporting on Agri-science cluster results to industry stakeholders. The Agri-science clusters represented a mix of agricultural commodities and products.
AAFC representatives 7 Representatives included AAFC Programs Branch directors, managers and program officers involved in the delivery of Stream B.
Total 17 Data Not Available

4) Benchmarking Analysis

A comparative analysis of the AIP with similar programs in other jurisdictions including the United States and Australia was undertaken to obtain evidence to assess program relevance. The analysis also utilized key agricultural innovation and capacity benchmark data to compare the performance of similar programs in different jurisdictions, and identify potential weaknesses, issues or risks in the Canadian agricultural innovation system. The analysis involved an extensive review of existing documents, literature and reports such as An Overview of the Canadian Agriculture and Agri-Food System (AAFC, 2016), Organization for Economic Co-operation and Development studies of agricultural innovation systems in different countries, and benchmarking analysis and literature reviews conducted as part of recent evaluations of other AAFC science-related programming. A review of additional websites, data, and literature was conducted to supplement existing documents and research.

5) Economic Impact Assessment

The economic impact assessment estimated the impacts on the Canadian economy brought on by Industry-led R&D activities (Agri-science clusters and projects) funded through the AIP Program Stream B from 2013-14 to 2015-16 fiscal years, based on annual expenditures incurred. It provides information on the amount and nature of spending generated by AAFC spending on Stream B.

Beyond the direct expenditures, economic impact models can be utilized to analyze additional benefits to the local economy in the form of businesses providing goods and services to entities where direct expenditures occur. In addition, as a result of increased local household income, there may be further increases in overall expenditure. The latter is considered as a spin-off impact, which can also be captured by economic impact models. The three types of impacts are described briefly here:

It is generally acknowledged that direct impacts alone are incomplete and the total impact may sometimes overestimate the impacts of initial spending. In this study, we presented direct, indirect, and combined facilitative impact values.

Analytical Framework

To capture the overall impact of program spending on the Canadian economy, expenditures were applied to the Statistics Canada's national input-output model. The model provides estimates of the overall impact on output, Gross Domestic Product, employment and other variables in the Canadian economy.

The input-output model is as follows: when a person spends money on a product (goods and/or services), that amount of expense creates a direct requirement for the production of that product. The economic impact, however, does not end there. The increased production of this product leads to increased production of all the intermediate goods and services that are used to make this product, and the increased production of intermediate goods and services will in turn generate more demand for other goods and services that are used to produce these intermediate products. As demand rises, workers are able to earn a higher wage, and they sometimes decide to spend a portion of their extra earnings to purchase more goods and services. As such, an initial demand for a product creates a chain effect down the production process. An economic impact assessment is designed to study such inter-linkage between industries in order to evaluate how a change in an initial demand for goods or services contributes to changes in other industries' levels of production and overall economic activity level within a region.

The input-output model is based on the input-output structure of the Canadian economy, which is essentially a set of tables describing the flow of goods and services amongst the various sectors of the economy. Such a model is useful in determining how much additional production is generated by a change in the demand for one or more products or by a change in the output of an industry. Currently, Statistics Canada uses the 2010 national input-output model to estimate the economic impact, and the results are used for comparative analysis purposes. It should be noted that employment impact estimates from this model are based on the 2010 total compensation per job. As such, it is necessary to deflate the program expenditures incurred in 2013, 2014 and 2015 to 2010 dollars to get a more accurate estimate of the employment impact.

The attributable economic impacts represent the portion of economic impacts which are attributable to the program (that is, would not have occurred in the absence of AAFC assistance). Attributable impacts were calculated using data obtained from interviews with Agri-science cluster leads with respect to the likelihood the project would have taken place in the absence of AAFC assistance. The incrementality factor represents the average Agri-science cluster lead perception of the likelihood that the cluster activities would either not have proceeded, or would have proceeded but with a narrower scope or slower pace in the absence of AAFC assistance.

Analysis Steps

The steps that were undertaken for this analysis are as follows:

  1. Developed a database of total planned project expenditures by funding source from Contribution Agreements and Collaborative Research and Development Agreements for the majority of Stream B Agri-science clusters and projects.
  2. Developed estimates for total project expenditures using both the actual AAFC expenditures and planned project expenditures. The planned project expenditure data was used to develop the percentage breakdown of funding sources (AAFC, other government, and industry) for each year. These percentages were applied to validated actual AAFC expenditures for each respective year to develop an estimate for the total expenditures per year.
  3. Developed estimates of the percentage of expenditures (based on actual Vote 1 and Vote 10 expenditures) taking place within AAFC research centres and external to AAFC (for example, universities or private sector R&D facilities).
  4. Determined the percentage breakdown in terms of major cost categories. Detailed project budget roll-ups were reviewed from a representative sample of 37 Agri-science projects and 5 Agri-science clusters to determine the estimated overall project cost breakdown in terms of major costs categories including administration, salaries/benefits, contracted services, travel, capital/assets, and other direct project costs. A separate breakdown was identified for externally led and AAFC-led activities.

    Within each major cost category further sub-categories and percentages were identified in order to match the data with an appropriate industry code. Estimates were developed by coding and tabulating detailed cost descriptions for a sub-sample of project budgets. Investment analysis data on collaboration was also reviewed to develop an estimate for the proportion of external research that likely took place at a university versus a private sector or independent R&D lab. North American Industry Classification System based Input-Output Industry Classification codes were assigned to each expenditure sub-category. Further assumptions were made as to the likely percentage of the expenditure that is attributable to Canadian industries (for example, only 10% of computer purchase expenditures were included since most computer manufacturing takes place outside of Canada).

  5. Developed estimates for the direct and indirect economic impacts (for example, output, Gross Domestic Product, employment, and wages and salaries) using the Statistics Canada input-output multipliers and simulations.
  6. Estimated the total facilitative economic impacts. The total facilitative economic impact represents the combined AAFC investment, leveraged funding from private sector and other government sources, and indirect output impacts.
  7. Incorporated the incrementality factor of 80% to calculate total attributable economic impacts based on average Agri-science cluster lead interview estimates that there is a 20% likelihood that projects would have proceeded as planned in the absence of AAFC assistance.Footnote 51

Annex C: References

Report a problem on this page
Please select all that apply:
Date modified: