Archived - Evaluation of the Canadian Food Inspection Agency's Enhanced Feed Ban Program

February 8, 2013

Table of Contents

• List of Abbreviations
• Executive Summary
• 1.0 Introduction
  • 1.1 Program Overview
  • 1.2 Evaluation Objectives and Scope
  • 1.3 Quantitative Method
  • 1.4 Qualitative Methods
  • 1.5 Stakeholders Interviews
  • 1.6 Analysis and Integration of Data
  • 1.7 Challenges, Limitations and Mitigation Strategies
• 2.0 Findings
  • 2.1 Relevance: Continued Need for Program (R1)
  • 2.2 Relevance: Alignment with Government Priorities (R2)
  • 2.3 Performance: Achievement of Expected Outcomes (P1)
  • 2.4 Performance: Demonstration of Efficiency and Economy (P2)

Tables and Figures

• Table 1 - Financial resources received and spent for the CFIA's EFB and variance between planned budget and actuals ($'000s)
• Table 2 - Definitions of terms used to indicate the proportion of interviewees
• Table 3 - Overview of evaluation challenges, limitations and mitigation strategies
• Table 4 - Number of BSE cases confirmed in Canada, 2003-2011
• Figure A - Changes in the attitude towards the EFB over time as reflected in articles related to the EFB published by select industry and market sources, by year and tone, 2004-2010
• Figure B - Changes in the attitude towards the EFB over time as reflected in articles related to the EFB published by general media source, by year and tone, 2004-2010

List of Abbreviations

AAFC

Agriculture and Agri-Food Canada

AEB

Audit and Evaluation Branch

AMP

Administrative Monetary Penalties

APRI

Alberta Prion Research Institute

BSE

Bovine Spongiform Encephalopathy

BVCRT

Beef Value Chain Roundtable

CFIA

Canadian Food Inspection Agency

CVS

Compliance Verification System

DPR

Departmental Performance Reports

EAC

Evaluation Advisory Committee

EFB

Enhanced Feed Ban

EOR/PMRS

Enterprise Operational Reporting/Performance Management Reporting Solution

FBTF

Feed Ban Task Force

FDA

Food and Drug Administration

FTE

Full-time Equivalent

GATT

General Agreement on Tariffs and Trade

MCAP

Multi-Commodity Activities Program

MRAP

Management Response and Action Plan

MRRS

Management, Resources and Results Structure

OFGD

Other Federal Government Departments

OIE

World Organisation for Animal Health

OPM

Operations Planning Module

PAA

Program Alignment Architecture (formerly Program Activity Architecture)

PHAC

Public Health Agency of Canada

PWGSC

Public Works and Government Services Canada

QMS

Quality Management System

SRM

Specified Risk Material

TB

Treasury Board

TBS

Treasury Board Secretariat

TSE

Transmissible Spongiform Encephalopathy

UK

United Kingdom

US

United States

USDA

United States Department of Agriculture

WHO

World Health Organization

WTO

World Trade Organization

Executive Summary

Program Background

The Enhanced Feed Ban (EFB) is a part of the Government of Canada's Bovine Spongiform Encephalopathy (BSE) Program which is a horizontal initiative led by the Canadian Food Inspection Agency (CFIA). The main objective of the EFB is to accelerate Canada's progress in BSE management by preventing more than 99% of potential infectivity from entering the feed system as well as to enhance risk management of transmission of BSE in the cattle herd. To this end, the EFB aims to:

• Strengthen animal feed restrictions through amendments to the relevant regulations;
• Ensure compliance with control measures around prohibited materials and specified risk material (SRM) removal^Footnote 1; and
• Increase the level of verification and confidence that SRM is segregated from feed, fertilizer and pet food and that prohibited materials are not fed to ruminants.

The removal of SRM from animal feed is an important animal health protection measure and an indirect public health protection measure. In the short-term, the most effective measure to protect public health from BSE is the removal of SRM from food and other consumer products, whereas over the long-term, accelerating the eradication of BSE from the animal population through the EFB will further ensure the protection of both animal and public health.^Footnote 2 In addition, the EFB is also intended to help maintain consumer confidence and regain and expand market access for the regulated products by ensuring Canada's "controlled" BSE risk status from the OIE.^Footnote 3

Multiple branches and programs within the CFIA are involved in the design, implementation and delivery of the EFB.

Evaluation Overview

In accordance with the Treasury Board Secretariat (TBS) Policy on Evaluation, the primary objective of this evaluation was to assess the relevance (continued need, alignment with government priorities, and alignment with federal government roles and responsibilities) and performance (achievement of expected outcomes, as well as demonstration of efficiency and economy) of the EFB and provide recommendations to improve program effectiveness and efficiency, as necessary.

The evaluation team was composed of evaluators from an independent program evaluation firm, and the Audit and Evaluation Branch (AEB) of the CFIA. The evaluation approach and methodology were developed in consultation with the CFIA Evaluation Advisory Committee (EAC) and an independent scientific expert.

Note that documents and files served as the key sources for detailed information on the development and implementation phases, while interviews were often the primary source of data for evaluating the adequacy and effectiveness of EFB activities and procedures during the post-implementation phase. In addition, the available quantitative and performance data on EFB-specific activities and outcomes often had significant limitations and limited the ability to conclusively determine the effectiveness and efficiency of most of the EFB activities in the post implementation phase.

Findings – Relevance

Continued need for the Program

• The circumstances and factors that prompted the introduction of the EFB have not changed significantly and warrant continuation of the EFB.
• There are ongoing efforts led by the CFIA to identify potential modifications to the design and delivery of the EFB that would better address the needs of CFIA partners and stakeholders, including the industries regulated by the EFB.

Alignment with government priorities

• The EFB is aligned with federal government priorities, and the CFIA mandate.

Alignment with federal roles and responsibilities

• The EFB has two overarching objectives:
  1. safeguarding animal and public health; and
  2. facilitating market access for the regulated products.

While both objectives are recognized, there is a lack of consensus on the relative emphasis placed on animal and public health versus market access objectives of the EFB.

The federal government has a legitimate and necessary role in the EFB. Overall, the roles and responsibilities of the CFIA as well as those of the other partners and stakeholders are clearly defined and well understood internally and externally. However, there is a lack of awareness of the appropriate role for the CFIA vis-à-vis market access.

Findings – Performance

In this evaluation, it is important to note that achievement of the long-term outcomes (i.e. the effectiveness of the EFB) can only be assessed after 2015 due to the long incubation period of BSE.

Achievement of expected outcomes

• The process to develop the EFB regulatory framework was extensive and well-coordinated. The EFB regulatory framework was developed based on an adequate and appropriate level of consultation and risk assessment.
• The EFB implementation support activities were effective, particularly those relating to coordination and technical support.
• There is room for improvement in the current EFB management, decision-making and internal communication and coordination structures to better meet program objectives. In particular, there remains room for improvement in the system for results-based management specifically for reporting on performance, both internally and externally, to the CFIA.
• The CFIA has taken necessary and appropriate steps toward minimizing any negative impacts of internal and external factors on the performance of the EFB.

Demonstration of efficiency and economy

Management decisions affecting the re-allocation of staffing and resources and the effect of this re allocation on EFB planning and delivery across branches, divisions, and sections within the CFIA were not documented. This analysis also noted that certain EFB activities were not delivered as originally planned. The impacts (if any) of these changes on the achievement of the intended EFB objectives could not be determined. It is important to note that achievement of the long-term outcomes (i.e. the effectiveness of the EFB) can only be assessed after 2015 due to the long incubation period of BSE.

• The efficiency of the EFB relative to alternatives could not be assessed. Opportunities exist to address the challenges and limitations around financial tracking to better assess the efficiency of the EFB going forward.
• The CFIA had been successful in leveraging resources and avoiding duplication of activities in relation to the EFB.

Conclusions

The EFB is an important component of the Government of Canada's response to BSE, given the ongoing need to reduce the risk of transmission of BSE in the Canadian cattle herd to protect animal and public health and to facilitate market access for Canadian beef and other related products. Given its alignment with long-term federal government priorities and goals as well as current departmental responsibilities (including regulatory responsibilities), the EFB continues to remain relevant to the mandate of the CFIA.

The planning and implementation of the EFB both warranted and benefited from large-scale consultation and outreach, coordination and collaboration efforts and activities involving numerous stakeholders within Canada and some stakeholders outside Canada. The CFIA was successful and effective in bringing together and working with a large number of diverse stakeholders with a view to establishing consensus to develop and implement the necessary EFB regulatory framework and corresponding mechanisms and tools. To date, these immediate intended outcomes of the EFB have largely been met. However, it is too soon to assess the achievement of longer-term outcomes (i.e., the effectiveness of the EFB), due to the long incubation period, that is, the time from when an animal becomes infected until it first shows symptoms of the disease.

The EFB has faced some challenges since implementation. These challenges relate to the need for:

• strengthening the governance structure for the EFB; enhancing communication and coordination around the EFB internally and externally to ensure that the EFB adequately reflects the needs, priorities, and mandates of key EFB stakeholders and partners;
• facilitating the development of SRM testing and handling, disposal and use options and enabling technologies;
• improving performance measurement and management as well as financial tracking processes and systems; and
• addressing potential gaps in delivery of the EFB based on risk profiles and available resources.

Recommendations

Four recommendations have been formulated with a view to ensuring continuous improvement of the CFIA's EFB programming and achievement of the targeted outcomes.

The scope for this evaluation is 2004-2005 to June 2011. The BSE Program Management Committee was established in 2010 and was instructed to report to the BSE Fund Oversight Committee, which was created in June 2011. These two committees were renamed in 2012: the BSE Program Advisory Committee and the BSE Steering Committee, respectively. In September 2011, a new governance structure was put in place at the CFIA and the Animal Health Business Line Committee (AHBLC) was established. The BSE Steering Committee has been leading the BSE program oversight activities. This committee reports to the AHBLC and seeks recommendations from the BSE Program Advisory Committee for all of the BSE programming. The AHBLC makes activity prioritization recommendations to the Policy and Programs Management Committee (PPMC). An assessment of this new governance structure was not included in the scope of this evaluation.

Strengthen oversight, coordination and communication with regard to EFB-related activities, both internally and externally, by:

Recommendation 1

Strengthen the integration of the CFIA's EFB program objectives within the existing BSE committees as directed by the AHBLC or equivalent. As noted, the BSE Program Management Committee is tasked with ensuring that the CFIA's BSE-related activities for all program elements, including the feed ban, are discussed and developed internally in a coordinated, complementary and consistent manner. The BSE Fund Oversight Committee is tasked with ensuring that Agency activities pertaining to the distribution and use of BSE funds are in accordance with what was planned originally as well as developing a long-term strategic action plan to address reallocated funds.

Recommendation 2

Develop and communicate a plan for science-based policies related to SRM in support of the CFIA's EFB program objectives. As noted in Agency documents such as the "Discussion paper for drafting Canada's BSE Strategic Roadmap", the CFIA is committed to ensuring that any changes to its Enhanced BSE Programming (including the EFB) going forward are based on the latest scientific knowledge. Consequently, it is important for the CFIA to have access to the most current scientific knowledge with respect to BSE and SRM to ensure that policy and programming decisions reflect sound scientific evidence.

Mitigate the EFB challenges and gaps identified in this evaluation to ensure its effectiveness and efficiency by:

Recommendation 3

Improve performance measurement and management as well as budgeting and financial tracking procedures in relation to the EFB programming. There are related past and ongoing initiatives (e.g., upgrading information management systems within the CFIA, revising the performance measurement framework for BSE programming, and establishing the BSE Fund Oversight Committee) within the CFIA to strengthen tracking and reporting processes and systems for the Enhanced BSE Programming (and the EFB, as one of the components of BSE programming). Senior management commitment and oversight will ensure a sustained and coordinated approach in addressing both BSE and systemic issues that span across branches and programs.

Recommendation 4

Strengthen the planning for the allocation of EFB resources (human and financial) within the current governance process to address adjustments to the originally planned inspection scope and frequency. This would assess and address potential gaps in delivery of the EFB by ensuring that activities and establishments are being inspected at a level appropriate to their risk profiles and that adequate resources are available to each branch/program to support their respective EFB responsibilities.

The concerns over inconsistent application of the EFB between federal and provincial establishments/facilities as well as across federal establishments/facilities in different provinces could also be validated and addressed, if necessary. Ultimately, this exercise would place the CFIA in a position to better demonstrate the effectiveness of the EFB through sound and reliable EFB inspection and compliance verification activities despite the limitations of the SRM tests presently available to confirm the removal of SRM from the feed chain.

1.0 Introduction

1.1 Program Overview

1.1.1 Context

Bovine spongiform encephalopathy (BSE) is a progressive, fatal disease that affects the nervous system of cattle. Commonly referred to as "mad cow disease," it belongs to a group of diseases known as transmissible spongiform encephalopathies (TSEs). Research is ongoing, but to date there is no treatment or vaccine to protect against BSE. It is commonly acknowledged that cattle become infected by consuming feed contaminated with meat and bone meal derived from infected animals. Feed restrictions are recognized internationally as a key method in containing the spread of BSE.

In 1997, acting on the recommendations of the World Health Organization (WHO), Canada introduced through an amendment to the Health of Animals Regulations a pre-emptive feed ban to guard against the potential spread of BSE should it have been previously introduced. The 1997 feed ban prohibits proteins derived from most mammals from being fed to ruminant animals (such as cattle, sheep, goats, deer, elk, bison, etc.) At the time, this feed ban was considered to be an appropriate level of risk management for a country in which BSE had not been confirmed. The CFIA was responsible for delivering inspection and enforcement activities in relation to this feed ban.

Following the 2003 detection of a domestic case of BSE and subsequent detection of additional cases, the framework of the original feed ban was thoroughly reviewed. While surveillance results and the investigations of Canada's BSE cases detected since 2003 indicated that the 1997 feed ban was effective at limiting opportunities for the spread of BSE, enhancements to the ban were still deemed necessary. Consequently, a new regulatory framework was developed and the Enhanced Feed Ban (EFB) came into effect on July 12, 2007.

Of most significance, the enhanced 2007 feed ban regulations require the complete removal of specified risk material (SRM), which includes the skull, brain, trigeminal ganglia, eyes, tonsils, spinal cord and dorsal root ganglia of cattle 30 months or older, and the distal ileum (portion of the small intestine) of cattle of all ages, from animal feed, pet food and fertilizer. SRM has been excluded from the human food chain since 2003, and together with other ruminant tissues, were classified as prohibited material and banned from being fed to ruminants but not to other species. Under the EFB, SRM have been specifically banned from the entire terrestrial and aquatic animal feed chain, as well as fertilizer. Consequently, SRM are segregated at source and redirected to disposal or destruction. A permit system has been established to control the collection, transport, treatment and disposal of SRM. In addition, other enhancements have been made to the existing 1997 feed ban framework (e.g., to prevent feed cross-contamination at all points along the feed production and distribution chain from rendering facilities, to feed mills, to retailers and livestock producers). Altogether, amendments related to the EFB were made to regulations under the Feeds Act, the Health of Animals Act, the Fertilizers Act, and the Meat Inspection Act.

The EFB is a part of the Government of Canada's BSE Program, which is a horizontal initiative led by the CFIA. Under the BSE Program, the CFIA verifies that SRM is removed from the animal feed chain and the human food chain; monitors products entering and leaving Canada for adherence to Canadian standards or the standards of the importing country; monitors for the prevalence of BSE in the cattle population through surveillance; verifies that measures to control potential outbreaks are in place; and explains Canada's BSE control measures to domestic and international stakeholders (for example, through the veterinarians abroad program) to maintain confidence in Canada's BSE prevention and management efforts.^Footnote 4

The other federal government partners in the horizontal BSE Program include the Public Health Agency of Canada (PHAC), Health Canada, and Agriculture and Agri-Food Canada (AAFC). Health Canada conducts research and risk assessments regarding human exposure to BSE and other TSEs whereas PHAC carries out public health surveillance and targeted research in this area. AAFC has been involved in supporting, stabilizing and repositioning Canada's beef and cattle industry, including through the provision of compensation payments to stakeholders impacted by BSE in Canada. The work of these four federal government organizations is interrelated in terms of protecting animal and human health, as well as in facilitating market access for Canadian agriculture, agri-food and agri-based products.^Footnote 5 However, the entirety of the EFB specific funding ($26.6M per year, ongoing) is allocated to the CFIA for the delivery of EFB activities.

The following sections present an overview of CFIA's EFB objectives, activities, governance and resources. A more detailed program profile, as well as the program logic model, is provided in Appendix A.

1.1.2 EFB objective and outcomes

The main objective of the EFB is to accelerate Canada's progress in BSE management by preventing more than 99% of potential infectivity from entering the feed system as well as to enhance risk management of transmission of BSE in the cattle herd. To this end, the EFB aims to^{Footnote 6 Footnote 7}:

• Strengthen animal feed restrictions through amendments to the relevant regulations;
• Ensure compliance with control measures around prohibited materials and SRM removal;^Footnote 8 and
• Increase the level of verification and confidence that SRM is segregated from feed, fertilizer and pet food and that prohibited materials are not fed to ruminants.

1.1.3 EFB phases

For the purpose of this evaluation, the EFB activities were grouped into phases as follows:

• Development phase activities (2004-2007): consultations with regulated parties and other stakeholders (further details on EFB target groups and beneficiaries are provided in the Appendix A); risk assessments; and development of amendments to the relevant regulations;
• Implementation phase activities (2006-2007): included technical support and coordination, communication, and outreach activities prior to and after the EFB regulations came into effect on July 12, 2007; and
• Post-implementation phase activities (post-2007): ongoing activities such as inspections, verifying compliance and enforcing the new regulatory framework at federally regulated plants and feed, fertilizer and pet food establishments; issuing SRM permits and export certifications; and conducting research on SRM analytical and testing methods.

1.1.4 EFB delivery and governance

Multiple branches and programs within the CFIA are involved in the design, implementation and delivery of the EFB. Some branches and programs were involved intermittently or as necessary, such as the Public Affairs Branch and the Information Management and Information Technology Branch. Several branches and programs are involved in the EFB on an ongoing basis. They include:

• three branches: Operations, Policy and Programs, and Science
• four programs: Animal Health, Fertilizer, Feed, and Meat Hygiene

The Operations, Policy and Programs, and Science branches assume the lead role within the CFIA. Their primary responsibilities include (but are not limited to) the following:^Footnote 9

The Operations Branch is involved in frontline inspection activities, including education and awareness and identification of compliance and non-compliance with regulations administered by CFIA. In this capacity, the Operations Branch has assumed a lead role with respect to enforcement activities.

The Policy and Programs Branch is responsible for policy and regulatory development; ongoing revision, interpretation, communication and management of policy issues; inspection program design; and development of training materials.

The Science Branch is responsible for conducting risk assessments, developing compliance validation technologies, and analyzing samples.

Branches and divisions implement and deliver the EFB, both individually and in collaboration. Each of the branches and divisions had roles and responsibilities relating to the delivery of the original 1997 feed ban (i.e., that pre-date the EFB). In this context, the EFB provided supplemental funding across branches and divisions to allow them to implement and deliver additional activities under the EFB.

In 2007, a Feed Ban Task Force (FBTF) was established to support the implementation of the EFB. The FBTF had specific roles and responsibilities vis-à-vis the EFB and was led by the Operations Branch. Following implementation, specific EFB activities and tasks gradually reverted to the relevant and appropriate branches and programs. The FBTF was disbanded at the end of 2007.

The current governance structures and consultation mechanisms (2011) related to various aspects of the EFB are described below.

The Animal Health Business Line Committee and the BSE Fund Oversight Committee within the CFIA: Under the new governance structure of the CFIA, launched in September 2011, business lines have been tasked with managing specific funds for which they are generally the majority recipient.^Footnote 10 As a result, the Animal Health Business Line Committee has been tasked with having an oversight role for BSE funds for the CFIA. The mandate of this Committee is to ensure that CFIA activities pertaining to the distribution and use of BSE funds occur in accordance with Agency commitments and to develop a long-term strategic action plan to address reallocated BSE funds.

The BSE Program Management Committee within the CFIA: In 2010, in response to the 2008 summative evaluation of the Enhanced BSE Initiative, the CFIA established an inter-branch BSE Program Management Committee to improve communication and coordination of BSE-related activities to strengthen the linkage between the development and approval processes of individual activities (i.e., within branch business lines) and the horizontal, cross-branch priority-setting and policy development for the BSE program.^Footnote 11 The overall objective of the Committee is to ensure that the CFIA's BSE-related activities for all program elements, including the feed ban, are discussed and developed in a coordinated, complementary and consistent manner.

The EFB Working Group: A joint government-industry EFB Working Group, co-chaired by the CFIA and AAFC, was established in 2007.^Footnote 12 The goals of the Working Group are to review EFB-related measures being applied by packers and the rendering industry, their associated costs, and their contribution to the overall effectiveness of the EFB. Work is currently being done by the EFB Working Group in several areas to reduce the costs associated with SRM removal. Such work includes: assessing technologies available to remove specific components from slaughtered animals; making regulatory amendments to enable differences between the list of SRM to be removed from food and from animal feed; permitting wider use of composted SRM and SRM use in fertilizers; exploring options such as destruction technologies to generate less waste; and undertaking preparatory work for the 2012 CFIA review of the EFB.

1.1.5 EFB funding

The CFIA received $28.2 million over the first two years (2004-05 and 2005-06) for the EFB, followed by ongoing funding of $26.6 million per year since 2006-07. Table 1 provides a breakdown of the planned and actual allocation of EFB funding across relevant CFIA branches between 2005-06 and 2010-11.

During that time period, the CFIA spent almost $57 million less on the EFB than actually planned. While the CFIA noted that some of the budget variance could possibly be attributed to EFB-related activities carried out by other CFIA branches (e.g., Public Affairs Branch, Human Resources Branch, Legal Services Branch), this could not be confirmed via current financial tracking procedures. According to CFIA's Departmental Performance Report (2010-2011), the variance between planned spending and actual spending is related to funding being reallocated to expenditures that support this initiative, along with other Agency priorities.

1.2 Evaluation Objectives and Scope

1.2.1 Evaluation objectives

The evaluation of the EFB was identified as a priority in the CFIA's 2010-2011 to 2012-2013 Evaluation Plan. This evaluation meets a Treasury Board requirement to report on funding to enhance the 1997 feed ban in response to Canada's first case of BSE in a domestic animal found in 2003. As required by the Treasury Board Secretariat (TBS) Policy on Evaluation (2009) and in accordance with the supporting directive and standard,^Footnote 13 the primary objective of this evaluation was to assess the relevance (continued need, alignment with government priorities, and alignment with federal government roles and responsibilities) and performance (achievement of expected outcomes, as well as demonstration of efficiency and economy) of the EFB and provide recommendations to improve program effectiveness and efficiency, as necessary. The evaluation issues and questions related to relevance and performance are provided in Appendix B.

1.2.2 Evaluation scope

This is the first evaluation of the EFB and covers the program since its inception, from fiscal year 2004-05 to June 2011. The evaluation encompassed all EFB activities for which the CFIA received funding, including activities undertaken by the CFIA individually as well as in collaboration with EFB partners (federal and provincial government departments). It took into consideration the 2007 Evaluation of the CFIA's Feed Program, and the 2008-09 Summative Evaluation of the CFIA's Enhanced BSE Initiative as well as the Management Response and Action Plans (MRAP) and follow-up for these past evaluations.

As the EFB was implemented in July 2007, the assessment of the achievement of outcomes focused primarily on targeted short-term outcomes as outlined in the EFB logic model in Appendix A. Because of the long incubation period, that is, the time from when an animal becomes infected until it first shows symptoms of the disease, the evaluation of the long-term EFB outcome (i.e., the effectiveness of the EFB in controlling BSE) is possible only after 2015.

It is also important to note that:

• this evaluation is separate from the 2012 CFIA review of the EFB.
• the scope for this evaluation is 2004-2005 to June 2011. The BSE Program Management Committee was established in 2010 and was instructed to report to the BSE Fund Oversight Committee, which was created in June 2011. These two committees were renamed in 2012: the BSE Program Advisory Committee and the BSE Steering Committee, respectively. In September 2011, a new governance structure was put in place at the CFIA and the Animal Health Business Line Committee (AHBLC) was established. The BSE Steering Committee has been leading the BSE program oversight activities. This committee reports to the AHBLC and seeks recommendations from the BSE Program Advisory Committee for all of the BSE programming. The AHBLC makes activity prioritization recommendations to the Policy and Programs Management Committee (PPMC). An assessment of this new governance structure was not included in the scope of this evaluation.

The evaluation team was composed of evaluators from a consulting firm and the Audit and Evaluation Branch (AEB). The AEB oversaw the implementation of the evaluation and the consulting firm carried out the evaluation. The program data review, document and literature review and stakeholder interviews were undertaken from September to December 2011.

The evaluation approach and methodology were developed in consultation with the CFIA Evaluation Advisory Committee (EAC) and an independent scientific expert. The EAC was co-chaired by CFIA Executive Directors from the Evaluation Directorate and the Animal Health Directorate, and composed of representatives from CFIA branches (Policy and Programs, Operations, Science and AEB), as well as an external senior evaluation officer from a federal organization.

The evaluation was carried out in three phases.

1. In the first phase, the evaluation team developed a detailed evaluation framework, including specific evaluation questions, indicators, and corresponding data sources and data collection methods.
2. The second phase consisted of fieldwork to gather the necessary information and data.
3. In the third phase, the evaluation team analyzed and integrated the collected information and data and then synthesized and reported the evaluation findings. EAC committee members provided input and feedback on preliminary findings and on the draft evaluation report. Any clarifications and revisions necessary in light of the comments received were made accordingly.

The evaluation methodology incorporated the collection and review of data and information from multiple primary and secondary sources as well as through multiple methods to facilitate triangulation and cross-examination of findings from multiple lines of evidence for each evaluation question. The following sections present an overview of the evaluation methods and their limitations. A more detailed description of the data collection and analysis methodology, as well as limitations and their implications on evaluation results, is provided in Appendix C and D.

1.3 Quantitative Method

1.3.1 Program data review

A list of the various EFB-related data initially provided by the CFIA was developed and used to guide systematic data review. Additional data was requested from relevant internal programs and branches following the development of the initial list, particularly performance measurement data. This review was used primarily to develop a financial profile of the EFB programming with a view to assessing program economy and efficiency as well as to verify and complement findings from other lines of evidence.

Reviewed data included funding and expenditure data; performance indicators and raw data; outputs and activities data from committees and working groups; and data extractions from resource/activity/compliance databases, progress and summary reports, annual reports, audits, submissions to external/international organizations, internal correspondence, etc. See Appendix D for more details regarding methodology.

1.4 Qualitative Methods

1.4.1 Document and literature review

A comprehensive review of EFB-related documentation and literature internal and external to the CFIA was conducted.

1.4.2 Program documentation and file review

The objective of the review was to develop a thorough understanding of the various aspects of the EFB, such as its context, history, rationale, design, implementation, delivery, and outcomes, as well as to verify and complement findings from other lines of evidence.

More than 500 documents, files and reports related to the EFB were reviewed as part of this process. The CFIA provided electronic or print copies of internal documents, files and reports; additional materials were publicly available and accessed online. Reviewed materials included strategic, regulatory and policy documents; funding and expenditure data; departmental planning and performance reports; audits; submissions to external and international organizations; terms of reference for committees and working groups; public communication and outreach materials; as well as past evaluations and their associated MRAPs.

Literature Review: In addition to the EFB documents, files and reports from the CFIA, materials related to the EFB produced by external sources (e.g., provincial governments, industry associations) were reviewed, when available. A sentiment analysis of articles related to the EFB that were published by selected industry or market and media sources between 2004 and 2010 was also conducted to track changes in the attitude towards the EFB over time. The tone of each article was assessed and coded as positive, neutral or negative. The resulting analysis was used as a proxy for industry perception of the EFB to track change in perception over time relative to media views, and thus complement the findings from interviews with industry representatives.

1.5 Stakeholders Interviews

A total of 60 interviews were conducted with internal (31) and external (29) EFB stakeholders to obtain views and input on various aspects of the EFB to inform the evaluation of program rationale and performance issues. A breakdown of the planned and actual number of interviews by interview category is provided in Appendix D.

Internal Stakeholder Interviews: Interviews were conducted with 31 internal stakeholders in person or by telephone. The CFIA compiled the original list of potential interviewees within relevant branches, programs and regions; this list was complemented by suggestions for additional and back-up interviewees from CFIA regional staff. The sampling maximized the inclusion of key CFIA staff involved in the design or implementation and delivery of the EFB and balanced representation of CFIA staff across relevant branches and programs, as well as between the head office and regions. The interviewees included senior CFIA management representatives involved in the design, implementation and delivery of the EFB; managers from Operations, Policy and Programs, and Science branches; and regional and field staff from across the country.

External Stakeholder Interviews: Telephone interviews were conducted with 29 external stakeholders. Potential interviewees were identified by the CFIA, by an independent scientific expert in the field of BSE, and by representatives of stakeholder organizations. The interviewees included provincial government staff; industry and association representatives from relevant industries (cattle and beef, feed, fertilizer and pet food); academic and scientific experts on BSE and SRM; and staff from the Center for Veterinary Medicine within the United States Food and Drug Administration (FDA).

1.6 Analysis and Integration of Data

Following the fieldwork phase, the findings were analyzed and integrated across multiple lines of evidence. Information and data was validated within a method (i.e., corroborating the responses of multiple interviewees and multiple stakeholder groups) as well as between methods (i.e., corroborating interviewee responses with evidence from the document and literature review). The triangulation of findings across multiple lines of evidence was reinforced by having multiple evaluators engaged in the analysis and integration phase; these evaluators first independently and then collaboratively synthesized information and data from multiple sources and methods. Lastly, the preliminary evaluation findings were presented to AEB, the EAC and other relevant CFIA staff to identify factual errors and further confirm validity of results and conclusions. This feedback was addressed and taken into account in drafting this report.

Although the interviews were not designed or used to collect quantitative data, the interpretation of findings should take into account the extent to which certain perceptions or views were expressed or shared by interviewees. Table 2 defines terms (e.g., "some", "many") that are used in this report to quantify the proportion of interviewees or the proportion of groups of interviewees (e.g., internal stakeholders, industry or association representatives) who expressed similar experiences, views and opinions.

1.7 Challenges, Limitations and Mitigation Strategies

The evaluation challenges and limitations and the corresponding mitigation strategies are described in Table 3. The implications of these limitations are discussed in more detail in Appendix C.

2.0 Findings

2.1 Relevance: Continued Need for Program (R1)

2.1.1 Is there an ongoing need for the EFB and its components?

The circumstances and factors that prompted the introduction of the EFB have not changed significantly and warrant continuation of the EFB.

The continuation of the EFB is justified to a large extent by the fact that it is a requirement for Canada's BSE status, as determined by the OIE. Since 1998, the OIE has the mandate from the World Trade Organization (WTO) to officially recognize disease-free areas of countries for trade purposes.^Footnote 14 The procedure for the official recognition of disease status by the OIE is voluntary and currently applies to four diseases, BSE being one of them.^Footnote 15 Consequently, the OIE has developed standards to prevent the introduction of infectious agents pathogenic for animals and humans into an importing country during trade of animals and animal products, while avoiding unjustified sanitary barriers. The OIE also provides guidelines for countries to use in developing their respective import policies, based on a country's disease risk status. In the case of BSE, an effective feed ban is one of the conditions for maintaining Canada's controlled risk status as classified by the OIE, as well as for eventually qualifying for the negligible BSE risk status.^Footnote 16

Support for some type of a feed ban (though not necessarily the EFB) was also noted in 2010, when the CFIA hosted a forum to clarify the range of BSE stakeholder perspectives and their implications for modifying the EFB, as well as to identify the important elements that must be a part of a revised strategy for the future of BSE control in Canada. The forum, with participation from government and industry representatives as well as other stakeholders (e.g., BSE researchers and experts), affirmed the continued need for the EFB going forward.^Footnote 17

The view that the EFB should continue was also shared by the majority of internal and external stakeholders interviewed. The reasons cited for continuation of the EFB were twofold.

• Internal stakeholders and, to a lesser extent, external stakeholders noted that it was too early to assess the efficacy of the EFB in eliminating potential BSE infectivity in the Canadian cattle herd (as well as the associated animal and public health risks) due to the long incubation period of BSE, which ranges from 30 months to 8 years. Therefore, many stakeholders (especially internal) argued that the EFB should continue until enough time has passed to assess its effectiveness (i.e., after 2015).
• The other rationale centered on market access and was mentioned more frequently by external stakeholders. Some major export markets remain only partially open to Canadian beef products,^Footnote 18 whereas international market access for Canadian feed and pet food products regulated by the EFB continues to be restricted, with a few exceptions.^Footnote 19 As explained above, ongoing and future exports of these products are partially contingent on Canada continuing to meet OIE standards as a "controlled risk" country for BSE, which requires the demonstration of an effective feed ban.

To what extent does the EFB respond to the needs, priorities and mandates of its major stakeholders?

How could the design and delivery of the EFB be changed to better address and answer the needs of participating CFIA branches and divisions, other federal government departments and agencies, provincial government departments, and industries?

While support for the EFB is strong and widespread, there is no consensus on the appropriate or necessary level of feed ban enhancement to address the needs, priorities and mandates of key EFB stakeholders.

Stakeholder support for the continuation of the EFB was often contingent on further analysis of several interrelated issues and factors that, if addressed, would better meet their needs. The majority of stakeholders, both internal and external, highlighted a number of factors that should be taken into consideration in deciding whether changes and adjustments to the EFB are warranted going forward.

Some internal and external stakeholders noted that the level of effort required from both the CFIA and regulated parties to ensure the delivery of and compliance with the EFB across Canada was disproportionate relative to other diseases that pose a greater risk to animal (and human) health, but could be justified in light of the impacts of BSE on market access. While the challenge to align market access outcomes with the appropriate level of regulatory oversight extends to other CFIA regulations as well, it is particularly relevant in the case of the EFB since the impacts of BSE on market access can be significant. Therefore, these stakeholders indicated that there could be a need to develop a concomitant initiative on how to use the EFB more strategically and aggressively to differentiate Canadian products and gain competitive advantage in trade promotion and negotiation.

In addition some internal and external stakeholders reported the need for potential adjustments and revisions to the EFB in light of new knowledge about BSE and related technologies and systems since 2007 (e.g., atypical BSE strains, detection and testing techniques, surveillance methods).

It is important to note that the CFIA has been collecting input via stakeholder consultations on the need for and types of potential adjustments to the EFB in preparation for the 2012 review of the EFB. CFIA efforts in this regard include consultations with regulated parties carried out by the Enhanced Feed Ban Preliminary Implementation Review Working Group and the CFIA-AAFC Working Group focused on identifying options to reduce compliance costs.^Footnote 20 While such consultations are geared towards developing a comprehensive and consensus-based path forward with respect to BSE management in Canada, they also involve discussions around the need for/type of a feed ban, which is one of the key enabling mechanisms for BSE management.

2.2 Relevance: Alignment with Government Priorities (R2)

2.2.1 Does the EFB continue to be consistent with government-wide priorities and the CFIA mandate?

The EFB is aligned with federal government priorities and the CFIA mandate. However, there is a lack of awareness of the market access component of the CFIA mandate, both internally and externally.

Interviews with internal and external stakeholders, as well as a review of the Speech from the Throne (2010) and related documents and websites, confirmed that the EFB is consistent with federal government priorities. More specifically, the EFB is consistent with:

• Ongoing federal government commitments to supporting the Canadian livestock industry, including beef producers, processors and renderers, to improve their profitability and competitiveness, as well as to support other related industries (e.g., fertilizer, pet food);^Footnote 21
• The CFIA's goal to eventually obtain a negligible BSE risk status for Canada, as classified by the OIE;^Footnote 22 and
• The CFIA's mission ("dedicated to safeguarding Canada's food, animals and plants, which enhances the health and well-being of Canada's people, environment and economy")^Footnote 23 and its strategic objectives:
  • A safe and sustainable plant and animal resource base;
  • Public health risks associated with the food supply and transmission of animal disease to humans are minimized and managed;
  • Contributes to consumer protection and market access based on the application of science and standards.^Footnote 24

When discussing the alignment between the mandate of the CFIA and the objectives of the EFB, the majority of internal and external stakeholders referred to several aspects of the mandate of the CFIA, such as protecting Canadian livestock from diseases and pests, contributing to the safety of Canada's food supply, and safeguarding Canadians from preventable health risks.

However, market access was seldom mentioned or linked to the mandate of the CFIA by stakeholders. This could have been due to the stakeholders not being aware of the 2008-09 revisions to the CFIA's PAA (more specifically, the inclusion of "Domestic and International Market Access" as a CFIA "Program Activity"). The PAA was revised to highlight the significant CFIA programs and enable effective planning and reporting more strategically.^Footnote 25 The efforts to raise the profile of market access within the CFIA's suite of activities reflect the increasingly important interrelationship between product safety and market access in a global economy.

Relevance: Alignment with Federal Government Roles and Responsibilities (R3)

2.2.2 Are the objectives of the EFB clear and relevant?

The EFB has two overarching objectives—safeguarding animal and human health, and facilitating market access for the regulated products. While both objectives are recognized, there is a lack of consensus on the relative emphasis placed on animal and public health versus market access objectives of the EFB.

The intent of the EFB is to accelerate Canada's progress toward eradicating BSE from the national herd; removing SRM as per the EFB eliminates more than 99% of the potential infectivity from the feed system.^Footnote 26 The removal of SRM from animal feed is an important animal health protection measure and an indirect public health protection measure. In the short-term, the most effective measure to protect public health from BSE is the removal of SRM from food and other consumer products, whereas over the long-term, accelerating the eradication of BSE from the animal population through the EFB will further ensure the protection of both animal and public health.^Footnote 27 In addition, the EFB is also intended to help maintain consumer confidence and regain and expand market access for the regulated products (by ensuring Canada's "controlled" BSE risk status from the OIE, see above).^Footnote 28

Internal and external stakeholders did not raise any issues with respect to the clarity and relevance of the objectives of the EFB and affirmed the need for the removal of SRM from the feed system under the EFB. There was however, a lack of consensus on the primary objective or rationale behind the EFB. For many external stakeholders and some internal stakeholders, it was often their perception of the contribution of the EFB in sustaining and reopening export markets for Canadian products that was cited as justifying the considerable costs and efforts associated with the EFB rather than the contribution to achieving public and animal health. Although these two outcomes are closely interlinked, interviewees raised the fact that the comprehensive suite of EFB measures was implemented to address relatively small risks to public and animal health, especially compared to other diseases.

2.2.3 Is there a legitimate and necessary role for the federal government in the EFB?

The federal government has a legitimate and necessary role in the EFB. Overall, the roles and responsibilities of the CFIA as well as those of the other partners and stakeholders are clearly defined and well understood internally and externally. However, there is a lack of awareness of the appropriate role for the CFIA vis-à-vis market access.

The continued involvement of the federal government in the EFB through the CFIA was judged to be appropriate by the majority of internal and external stakeholders, given the Agency's past and ongoing responsibilities and activities. In particular, they highlighted the fact that the EFB built on a pre-existing feed ban overseen by the CFIA, as per federal regulations. In addition, the industries regulated through the EFB fall primarily under federal jurisdiction, with inspections delivered and overseen by the CFIA, as per their legislative authority relative to the Feeds Act and Regulations, the Health of Animals Act and Regulations, the Fertilizers Act and Regulations, and the Meat Inspection Act and Regulations. For example:

• Almost 95% of Canadian cattle are slaughtered in establishments and plants that are federally registered and inspected;^Footnote 29
• Federal abattoirs generate an estimated 90% of the SRM waste in Canada (the remaining SRM waste mostly comes from provincial abattoirs);^Footnote 30
• Feed mills are federally regulated businesses, and the federal government also regulates all fertilizers used in Canada;^Footnote 31 and
• While the pet food industry is not regulated in Canada, the CFIA regulates pet food imports and provides verification and certification services for pet foods that are made in Canada and intended for export.^Footnote 32

Are the roles and responsibilities of the CFIA and other federal and provincial departments involved in the EFB clear and well understood by internal and external stakeholders?

The majority of internal and external stakeholders generally reported that the roles and responsibilities of the different partners and stakeholders with respect to the EFB are clearly defined and well understood. Since implementation of the EFB in 2007, the roles and responsibilities of various CFIA partners and stakeholders (e.g., other federal government departments, provincial government bodies, industry associations and groups) with respect to the EFB have been further refined, clarified and reinforced through extensive discussions, as well as through the experience of working together, both formally and informally.

Some internal and external stakeholders indicated that another federal government organization, such as AAFC, would be more appropriate to address the market access objective of the EFB and envisioned the CFIA playing a supporting or complementary role in the area of market access. Greater clarity and communication on a number of issues would be beneficial in securing ongoing buy-in for the EFB from within the CFIA, as well as from the regulated parties. In particular, the following areas could be clarified:

• The extent to which safeguarding animal (and public) health remains the key driver of the EFB;
• The contribution of the EFB towards facilitating market access;
• The role of the CFIA in facilitating market access relative to other federal government organizations (e.g., as per the Memorandum of Understanding signed with the Market Access Secretariat of AAFC in 2010);^Footnote 33
• The rationale for the importance and relevance of the EFB in those regions of Canada where market access is not a priority issue (i.e., regions that generally do not export EFB regulated products and where the prevalence of cattle slaughtering or concentration of EFB regulated products is low) to demonstrate why EFB efforts are nevertheless necessary in those regions.

There was a consensus that the CFIA was the most appropriate organization to oversee the overall implementation of the EFB regardless of the emphasis stakeholders placed on animal and public health versus market access. Many internal and external stakeholders noted that the CFIA was the only government organization that had the necessary regulatory authority, mechanisms and resources in place to ensure compliance with the EFB across Canada.

2.3 Performance: Achievement of Expected Outcomes (P1)

It is important to note that achievement of the long-term outcomes (i.e. the effectiveness of the EFB) can only be assessed after 2015 due to the long incubation period of BSE.

2.3.1 To what extent has the development of the EFB regulatory framework by the CFIA been effective?

The process to develop the EFB regulatory framework was extensive and well coordinated. The EFB regulatory framework was developed based on an adequate and appropriate level of consultation and risk assessment.

Design of the regulatory framework for the enhancement of animal feed restrictions (i.e., the Feeds Act, the Health of Animals Act, the Fertilizers Act, and the Meat Inspection Act): In December 2004, the CFIA announced amendments to relevant federal regulations regarding the prohibition of SRM from being used in all animal feed, including pet food and fertilizer. The proposed amendments were not finalized and published in the Canada Gazette until June 2006 to allow for adequate time to draft the amendments to incorporate the input and feedback from consultations and risk assessments.

Interviews with internal and external stakeholders as well as document review noted the significant contribution of consultations and risk assessments to the development of a sound and comprehensive regulatory framework, as described below. The finalized amendments did not come into effect until July 2007 to enable both the regulated parties and the CFIA to prepare for their implementation. The CFIA worked extensively with key stakeholders to determine which measures would best provide effective, rational and responsible feed ban controls. This included determining scientific merit, economic impacts, practical feasibility, and the consistency of approaches with other jurisdictions, including international trading partners, provinces and territories. The prolonged consultation period ensured that industry and other stakeholders, in cooperation with the CFIA, could identify and begin implementing the operational and infrastructural changes necessary to comply with the EFB.

The design of the EFB regulatory framework clearly took the consultations and risk assessments into account. The EFB regulations were drafted in a flexible manner so that relevant new knowledge and technologies related to BSE/SRM/prions could be reviewed or incorporated at a later stage.

Risk assessments and other studies to inform the regulatory framework: Thorough risk assessments based on the most up-to-date and complete information available about BSE and prions at the time were conducted by the CFIA to inform the development of the EFB regulatory framework. The CFIA evaluated different enhanced feed ban options using a risk reduction model to compare the time it would take to eradicate BSE from the cattle population.^Footnote 34 The CFIA also conducted 12 qualitative risk assessments to inform proposed EFB regulations as indicated below.^Footnote 35

• Risk reduction options (controlled incineration, cement kiln, alkaline hydrolysis);
• Containment options (on-farm burial, landfill and mass burial);
• Technologies under investigation (on-farm composting, mass composting, thermal hydrolysis, gasification only, and gasification with incineration); and
• Land application of sludge (treated wastewater from abattoirs and rendering plants).

These assessments focused on the risks associated with new cases of BSE arising in cattle and other domestic ruminants resulting from various on-farm and industrial disposal options of raw or rendered SRM or carcasses. They also provided an assessment of the risks arising from BSE via release and exposure pathways by spreading the residues on agricultural land or landfill, in connection with the domestic ruminant population. The establishment of infectivity reduction and the level of risk presented by those disposal options were also assessed.

2.3.2 To what extent have implementation support activities for the EFB regulatory framework by the CFIA been effective?

The implementation support activities have been effective, particularly those relating to coordination and technical support.

Most internal and external stakeholders indicated that extensive collaboration, coordination and communication within the CFIA and between the CFIA and relevant partners and stakeholders served as key success factors for effective implementation of the EFB. Some internal stakeholders noted that the breadth and depth of consultation, coordination, communication, and outreach efforts surpassed those for any previous CFIA initiative they had been involved with.

Feed Ban Task Force coordination and technical support activities: Interviews and documents showed that coordinated and targeted outreach efforts effectively built rapport between the CFIA and industry/other stakeholders.^Footnote 36 The FBTF was heavily involved in these efforts and carried out 204 communication activities in 2007 (e.g., preparing briefing notes, making presentations, giving media interviews).^Footnote 37 In addition, the FBTF issued numerous decision documents outlining policy clarifications to help the regulated parties to meet the intended health and safety outcomes of the enhanced feed ban more effectively.^Footnote 38

The FBTF was disbanded and incorporated into a Strategic Projects Team in January 2008. The CFIA recognized the use of cross-sectional groups such as the FBTF as a best practice.^Footnote 39 The formation of such groups was critical for effective and timely decision making by fostering a team approach to implementing a complex set of regulations that affected numerous stakeholders.

Other coordination, communication and outreach activities: In addition to the FBTF other key EFB groups and committees including the Enhanced Feed Ban Preliminary Implementation Review Working Group, the Feed Ban Steering Committee, the Industry Liaison Team, the Technical Implementation Committee, and the Inspection Oversight Evaluation Team. Many internal and external stakeholders indicated that these groups contributed to building consensus around complex policy and enforcement issues involving a diverse range of stakeholders while facilitating the provision of necessary implementation support internally and externally.

The Public Affairs Branch of the CFIA also drafted a broad communications plan targeting industry and industry associations, provinces, farmers, and the general public. The communications plan was further enhanced four months prior to implementation of the EFB to include an information package for all Members of Parliament, public notices in newspapers, and "Questions & Answers" on the CFIA website, among others. For the twelve months prior to July 2007, the CFIA initiated a comprehensive, national communications campaign to ensure that all regulated parties were fully aware of their responsibilities. Communication material was sent to the regulated parties across Canada, including industry associations, small abattoirs, feed retailers, pet food manufacturers, salvage operators, transporters, waste management corporations, zoos and auction markets. Booklets, brochures and posters were distributed to individuals, groups and organizations, and information was made available through a special page on the CFIA website. Public notices appeared in agricultural publications and community newspapers across Canada in April and July 2007, urging the regulated parties to contact the CFIA for information about the EFB regulations. A series of radio advertisements aired on rural community radio stations in the weeks leading up to the July 2007 implementation date.^Footnote 40

2.3.3 To what extent was the implementation of the EFB regulatory framework by the CFIA well coordinated and effective in branches and regions across the Agency?

The implementation of some of the EFB components (e.g., training) was effective and generally well coordinated within the CFIA, whereas certain other components (e.g., development of laboratory analytical methods) could have been better coordinated and delivered more effectively.

EFB staff recruitment and training activities: CFIA representatives interviewed as part of the evaluation provided inconsistent views on whether CFIA recruitment efforts related to the EFB were adequate and effective. The 2005-06 Annual Performance Report for the EFB indicated that during the 2005-06 fiscal year, approximately 90% of the staffing for feed specialist inspection positions was completed for the EFB (i.e., approximately 115 staff hired), but limited information on EFB staffing numbers is available after this date or for other program areas (e.g., animal health).^Footnote 41 Some CFIA representatives indicated that the number of staff hired was adequate, whereas others stated that insufficient funds were provided to individual branches and programs to attain the appropriate level of staff (see also Section 2.4).

CFIA and provincial government staff indicated that comprehensive training sessions and manuals organized and developed by the CFIA, as well as mentoring activities, adequately supported the work of staff, including new hires and provincial staff, involved in inspection and compliance verification activities. Both internal interviewees and the document review confirmed that the CFIA undertook training activities across Canada. These activities included a combination of conference calls and training sessions for at least 500 staff working in the areas of meat hygiene, animal health, feed and fertilizer as well as for SRM and EFB inspectors, supervisors and managers, and other relevant field staff.^Footnote 42

Development of laboratory analytical methods to support a sampling verification system in compliance with the regulations: A few external and internal stakeholders indicated that the laboratory technologies and analytical methods used in Canada in relation to the EFB were on par with the best technologies and methods available worldwide at this point. However, current international SRM testing and analytical methods have significant limitations for use in the Canadian context. This is due to their inability to conclusively distinguish between prohibited proteins of any particular species and blood or milk proteins of the same species (the latter are exempt under the EFB).^Footnote 43 It is worth noting that blood and milk proteins from all species (including ruminants) were excluded from the scope of the original 1997 feed ban. Based on the best available scientific evidence at that time, they were considered very low risk tissues for transmitting BSE. This position remained unchanged following the review of the original 1997 feed ban in response to findings of domestic BSE cases and, as a result, the exclusions were upheld in the scope of the EFB.

The document review confirmed a number of projects carried out by different CFIA labs to develop the necessary testing and analytical methods to aid verification of compliance with the EFB.^Footnote 44 For example, the CFIA developed or adapted testing methods for the detection and identification of material derived from different animal species (bovine, sheep, goat, elk, deer, pig, horse, poultry and fish) in feeds to distinguish between prohibited (ruminant) and exempt species. However, up-to-date information was not available on the status of some of these projects, and more specifically whether any tests have been validated and transferred to a diagnostic laboratory for use to address the aforementioned limitations.

Due to the limitations of the presently available SRM testing and analytical methods, the CFIA relies primarily on inspection and compliance verification activities in support of the EFB. Inadequate use of testing or analytical methods to support EFB inspection and compliance verification activities had been highlighted as a potential area of concern by the CFIA^Footnote 45 since the effectiveness of the EFB could not be fully demonstrated to internal and external stakeholders using scientific methods.

Inspections and enforcement activities, and import verifications, permits (tracking movement of SRM material) and export certifications: The CFIA developed a comprehensive and coordinated compliance and enforcement strategy so that the four relevant programs within the CFIA (feed, fertilizer, animal health, and meat hygiene) could assess compliance with EFB regulations and take enforcement actions if necessary in a consistent and unified manner.^Footnote 46 Compliance with legislative requirements in a wide range of establishments and facilities was designed to include a range of inspection activities, such as monitoring, surveys, compliance verifications, audits of facilities and operations; sampling and laboratory analyses of regulated products; and reviewing of records and documents.

Enforcement is based on a graduated approach focussing on cooperation, education and compliance. Under this graduated approach, the CFIA can take one or more enforcement actions in response to non-compliance. Enforcement actions may include:

• Warning letters when an infraction is unlikely to have resulted in significant or serious harm, is obviously unintentional and is easily corrected, and where the company has made reasonable efforts to control the potential risk;
• Corrective action requests for situations in which the risk has been effectively controlled but which require the company to implement an action plan to prevent the reoccurrence of the event by an agreed date;
• Seizing and detaining products which may have been potentially cross contaminated;
• Recall orders for products that pose a human health, animal health or environmental risk;
• Prosecutions; and
• suspending or cancelling permits for serious or ongoing unresolved issues.

However, some internal stakeholders noted that in some instances, existing enforcement provisions were not strong enough or the original enforcement protocols were not implemented as planned. In particular, the fact that the use of administrative monetary penalties (AMPs) had not been implemented under the Feeds Act, the Fertilizers Act, the Meat Inspection Act to address certain cases of non-compliance, despite this measure being explicitly identified in original compliance and enforcement plans for the EFB.^Footnote 47

Documents and interviews confirmed that compliance inspections now occur at various types of facilities including inedible rendering plants, commercial feed manufacturers, fertilizer/fertilizer supplement manufacturers and composting facilities, feed and fertilizer retail outlets, on-farm feed manufacturers, and ruminant feeders.^Footnote 48 With regard to expanded inspection activities for feed and fertilizer products, the document review also confirmed that inspections now include verification of feed and fertilizer invoices and documentation, compliance with permit conditions, and presence or absence of prohibited materials, as appropriate.

Some internal and external stakeholders also provided specific examples of enhanced compliance and enforcement measures, such as an increased record retention period (i.e., 10 years versus 2 years); inclusion of operations previously not inspected by the CFIA (e.g., fish feed mills, retailers); new/augmented import/export certification and permitting requirements relating to SRM for animal feed, fertilizer and pet food products; and issuance of permits for SRM transport, disposal and use.

Attempts were made during the evaluation to determine the number of inspections, enforcement activities, and issuances of SRM permits, import verifications and export certifications. However, as discussed in Section 2.3.5, the compilation of such data at the level of the EFB is limited by the fact that EFB activities are delivered across multiple program areas that each use a variety of different performance measurement systems. Based on the available information, it is possible to state that the Feed Program delivered the following inspection tasks in the 2010 calendar year:^Footnote 49

• Commercial feed mills: 4,884 tasks were performed at 460 facilities
• Rendering facilities: 884 tasks were performed at 45 facilities
• Retail: 2,473 tasks were performed at 468 facilities
• Farms: 332 tasks were performed at 73 facilities

In addition, SRM-related tasks at the 35 federally regulated establishments that slaughter and process Canadian cattle (representing 95% of the total) are inspected on a daily, weekly or monthly basis.^Footnote 50 Finally, the CFIA renewed and issued a total of 1,804 SRM permits in 2009-10 and 1,401 permits in 2010-11.^Footnote 51

2.3.4 Are the EFB activities producing the expected outcomes?

Based on available information, the immediate intended outcomes of the EFB have largely been met, but it is too soon to assess the achievement of longer-term outcomes (i.e., the effectiveness of the EFB) after 2015 due to the long incubation period of BSE.

Animal feed restrictions have been strengthened (i.e., Feeds Act, Health of Animals Act, Fertilizers Act and Meat Inspection Act): The majority of internal and external stakeholders as well as the document review confirmed that animal feed restrictions had been strengthened through amendments to the relevant regulations (Feeds Act, Health of Animals Act, Fertilizers Act, and Meat Inspection Act), thereby granting the CFIA the required authority to verify and enforce compliance with the EFB. The strengthened regulations also complemented the "Removal of SRM from the Human Food Supply" component of the CFIA Enhanced BSE Programming by preventing SRM from entering the non-food pathways of exposure to the public.^Footnote 52

Compliance with control measures (prohibited material/SRM removal); increased verification and confidence that SRM is segregated from feed, fertilizer and pet food and that prohibited materials are not fed to ruminants: Performance data to determine the adequacy and impacts of these enhanced compliance and enforcement measures were limited and therefore the overall effectiveness and efficiency of the EFB could not be determined. However, some data were available related to specific control measures or tasks or types of facilities, as these are prepared for various internal and external reporting needs.

For example, the CFIA's November 2010 report to the OIE^Footnote 53 indicated that compliance with a number of different verification tasks was high in rendering facilities (98% or greater) and feed mills (97% or greater for feed mills handling prohibited material and 96% for mills not handling prohibited material) that were inspected in 2009-10. Only one potentially serious infraction was found, and measures were taken that ensured that the risk of ruminant feed being cross contaminated with SRM was averted. Meanwhile, compliance rates for four SRM-specific inspection tasks at federally registered beef slaughter and boning plants were also very high during this period, between 99.6% and 100%. For over 5,492 individual inspection activities, a total of five compliance action requests (CARs) were issued in relation to excluding SRM from the animal feed chain. While the actual risk of cross contamination in each instance was averted as a result of subsequent control actions, each one was considered serious enough to warrant a CAR so that further non-compliance could be averted in the future. Overall, while internal and external stakeholders often expressed confidence in the enhanced measures based on their own observations at work and/or anecdotal evidence, others pointed to the need for better performance data to make an informed opinion.

Enhancing risk management of transmission of BSE in cattle: As indicated in Table 4, the number of BSE cases identified in Canada has been low in recent years (one case confirmed per year since 2009). However, this decline cannot be directly attributed to the EFB since the EFB is only one of the components of the CFIA Enhanced BSE Programming. More importantly, due to the long incubation period of BSE, it is too early to determine the extent to which the strengthened EFB regulations, along with enhanced compliance and enforcement measures, will have contributed towards the achievement of this long-term intended EFB outcome. Consequently, this outcome fell outside the scope of this evaluation.

Source: CFIA^Footnote 54

2.3.5 Are appropriate management, decision-making and internal communication and/or coordination structures in place to meet the program objectives?

There is room for improvement in the current EFB management, decision-making and internal communication and coordination structures to better meet program objectives, particularly relating to performance measurement and monitoring.

Has there been an assessment and strategic use of reviews and other lessons learned? How have the management response and action plan(s) and findings of past reviews contributed to the effective delivery of the EFB?

The document review confirmed CFIA efforts to improve the design and delivery of the EFB based on the findings of relevant past evaluations,^Footnote 55 and on best practices and lessons learned.^Footnote 56 For example, following a recommendation from the summative evaluation of the Enhanced BSE Initiative, the BSE Program Management Committee was created in 2010 to better coordinate the BSE activities within the CFIA.^Footnote 57 In addition, during the implementation and maintenance of the EFB, the CFIA formally and informally tapped into and built on the previous EFB structures, linkages and relationships established both internally and jointly with partners and stakeholders (e.g., the EFB Working Group).^Footnote 58

Are these structures working as intended?

Many internal and external stakeholders indicated that a lack of clarity around the EFB governance structure (i.e., accountability, roles and responsibilities, point of contact for specific EFB issues) gradually emerged following the post-implementation, after the disbanding of the majority of the internal and external cross-sectional committees and working groups (especially the FBTF). For example, internal stakeholders seldom made reference to the BSE Program Management Committee when discussing the current EFB governance structure. The BSE Program Management Committee and other existing EFB structures and mechanisms in place to oversee the post-implementation phase do not have a strong and clear EFB-specific mandate (i.e., responsibility/accountability).

Internal stakeholders and the document review also pointed to a disconnect between the Policy and Programs Branch and the Operations Branch, which affects planning and delivery activities.^Footnote 59 More specifically, activity targets set by the Policy and Programs Branch may not necessarily take into account the actual resources available to the Operations Branch and/or the actual activities carried out by the Operations Branch in the past, which makes the planning of inspection and compliance activities less effective. Document review confirmed this view (e.g., a 2007 evaluation of the Feed Program found that Operations Branch reversed a new inspection form released by Policy and Programs Branch due to a lack of clear inter-branch communication and Operations Branch staff had variable degrees of understanding/interpretation of inspection frequencies as per Policy and Programs Branch guidelines).^Footnote 60

In 2010, the CFIA launched a new committee-based matrix structure to bring its overall approach to governance more in line with evolving business needs.^Footnote 61 The new governance structure focuses on the importance of sharing information internally and seeks to ensure a more efficient and streamlined senior-level committee structure. It is expected that the renewed structure will foster a holistic approach to decision making and support day-to-day operations across the CFIA. To ensure that business line perspectives are integrated into decision making, three senior-level committees are supported by four committees: Animal Health, Plant, Food and Horizontal Management. It is too early to determine whether the new matrix structure around business lines will improve communication and coordination within CFIA branches/divisions.

Performance management and performance monitoring tools and activities

The performance measurement system for the EFB has significant ongoing challenges and limitations. There remains room for improvement in the system for results-based management, as well as for reporting on performance externally to the CFIA.

Despite improvements in the performance measurement systems within the CFIA, the information requirements to measure and track the effectiveness and performance of the EFB identified throughout the design and implementation of the EFB have not been adequately addressed post implementation. An Effectiveness Working Group met several times in 2008 to lead the development of "informatics" or key performance indicators as well as to improve performance data measures and systems to ensure that required EFB data is collected and compiled to support internal and external needs. These needs pertain to informing the internal results-based management of the EFB (i.e., to assess and confirm that EFB requirements are being implemented effectively, to identify areas for improvement and to demonstrate and report the effectiveness of the EFB to both international and domestic audiences (e.g., OIE, re-opening of export markets, Departmental Performance Reports [DPR]).

However, despite improvements to certain systems, multiple lines of evidence suggest that the challenges identified by the Effectiveness Working Group had not been addressed completely. The EFB involves four previously distinct program sectors and inspection frameworks (i.e., Feed, Animal Health, Meat Hygiene and Fertilizer), with the Operations Branch playing a key role in national and regional delivery of EFB activities. A key challenge is to coordinate and harmonize the different data collection processes and systems used by individual program sectors and the Operations Branch. The CFIA approach to adapt existing information management systems for the EFB as opposed to developing a customized system to meet EFB data needs also makes it challenging to gather EFB-specific data (e.g., due to a lack of customized features).

Overall, the document review,^Footnote 62 stakeholder interviews, and an in-depth review of the available performance data by the evaluation team showed that the information management systems and procedures are limited in terms of performance measurement and reporting of EFB-specific data and indicators. Quantitative performance data can be extracted from individual systems, such as in the case of compliance for inspections conducted under the Feed Program in specific facility types (e.g., renderers, feed mills), but interpretation of these data often requires additional steps thereby affecting the timeliness of results.

While these issues do not affect only the EFB alone, in the context of this evaluation, they limited the ability to use EFB-specific performance data to assess the achievement of program outcomes. As discussed under Section 2.3.3, available performance information for the EFB does show that several activities are being delivered by the CFIA to enhance control measures under the EFB's strengthened regulatory framework. However, currently available performance data cannot be rolled up at the level of the EFB, such that it is not possible to use quantitative data make a sound assessment of the effectiveness of the EFB.

It is important to note that the CFIA is currently working on an improved performance measurement framework for BSE programming to be able to report reliably on specific program objectives and improve program decision making. The CFIA's Agency-level 2011-12 performance measurement framework (based on its PAA) identifies a number of indicators related to the EFB but, based on available data, these indicators have yet to be measured and tracked as defined in the framework.^Footnote 63 Evidence of recent efforts within the Feed Program also indicates that ongoing work is also taking place within individual programs to address the limitations of the current performance measurement and monitoring data with regard to a range of initiatives or activities.^Footnote 64

2.3.6 Did any external or internal factors influence the performance of the EFB?

Were any actions taken as a result of these factors?

Several internal and external factors impacted the performance of the EFB. The CFIA has taken necessary and appropriate steps toward minimizing any negative impacts on the performance of the EFB as a result of these factors. There is a need for the CFIA to continue its efforts in some areas (particularly, SRM management and disposal costs and variable interpretation/inconsistent application of the EFB).

Incomplete or confusing policies, rules and guidelines: Some internal and many external stakeholders stated that incomplete or confusing policies, rules and guidelines in some instances contributed to increased compliance costs or challenges, particularly leading up to and during the implementation stage (2007). This was partly because the regulated parties encountered unique and unforeseen challenges and issues in their initial efforts to comply with the EFB and the CFIA did not have readily available guidelines to address them. In addition, internal stakeholders noted initial uncertainty within and outside the CFIA over certain procedures and measures as to whether or not they would meet EFB requirements as well as a lack of uniform and consistent interpretation of some policies and rules, whereas external stakeholders noted changing CFIA policy on composting of SRM. Internal and external stakeholders as well as the document review^Footnote 65 indicated that these issues had been addressed and the necessary clarifications had been made by the CFIA throughout the implementation and post-implementation stages.

SRM management and disposal costs: Some internal and many external stakeholders noted that SRM management and disposal costs were underestimated, making it more challenging for the regulated parties to comply with the EFB. The document review confirmed industry concerns over compliance costs.^Footnote 66

Since the implementation of the EFB, the CFIA has held ongoing discussions with industry to find ways to address the impacts of compliance costs, particularly through the Enhanced Feed Ban Preliminary Implementation Review Working Group and the EFB Working Group.^Footnote 67 Although some progress has been made in this regard in terms of reduction in compliance costs,^Footnote 68 industry concerns over ongoing compliance costs have not been completely addressed.^Footnote 69 Furthermore, as outlined in Figure 1, an analysis of approximately 100 EFB-related articles published by industry and associations between 2004 and 2010 indicates that industry opinion of the EFB had become less favourable over time, largely because of the negatively perceived impacts of the EFB on industry operations and costs. Interestingly, this contrasts with the tone of general media articles, which were the most negative in 2005 and immediately after the EFB was implemented (2008), but have since become primarily positive or neutral.

Description for image - Figure A

• In 2004, 2 articles were published; 50% were positive and 50% neutral.
• In 2005, 12 articles were published; 50% were positive, 25% were neutral and 25% were negative.
• In 2006, 16 articles were published; 80% were positive and 20% were neutral.
• In 2007, 29 articles were published; 38% were positive, 21% were neutral and 41% were negative.
• In 2008, 20 articles were published; 20% were positive, 25% were neutral and 55% were negative.
• In 2009, 17 articles were published; 11% were positive, 37% were neutral and 52% were negative.
• In 2010, 10 articles were published; 20% were positive, 20% were neutral and 60% were negative.

Description for image - Figure B

• In 2004, 8 articles were published; 25% were positive, 25% were neutral and 50% were negative.
• In 2005, 29 articles were published; 20% were positive, 9% were neutral and 71% were negative.
• In 2006, 37 articles were published; 35% were positive, 40% were neutral and 25% were negative.
• In 2007, 15 articles were published; 12% were positive, 41% were neutral and 47% were negative.
• In 2008, 20 articles were published; 15% were neutral and 85% were negative.
• In 2009, 4 articles were published; 50% were positive, 25% were neutral and 25% were negative.
• In 2010, 6 articles were published; 50% were positive, 35% were neutral and 15% were negative.

Variable interpretation and inconsistent application: Some internal and external stakeholders shared concerns about variable interpretation and inconsistent application of the EFB, both between federal and provincial establishments and facilities, and across federal establishments and facilities in different provinces. Lack of readily available and uniform documentation specific to EFB activities (e.g., inspection and compliance verification, see Section 2.3.4) makes it difficult to ascertain that identical standards and procedures are being used across the country. There are also concerns that some operations (e.g., farms, provincial abattoirs, and non-registered slaughter facilities, and facilities that do not undertake cattle slaughtering on a commercial scale and therefore, are not federally or provincially registered) are not subject to the appropriate level of oversight and rigour given their risk profiles. The Feed Program Delivery and Emergency Management Committee redesigned overall inspection of commercial feed mills, rendering plants, feed retail outlets and on-farm feed mills based on an updated risk-based approach in 2009.^Footnote 70 However, information was not available on whether, and if so how, such changes might have affected EFB-specific inspections.

Financial support: An important external enabler of the EFB was financial support provided by AAFC in partnership with provincial governments to assist the regulated parties to comply with the EFB. The negotiations for this funding were facilitated in part by the FBTF.^Footnote 71 Staff from AAFC and the FBTF travelled across the country, in a short period of time, to facilitate the negotiation of the federal/provincial funding agreements for the SRM Management Programs. By the July 2007 deadline, agreements were put in place with all provinces except Newfoundland and Labrador, for which a working group was established to continue efforts to get a funding agreement in place. Through the provincial SRM Management Programs, federal government funding of $80 million leveraged an additional $51 million in provincial funding. Although the funding disbursed by AAFC was not part of the funding allocated to the CFIA for the EFB, the SRM Management Programs greatly enabled the implementation of compliance measures for the EFB by regulated parties. Most external stakeholders indicated that without financial assistance from the provincial SRM Management Programs, many operations affected by the EFB might not have been able to implement the necessary operational and infrastructural changes before the July 2007 deadline or might have been forced to shut down. However, this funding also resulted in expectations of continued financial assistance among the regulated parties.

2.3.7 Has the EFB produced additional outputs or outcomes (e.g., unintended [positive or negative] outcomes)?

Were any actions taken as a result of these unexpected or unintended outcomes?

Internal and external stakeholders noted some unintended outcomes of the EFB, both positive and negative. There is a potential role for the CFIA in managing these unexpected and unintended outcomes for the benefit of the EFB.

• The EFB, as part of the CFIA's Enhanced BSE Programming, contributed to the development of BSE expertise in Canada, both within and outside the CFIA. For example, federal and provincial governments have provided funding for R&D to improve SRM detection tests, develop more efficient SRM disposal techniques, and identify alternate use of SRM. The document review confirmed such funded projects.^Footnote 72 Some internal and external stakeholders advocated increased focus and funds/collaboration to develop more conclusive SRM detection and testing techniques and better SRM handling, disposal and use mechanisms.
• The EFB triggered various changes in how the regulated parties operate as they adapted to strengthened regulations and enhanced compliance and enforcement measures. Some key changes that were reported by industry included decreased processing of cattle over 30 months of age, increased processing of Canadian live cattle in the US, mixed species operations switching to single species, and changes in pet food formula to use alternate raw materials. These changes were generally considered negative by the regulated parties due to the additional efforts/costs associated with them and/or their impacts on the bottom line.
• Increased on-farm disposal of carcasses was highlighted as another unintended consequence. Many external stakeholders noted that following the implementation of the EFB, it has become more difficult to send cattle carcasses to rendering due to increased service charges to cover the costs of compliance with the EFB. As a result, more farmers are opting for disposal of carcasses on undeveloped or vacant land adjacent to farms. While this practice is legal in most provinces, external stakeholders voiced concerns that such unsupervised disposal of carcasses could increase the risks of disease transmission to livestock/wildlife as well as water contamination.
• The private sector did not invest in operations and infrastructure for alternate SRM use as anticipated. Some internal and external stakeholders indicated that private sector investments were discouraged by several factors, such as negative perception of SRM (e.g., a cement plant that explored the feasibility of using SRM as a fuel source was met with opposition from the community and plant workers over health, safety and environmental concerns), lack of economies of scale/small size of operations (e.g., not having access to large quantities of SRM or limited resources and capacity to develop innovative solutions to dispose or reuse SRM), uncertainty over treatment of SRM in the future (e.g., how long the EFB will continue and the implications for guaranteed SRM supply going forward), and the global recession.

2.4 Performance: Demonstration of Efficiency and Economy (P2)

2.4.1 Is the Enhanced Feed Ban meeting its objectives within its allotted budget?

Management decisions affecting the re-allocation of staffing and resources and the effect of this re-allocation on EFB planning and delivery across branches, divisions, and sections within the CFIA were not documented. This analysis also noted that certain EFB activities were not delivered as originally planned. The impacts (if any) of these changes on the achievement of the intended EFB objectives could not be determined. Achievement of the long-term outcomes (i.e. the effectiveness of the EFB) can only be assessed after 2015 due to the long incubation period of BSE.

To what extent has there been an increase in inspection capacity to support effective implementation of the EFB across branches/regions of the Agency?

It was not possible to conclusively identify and assess EFB-specific FTEs or the number of FTEs hired under the EFB as part of the internal document and data review, largely because of challenges in tracking EFB-funded resources. Indeed, attempts to extract relevant information from internal databases and files did not yield coherent or reliable results. CFIA representatives confirmed that it was an ongoing challenge to determine and track EFB-specific FTEs over time since EFB-related tasks typically represent only a portion of the overall workload of an individual.

Overall, the document review suggests that about 115 inspection staff were hired in 2005-06 for the feed program,^Footnote 73 but data are not available for other program areas or subsequent years. More recent documents show the CFIA intended to have 270 full-time equivalent employees by July 2007 who would be responsible for enforcing the EFB.^Footnote 74 There was no follow-up data to confirm whether or not this target was met, or whether or not it was adequate. Interview data with CFIA representatives was inconsistent: some indicated that the number of staff hired was adequate, whereas others stated that insufficient funds were provided to individual branches/programs to attain the appropriate level of inspection capacity.

To what extent have additional resources from the EFB complemented the regular feed ban programming (baseline inspection and regulatory capacity)?

There was considerable variance between planned and actual allocation of EFB funding across three CFIA branches between 2005-06 and 2010-11. As confirmed by the 2009-10 and 2010-11 DPR, some internal stakeholders noted that changing priorities resulted in the reallocation of some EFB funds to other programs within the CFIA.^Footnote 75 As a result, some internal stakeholders indicated that the initial allocation of resources and risk-based prioritization of activities were modified. For example, on-farm inspections were a component of the EFB design. However, on-farm risks were found to be very low and this component of the design was removed. Other activities that were modified include staffing level, frequency and scope of inspection.

Concerns over how these changes might have affected the performance of the EFB were documented.^Footnote 76 However, the rationale for these changes as well as the EFB fund reallocation was not clearly outlined in documents, nor was their potential impact on the level of achievement of the intended EFB outcomes. Thus, it was unclear whether the CFIA had conducted an assessment of whether and how funding reallocation might affect the performance and outcomes of the EFB. The absence of documented rationale also made it difficult to ascertain whether the allocation of original funding to the CFIA for the EFB was more or less than necessary.

Some internal stakeholders reported that some branches/divisions in particular did not receive the funding needed to effectively carry out new or additional EFB-related duties when the EFB regulations came into effect. The lack of necessary funding affected the breadth and depth of their EFB activities (e.g., inability to hire staff, inadequate SRM R&D work, frequency of inspection below the level determined based on risk profiles). It was unclear whether the branch/division-specific funding shortfall resulted because of inadequate planning or communication within the CFIA (i.e., the branch/division did not anticipate or ask for the required level of funding internally) or if the original (and reallocated) EFB funding was inadequate to begin with. Interviews with internal stakeholders indicated that the lack of adequate and coordinated communication to inform all relevant staff about the decision to reallocate EFB funding and the corresponding rationale had resulted in different funding expectations and subjective assessments of the potential impacts of the funding reallocation within the CFIA.

2.4.2 Is the Initiative the most efficient means of achieving its intended objectives?

The efficiency of the EFB relative to alternatives could not be assessed. Opportunities exist to address the challenges and limitations around financial tracking to better assess the efficiency of the EFB going forward.

Are there alternative ways for the EFB to achieve the same results that are more efficient or economical? What would be the cost implications of these alternative approaches?

Because of the limitations of data availability, it was not possible to compare the economy and efficiency of the EFB against alternate delivery models.

How could the cost-effectiveness of the EFB be improved?

There are opportunities to improve contingency planning and accounting procedures within the CFIA to strengthen the funding reallocation process and better track reallocated funds. Efforts in this regard can be coordinated with the work being carried out by the Oversight Committee for BSE Funding, which was established in September, 2011 and chaired by the Executive Director of the Animal Health Directorate.^Footnote 77 Another coordination opportunity exists to address the need for better communication on changing priorities/strategic policy directions/funding reallocation decisions via the CFIA's Consultation Policy and Framework, which is currently being developed.^Footnote 78

2.4.3 How was duplication of efforts managed (or avoided) to ensure effective use of resources within the CFIA and with other jurisdictions for the control of SRM in the food chain?

The CFIA had been successful in leveraging resources and avoiding duplication of activities in relation to the EFB, especially in working with external partners.

Internal and external stakeholders indicated that the CFIA's close collaboration and partnership with EFB stakeholders ensured leveraging of resources and expertise while minimizing duplication and overlapping activities. For example, the CFIA utilized the Beef Value Chain Roundtable (BVCRT) as a vehicle for consultation with industry instead of establishing a new structure to engage industry. The BVCRT was launched by the AAFC prior to the EFB development phase to foster collaborative industry-government action to secure an enduring competitive advantage for Canada in international markets.^Footnote 79 The document review identified that, in facilitating discussions with provincial partners, the CFIA leveraged dedicated staff support from AAFC through the FBTF as AAFC had the responsibility to manage federal funds for disbursal to the provinces for assisting industry to comply with the proposed regulations under the EFB.^Footnote 80 The CFIA also leveraged provincial resources. In consultation with the CFIA, the provinces conduct inspections of provincial abattoirs quarterly, whereas the CFIA is responsible for auditing the provincial plant inspection regime annually to verify compliance with the EFB.^Footnote 81

Some internal stakeholders and the document review^Footnote 82 indicated that the frequency of facility inspection (e.g., abattoirs and rendering establishments) had been tailored to the level of risk to avoid unnecessary oversight or reduce costs. However, some internal stakeholders raised concerns about the availability and accuracy of the data needed to align the allocation of resources to the appropriate level of risk by determining the optimal frequency and scope of inspection for different risk levels. Coordination and communication within the CFIA could be improved, which, in turn, could contribute to better and more appropriate resource planning and utilization.

Conclusions

The EFB is an important component of the Government of Canada's response to BSE, given the ongoing need to reduce the risk of transmission of BSE in the Canadian cattle herd to protect animal and public health and to facilitate market access for Canadian beef and other related products. Given its alignment with long-term federal government priorities and goals as well as current departmental responsibilities (including regulatory responsibilities), the EFB continues to remain relevant to the mandate of the CFIA.

The planning and implementation of the EFB both warranted and benefited from large-scale consultation and outreach, coordination and collaboration efforts and activities involving numerous stakeholders within Canada and some stakeholders outside Canada. The CFIA was successful and effective in bringing together and working with a large number of diverse stakeholders with a view to establishing consensus to develop and implement the necessary EFB regulatory framework and corresponding mechanisms and tools. To date, these immediate intended outcomes of the EFB have largely been met. However, it is too soon to assess the achievement of longer-term outcomes (i.e., the effectiveness of the EFB).

The EFB has faced some considerable challenges since implementation and it is important that the CFIA address those challenges to continue building on its achievements to date. In particular, there is a need to improve communication around the EFB internally and externally to better demonstrate how the two overarching objectives of the EFB (animal and public health, and market access) are interrelated and aligned with the needs, priorities, and mandates of key EFB stakeholders and partners. Such communication efforts also need to clearly highlight the close linkage between product safety and market access within the context of CFIA's mandate. Lastly, the expected contribution of science in advancing SRM testing and disposal options as well as the role of the CFIA in facilitating such activities also need to be articulated.

A wide range of compliance verification and enforcement measures have been strengthened under the EFB. However, performance measurement and management systems as well as budgeting and financial tracking procedures within the CFIA need to be strengthened to improve decision making around the EFB, particularly with respect to assessing the adequacy, effectiveness and efficiency of the implemented compliance verification and enforcement measures. Without such improvements, it will not be possible to demonstrate conclusively that activities and establishments are being inspected at a level appropriate to their risk profiles and that adequate resources are being allocated to support compliance verification and enforcement measures. Both of these elements are critical since the effectiveness of the EFB cannot be fully validated using scientific testing methods. The ongoing challenges related to performance and budget tracking affect all branches/programs involved in the EFB, as well as BSE programming in general. Therefore, addressing them will require coordinated efforts along with senior management commitment.

Moreover, there is currently no cross-sectional committee or group within the CFIA with a clearly defined mandate and authority to coordinate and oversee the post-implementation phase of the EFB. The sound and effective governance principles and structures adopted and utilized during the development and implementation phases of the EFB can be emulated and adapted to create the necessary structure(s) to address the aforementioned challenges. There are also opportunities to build on past and ongoing related efforts within the CFIA, particularly in the areas of coordination, communications and performance measurement and management systems.

Lastly, concerns from CFIA partners and stakeholders (including the regulated parties) with respect to compliance costs and inconsistent application of the EFB regulations might affect the achievement of expected outcomes of the EFB if those concerns are not addressed adequately in collaboration with all EFB partners and stakeholders. See Appendix E for additional information.

Recommendations

Four recommendations have been formulated with a view to ensuring continuous improvement of the CFIA's EFB programming and achievement of the targeted outcomes. Appendix E outlines the alignment of the recommendations with the evaluation findings.

The scope for this evaluation is 2004-2005 to June 2011. The BSE Program Management Committee was established in 2010 and was instructed to report to the BSE Fund Oversight Committee, which was created in June 2011. These two committees were renamed in 2012: the BSE Program Advisory Committee and the BSE Steering Committee, respectively. In September 2011, a new governance structure was put in place at the CFIA and the Animal Health Business Line Committee (AHBLC) was established. The BSE Steering Committee has been leading the BSE program oversight activities. This committee reports to the AHBLC and seeks recommendations from the BSE Program Advisory Committee for all of the BSE programming. The AHBLC makes activity prioritization recommendations to the Policy and Programs Management Committee (PPMC). An assessment of this new governance structure was not included in the scope of this evaluation.

Strengthen oversight, coordination and communication with regard to EFB-related activities, both internally and externally, by:

Recommendation 1

Strengthen the integration of the CFIA's EFB program objectives within the existing BSE committees as directed by the AHBLC or equivalent. As noted before, the BSE Program Management Committee is tasked with ensuring that the CFIA's BSE-related activities for all program elements, including the feed ban, are discussed and developed internally in a coordinated, complimentary complementary and consistent manner. For its part, the BSE Fund Oversight Committee is tasked with ensuring that Agency activities pertaining to the distribution and use of BSE funds are in accordance with what was planned originally as well as developing a long-term strategic action plan to address reallocated funds.

Recommendation 2

Develop and communicate a plan for science-based policies related to SRM in support of the CFIA's EFB program objectives. As noted in Agency documents such as the "Discussion paper for drafting Canada's BSE Strategic Roadmap", the CFIA is committed to ensuring that any changes to its Enhanced BSE Programming (including the EFB) going forward would be based on the latest scientific knowledge. Consequently, it would be important for the CFIA to have access to the most current scientific knowledge with respect to BSE and SRM to ensure that policy and programming decisions reflect sound scientific evidence.

Mitigate the EFB challenges and gaps identified in this evaluation to ensure its effectiveness and efficiency by:

Recommendation 3

Improve performance measurement and management as well as budgeting and financial tracking procedures in relation to the EFB programming. There are related past and ongoing initiatives (e.g., upgrading information management systems within the CFIA, revising the performance measurement framework for BSE programming, and establishing the BSE Fund Oversight Committee) within the CFIA to strengthen tracking and reporting processes and systems for the Enhanced BSE Programming (and the EFB, as one of the components of BSE programming). Senior management commitment and oversight will ensure a sustained and coordinated approach in addressing both BSE and systemic issues that span across branches and programs.

Recommendation 4

Strengthen the planning for the allocation of EFB resources (human and financial) within the current governance process to address adjustments to the originally planned inspection scope and frequency. This would assess and address potential gaps in delivery of the EFB by ensuring that activities and establishments are being inspected at a level appropriate to their risk profiles and that adequate resources are available to each branch/program to support their respective EFB responsibilities.

As part of this, the concerns over inconsistent application of the EFB between federal and provincial establishments/facilities as well as across federal establishments/facilities in different provinces could also be validated and addressed (if necessary). Ultimately, this exercise would place the CFIA in a position to better demonstrate the effectiveness of the EFB through sound and reliable EFB inspection and compliance verification activities despite the limitations of the SRM tests presently available to confirm the removal of SRM from the feed chain.