Evaluation of the CFIA's Food Safety Program Modernization - Part 1
Final Report

PDF (1.18 MB)

List of Abbreviations

AC

Project Advisory Committee

AEB

Audit and Evaluation Branch

AHP

Animal Health and Zoonotics Program

CAHSN

Canadian Animal Health Surveillance Network

CFIA

Canadian Food Inspection Agency

CFSIN

Canadian Food Safety Information Network

CNPHI

Canadian Network for Public Health Intelligence

EC

Evaluation Committee

ESDP

Electronic Services Delivery Platform Project

FIMS

Food Inspection Modernization System

FSMI

Food Safety Modernization Initiative

FSP

Food Safety Program

GTA

Greater Toronto Area

HR

Human Resources Branch

IFIM

Improved Food Inspection Model

IM/IT

Information Management/Information Technology

ISO

International Organization for Standardization

MEL

Modernizing Equipment & Laboratories

MRAP

Management Response and Action Plan

OPS

Operations Branch

PAA

Program Alignment Architecture

PMC

Policy Management Committee

PPB

Policy and Programs Branch

PREP

Pre-Requisite Employment Program

PRP

Plant Resources Program

RPP

Report on Plans and Priorities

SCC

Standards Council of Canada

SFCA

Safe Food for Canadians Act

SFCAP

Safe Food for Canadians Action Plan

SMC

Senior Management Committee

TB

Treasury Board of Canada

TBS

Treasury Board Secretariat of Canada

WG

Working Group

Executive Summary

The Canadian Food Inspection Agency (CFIA or Agency) is a science-based regulatory agency guided by the following strategic outcome: "A safe and accessible food supply and plant and animal resource base" (CFIA, 2014a, p. 6).

The Food Safety Program (FSP) is an established, long-standing program at the CFIA.­ The objectives of the program are to "mitigate risks to public health associated with diseases and other health hazards in the food supply system and to manage food safety emergencies and incidents" (CFIA, 2014a, p. 21). The FSP uses a considerable portion of the Agency's annual resources. For example, from fiscal year 2011-12 through fiscal year 2014-15, the FSP accounted for between 45 and 50 per cent of the CFIA's overall expenditures.

Following the release of the Report of the Independent Investigator into the 2008 Listeriosis Outbreak (the Weatherill Report) in 2009, the Government of Canada committed funds in Budget 2011 to modernize Canada's food safety system (Government of Canada, 2011). As a result, the CFIA developed the Food Safety Modernization Initiative (FSMI) - a suite of eight projects designed to improve the FSP. CFIA recognizes the need for change within the Agency to address risks affecting its operations. These risks include the effectiveness and ability of programs such as FSP to support the overall strategic outcome.

This evaluation, carried out between January 2015 and March 2016, examines the $87.4 million initially allocated for the FSMI between fiscal years 2011-12 and 2014-15Footnote 1 . It was atypical of Government of Canada evaluations, as it did not attempt to directly assess an established and ongoing program.

Key Findings and Recommendations

Key Findings

Relevance: Need, Alignment with Government Priorities, and Alignment with Federal Roles and Responsibilities

Overall, the evaluation found the FSMI to be relevant and necessary for modernizing the FSP. Furthermore, FSMI projects are in line with the recommendations from the Weatherhill report, federal priorities, and the Government of Canada's Blueprint 2020 vision. The evaluation also found all FSMI projects were well designed to meet established program needs and objectives.

These projects represent the beginning of long-term change activities at the Agency; therefore, ongoing efforts will be required to fully realize their intended benefits on CFIA programs. Without this, there is a risk the effectiveness of the initiative's investments will be undermined. Ensuring appropriate levels of effort rests in part on maintaining CFIA staff and external stakeholder buy-in for change.

The following recommendation is meant to establish a culture of change and support Agency program improvements.

Recommendation 1: The Agency should establish and monitor an internal and external communication process to share ongoing information about the FSMI projects and their benefits.

Performance: Achievement of Outputs, Outcomes, and Demonstration of Efficiency and Economy

Despite some delays, all FSMI project activities are progressing along their intended plans and are producing outputs. Measureable outputs produced under the FSMI include:

Of particular concern, the evaluation demonstrates there is a lack of an established and effective means of measuring the influence of FSMI projects and their impacts on the FSP, as well as broader Agency programming. Without an established and effective means for measurement, there is a challenge in establishing the effectiveness of FSMI investments and, therefore, justifying future investments in Agency change initiatives.

The following recommendation is meant to establish the basis for measuring the effectiveness and efficiency of change initiatives affecting the FSP.

Recommendation 2: The Agency should develop and implement a performance measurement strategy to track how FSMI projects are affecting the Food Safety Program. The strategy should include:

1.0 Introduction

The Canadian Food Inspection Agency's (CFIA) 2014 Departmental Evaluation Plan identified a two-part evaluation on the Food Safety Program (FSP), an established and long standing program at the CFIA.Footnote 2 This report presents the findings of the FSP Modernization Evaluation - Part 1, with a focus on the impact of the Food Safety Modernization Initiative (FSMI). This initiative is a suite of eight projects designed to improve the FSP.

The evaluation was conducted in accordance with the Treasury Board (TB) Policy on Evaluation (2009) and its supporting Directive and Standard. The evaluation focused on the relationship between FSMI and FSP, and therefore examined issues of relevance and performance of both.

The evaluation was carried out between April 2015 and March 2016, and examined fiscal years 2011-12 through 2014-15.

1.1 Report outline

The remainder of this report is laid out in the following four sections:

2.0 The Food Safety Program

The CFIA is a science-based regulatory agency guided by the following strategic outcome: "A safe and accessible food supply and plant and animal resource base" (CFIA, 2014a, p. 6). The FSP is one of three main programs contributing to this strategic outcome. The other two programs are the Animal Health and Zoonotics Program and the Plant Resources Program.

These three main programs are supported by two others found under the Agency's Program Alignment Architecture (PAA):

Within the FSP, there are a number of commodity-based sub-programs. These include:

Figure 1 presents the Agency's PAA.

Program Alignment Architecture. Description follows.
Figure 1: CFIA Program Alignment Architecture

The CFIA Program Alignment Architecture is a series of boxes that display the CFIA's strategic outcome, programs and sub-programs. The figure is organized vertically with the CFIA strategic outcome at the top, programs in the middle and sub programs at the bottom.

Strategic Outcome

The strategic outcome box is a pale blue rectangle with the following text inside of it: "Strategic Outcome A safe and accessible food supply and plant and animal resource base"

Program

The program section of the diagram consists of five grey rectangles with the following text written in them from left to right:

  • P 1 - Food Safety Program
  • P 2 - Animal Health and Zoonotics Program
  • P 3 - Plant Resources Program
  • P 4 - International Collaboration and Technical Agreements
  • P 5 - Internal Services

Sub Program

Underneath the "P 1 - Food Safety Program" rectangle are seven blue rectangles, organized vertically, they have the following text written in them from top to bottom:

  • SP 1.1 - Meat and Poultry
  • SP 1.2 - Egg
  • SP 1.3 - Dairy
  • SP 1.4 - Fish and Seafood
  • SP 1.5 - Fresh Fruit and Vegetables
  • SP 1.6 - Processed Products
  • SP 1.7 - Imported and Manufactured Food Products

Underneath the "P 2 - Animal Health and Zoonotics Program" rectangle are three blue rectangles, organized vertically, they have the following text written in them from top to bottom:

  • SP 2.1 - Terrestrial Animal Health
  • SP 2.2 - National Aquatic Animal Health Program
  • SP 2.3 - Feed

Underneath the "P 1.3 - Plant Resources Program" rectangle are four blue rectangles, organized vertically, they have the following text written in them from top to bottom:

  • SP 3.1 - Plant Protection
  • SP 3.2 - Seed
  • SP 3.3 - Fertilizer
  • SP 3.4 - Intellectual Property Rights

The "P 4 - International Collaboration and Technical Agreements" rectangle does not have anything underneath it.

Underneath the "P 5 - Internal Services" rectangle are four blue rectangles, organized vertically, they have the following text written in them from top to bottom:

  • SP 5.1 - Governance and Management Support
  • SP 5.2 - Resource Management
  • SP 5.3 - Asset Management Services

2.1 Objectives

The FSP is an established, long-standing program at the CFIA. Its objectives are to "mitigate risks to public health associated with diseases and other health hazards in the food supply system and to manage food safety emergencies and incidents" (CFIA, 2014a, p. 21).

FSP aims to achieve these objectives by:

Figure 2 presents the FSP logic modelFootnote 3. The model demonstrates how groups of activities under the FSP are meant to influence the Agency's strategic outcome.

Food Safety Program Logic Model. Description follows.
Figure 2: FSP Logic Model

This is a diagram that describes the activities and outcomes of the Food Safety Program. On the left side of the diagram is a legend. The legend has the following text written in it from top to bottom:

  • Strategic Outcome
  • Intermediate Outcomes
  • Immediate Outcomes
  • Activities

The logic model is organized the same as the legend with "Strategic Outcome" at the top and "Activities" at the bottom.

Activities

The Activities section is surrounded by a rectangle with a blue dashed edge. Inside of this rectangle are eight blue rectangles organized horizontally in two rows of four boxes. The bottom four blue rectangles have the following titles from left to right:

  • Regulatory and policy analysis as well as development
  • Program design, advice, and training
  • Science Advice and Laboratory Services
  • Internal management

The top four blue rectangles have the following titles from left to right:

  • Communication and stakeholder engagement
  • Inspection, surveillance, and certification
  • Contingency and preparedness
  • International engagement and standard setting

There is an arrow going from the "Activities" section to the "Immediate Outcomes" section.

Immediate Outcomes

The Immediate Outcomes section is surrounded by a rectangle with a green dashed edge. Inside of this rectangle are five green rectangles organized in horizontally. The five green rectangles have the following titles from left to right:

  • Awareness of health related risks among the Canadian population
  • Awareness and policies, regulations, and legislation among the Canadian public and other stakeholders
  • Compliance with program, policies, requirements, and regulations among stakeholders
  • Preparedness to prevent, address, and manage food related emergencies
  • Contributions to international standards and agreements for hum-related risks

There is an arrow going from the "Immediate Outcomes" section to the "Intermediate Outcomes" section.

Intermediate Outcomes

The Intermediate Outcomes section is surrounded by a rectangle with a green dashed edge. Inside of this rectangle are two green rectangles organized horizontally on top of six green rectangles organized horizontally. The two top green rectangles have the following titles from left to right:

  • International foods are accessible to Canadians
  • International markets are accessible to Canadian food

The six bottom green boxes have the following title from left to right:

  • General public engages in behaviour to maintain food safety
  • Domestic products are compliant with Canadian requirements
  • Import products are complaint with Canadian requirements
  • Canadian standards are recognized internationally
  • Risks to the Canadian food supply are mitigated, minimized, or managed
  • Canadian interests are reflected in science-based international rules, standards, and arrangements

There is an arrow going from the "Import products are complaint with Canadian requirements" rectangle to the "International foods are accessible to Canadians" rectangle.

There is an arrow going from the "Risks to the Canadian food supply are mitigated, minimized, or managed" rectangle to the "International markets are accessible to Canadian food" rectangle.

There is an arrow going from the "Canadian standards are recognized internationally" rectangle to the "International foods are accessible to Canadians" rectangle and the "International markets are accessible to Canadian food" rectangle.

There is an arrow going from the "Intermediate Outcomes" section to the "Strategic Outcome" section.

Strategic Outcome

The Strategic Outcome section is surrounded by a rectangle with a green dashed edge. Inside of this rectangle is a pale blue rectangle with the following text inside of it "A safe and accessible food supply and plant resource base."

2.2 Risks

Like the other CFIA programs, the FSP faces a number of risks to its effectiveness and its ability to support the Agency's strategic outcome. This includes risks related to:

2.3 Stakeholders

The FSP targets a number of stakeholders, including:

The CFIA regularly interacts with these groups, ensuring their perspectives are considered in the development of FSP policies and strategies (CFIA, 2014b, pp. 28-29).

2.4 Resources

The FSP uses a considerable portion of the Agency's annual resources. As Table 1 indicates, between fiscal years 2011-12 and 2014-15, the FSP accounted for between 45 and 50 per cent of the Agency's overall expenditures.

Table 1: FSP and CFIA Overall Expenditures - 2011-12 to 2014-15
Item 2011-12 2012-13 2013-14 2014-15
FSP overall expenditures $328,935,486 $353,600,998 $364,310,525 $421,520,442
CFIA total overall expenditures $737,696,357 $782,055,725 $805,751,653 $848,492,889
% FSP of CFIA overall expenditures 44.6% 45.2% 45.2% 49.7%

Source: (Government of Canada, 2014, 2016)

As Table 2 demonstrates, between fiscal years 2011-12 and 2014-15, approximately half of all Full-Time Equivalents (FTEs) at the CFIA were dedicated to the FSP. During this time period, there was an increase in FSP staffing relative to other programming.

Table 2: FSP and CFIA Staffing FTEs - 2011-12 to 2014-15
Item 2011-12 2012-13 2013-14 2014-15
FSP FTEs 3,238 3,216 3,296 3,250
CFIA total FTEs 6,623 6,446 6,378 6,138
% FSP of CFIA FTEs 48.9% 49.9% 51.7% 52.9%

Source: (Government of Canada, 2013a, 2013b, 2014)

2.5 Supporting change at the CFIA

For more than a decade, the CFIA has recognized the need for change. Changes to Agency operations have been, and continue to be, meant to modernize its servicesfood safety, and are driven by:

The need for change is also emphasized in the CFIA's Long Term Strategic Plan. The plan notes the Agency must incorporate the following across all programming:

The CFIA is also focussed on ensuring change activities align with the Government of Canada's Blueprint 2020 vision, which attempts to develop:

The Royal Assent of the Safe Food for Canadians Act (SFCA) on November 20, 2012, and the regulations to be made thereunder (the Safe Food for Canadians Regulations,) provide the basis for significant change in the food safety environment (CFIA & SFCA, 2012).

The SFCA directly addresses the need for updated legislation and regulations and will, upon fully coming into force, repeal and replace four existing Acts: the Fish Inspection Act, the Meat Inspection Act, the Canadian Agriculture Products Act, and the food related provisions of Consumer Packaging and Labelling Act.

The Act and forthcoming regulations will provide better management of food safety risks, more consistent inspection across all food commodities, greater clarification on established industry requirements, and improved consumer protection. They will also allow the Agency to focus more on prevention (CFIA & SFCA, 2012; Crawford, 2015).

2.6 The Food Safety Modernization Initiative (FSMI)

Following the release of Report of the Independent Investigator into the 2008 Listeriosis Outbreak (the Weatherill Report) in 2009, the Government of Canada committed funds in Budget 2011 to modernize Canada's food safety system (Government of Canada, 2011). As a result, the CFIA developed the FSMI.

The FSMI can be broken down into three main elements:

These three elements can be further broken down into eight distinct projects (sub-initiatives), which collectively support FSP modernization.

Table 3: FSMI Project Descriptions

Element #1: Inspection System Modernization
Project Description
Improved Food Inspection Model (IFIM) Table Note 4 Development of a single approach to food inspection that is consistent in its approaches to food safety risks and non-compliance issues. Inspection across all food commodities will be standardized so that one inspector can perform all necessary activities at a particular establishment.
Verifying Industry Compliance with Health Canada's Revised Listeria Policy Table Note 5 Enhancement of inspection and testing activities, as well as analytical laboratory capacity, to improve Listeria controls in all high-risk ready-to-eat foods. This will result in earlier identification of contamination in the food processing environment, leading to fewer product recalls and fewer high-risk products on the market.
Electronic Services Delivery Platform (ESDP) Creation of a modern, web-based portal that will make the Agency's programs and services accessible electronically to stakeholders. For example, the portal will provide easier access to CFIA regulations, standards, and inspection procedures. It will include an export requirements management tool, allowing export certification information to be electronically exchanged with foreign governments. This will help to facilitate the approval of Canadian commodities before they are shipped to other countries, and will support common domestic business functions at the Agency.
Recruitment and Training of Inspectors Creation of a national approach to inspector recruitment and training that will provide more consistency across all program areas and the Agency's 14 commodity groups. The new approach to recruitment and training is also intended to ensure that inspectors have the skills needed for the Agency's evolving work.

Table Notes

Table Note 4

This project is now referred to as the Integrated Agency Inspection Model (iAIM)

Return to table note 4  referrer

Table Note 5

This project was not examined during the FSP Evaluation Part 1. Listeria was examined under the CFIA's Evaluation of Meat Programs (2016).

Return to table note 5  referrer

Element #2: Enhanced Science Capacity
Project Description
Developing a Laboratory Network Strategy (CFSIN) Table Note 6 Development of a strategy for an integrated food laboratory network. This network will increase the ability of Canada's laboratories to detect and respond to food safety risks and hazards and share the information across food safety authorities.
Modernizing Equipment and Laboratories (MEL) To respond more efficiently to food-borne illnesses and outbreaks, the addition of modern equipment will help laboratories conduct more sensitive and rapid testing. Renovation to two laboratories - one in St. Hyacinthe, Quebec, and another in Scarborough, Ontario - will allow for more effective use of laboratory space for testing and analysis.
Enhancing Laboratory Response Capacity (ELRC) To help achieve earlier detection and faster response to food safety risks and hazards, the number of highly-skilled scientists working in CFIA laboratories will be increased, and new food safety testing methods to more quickly and accurately identify pathogens will be developed. The new personnel will work with international standard-setting organizations to validate new testing methods and maintain proficiency testing and laboratory quality assurance.

Table Notes

Table Note 6

The network developed under this project is now referred to as the Canadian Food Safety Information Network (CFSIN).

Return to table note 6  referrer

Element #3: Improved IM/IT
Project Description
Increased Efficiency through Improved IM/IT This initiative will provide the Agency's staff with up-to-date information, management capabilities, and tools. This involves ensuring that these tools will support the IFIM being developed through the FSMI. Information will be available at the point of inspection and in remote areas, enabling Agency staff to make proactive and risk-based decisions.

Figure 3 presents the FSMI logic model. Developed for the FSP Evaluation - Part 1, the logic model shows project linkages to the collective outcomes of the FSMI. The FSMI logic model is colour-coded to identify:

To the extent possible, the logic model is organized according to the elements noted in Table 3 above. The largest exception involves improved IM/IT. The outcome under this element supports the achievement of outcomes for all other FSMI elements and their associated projects.

Food Safety Modernization Initiative Logic Model. Description follows.
Figure 3: FSMI Logic Model

This is a diagram that describes the activities, outputs and outcomes of the Food Safety Modernization Initiative. On the left side of the diagram is a legend. The legend has the following text written in four vertical grey boxes from top to bottom:

  • Projects/Sub-initiatives
  • Activities
  • Outputs
  • Outcomes

On the top of the diagram is a horizontal legend made up of three grey boxes. These boxes represent the three elements of the FSMI. The three horizontal boxes have the following text written in them from left to right:

  • Enhanced Science Capacity
  • Inspection System Modernization
  • Improved IM/IT

Projects/Sub-initiatives

The projects/sub-initiative section is made up of eight boxes and they are organized horizontally. Seven of the eight boxes are green and one box is blue.

From left to right, the first three boxes are below the "Enhanced Science Capacity" vertical legend box. The following text is written within the boxes from left to right:

  • Developing a Laboratory Network Strategy, in a green box
  • Modernizing Equipment and Laboratories, in a green box
  • Enhanced Laboratory Response Capacity, in a green box

Continuing horizontally from left to right, the next four boxes are under the "Inspection System Modernization" vertical legend box. The following text is written within the boxes from left to right:

  • Verifying Industry Compliance with Health Canada's revised listeria policy, in a blue box
  • Recruitment and training of inspections, in a green box
  • Improved food inspection model, in a green box
  • Electronic service delivery platform (ESDP) , in a green box

Continuing from left to right, the last and eighth box is under the "Improved IM/IT" vertical legend box. The following text is written within the box:

  • Increased efficiency through improved IM/IT, in a green box

Activities

The activities section is made up of fourteen boxes and they are organized horizontally.

From left to right, the first four boxes are below the "Enhanced Science Capacity" vertical legend box. All fourteen boxes are green. The following text is written within the boxes from left to right:

  • Developing a plan for a Laboratory Network Strategy
  • Modernizing laboratory installations and equipment
  • Hire new laboratory staff
  • Develop research papers on rapid testing methods

Continuing horizontally from left to right, the following six boxes are under the "Inspection System Modernization" vertical legend box. The following text is written within the boxes from left to right:

  • Develop "refresher" training program
  • Develop core (PREP) training program
  • Develop national recruitment strategy
  • Develop new food inspection delivery model
  • Identify IM/IT solutions for the new food inspection delivery model
  • Develop ESDP system

Continuing horizontally from left to right, the last four boxes are under the "Improved IM/IT" vertical legend box. The following text is written within the boxes from left to right:

  • Pilot remote inspector IM/IT tools
  • Pilot new and standardized IM/IT tools
  • Develop data standards for information integration
  • Acquire additional data storage and foundational infrastructure

There is an arrow pointing downwards from the "Developing a Laboratory Network Strategy" projects/sub-initiatives box to the "Developing a plan for a Laboratory Network Strategy" activities box.

There is an arrow pointing downwards from the "Modernizing Equipment and Laboratories" project/sub-initiatives box to the "Modernizing laboratory installations and equipment" activities box.

There is an arrow pointing downwards from the "Enhanced Laboratory Response Capacity" to the "Hire new laboratory staff" and the "Develop research papers on rapid testing methods" activities box.

There is an arrow pointing downwards from the "Recruitment and training of inspections" projects/sub-initiatives box to the "Develop "refresher" training program," "Develop core (PREP) training program" and the "Develop national recruitment strategy" boxes.

There is an arrow pointing downwards from the "Improved food inspection model" projects/sub-initiatives box to the "Develop new food inspection delivery model," "Identify IM/IT solutions for the new food inspection delivery model" and "Develop ESDP system" boxes.

There is an arrow pointing downwards from the "Electronic service delivery platform (ESDP)" projects/sub-initiatives box to the "Develop ESDP system" activities box.

There is an arrow pointing downwards from the "Increased efficiency through improved IM/IT" projects/sub-initiatives box to the "Pilot remote inspector IM/IT tools," "Pilot new and standardized IM/IT tools," "Develop data standards for information integration" and "Acquire additional data storage and foundational infrastructure" activities box.

Outputs

The outputs section is made up of twelve boxes and they are organized horizontally in two vertical rows, the top row has ten boxes, and the bottom row has two boxes. Four boxes are blue and eight boxes are green.

From left to right, the first three boxes are below the "Enhanced Science Capacity" vertical legend box. Two boxes are on the top row and one box on the bottom row. The following text is written within the boxes from left to right on the top row:

  • Implementing the laboratory network strategy, in a blue box
  • Validate and embed rapid testing methods into laboratory operations, in a green box

The following text is written within the box on the bottom row:

  • Laboratories share scientific information, in a blue box

Continuing horizontally from left to right, the following seven boxes are under the "Inspection System Modernization" vertical legend box. Six boxes are on the top row, and one box is on the bottom row. One box is blue and six are green. The following text is written within the boxes from left to right on the top row:

  • Collection of listeria samples, in a blue box
  • Implement refresher training program, in a green box
  • Implement core (PREP) training program, in a green box
  • Implement national recruitment strategy, in a green box
  • Implement new food inspection delivery model, in a green box
  • Implement ESDP system, in a green box

The following text is written within the box on the bottom row:

  • Exchange of electronic information between CFIA, industry, and foreign authorities, in a blue box

Continuing horizontally from left to right, the following two boxes are under the "Improved IM/IT" vertical legend box. The following text is written within the boxes from left to right on the top raw:

  • Implement remote inspector IM/IT tools
  • Implement new and standardized IM/IT tools

There is an arrow pointing downwards from the "Develop a plan for a Laboratory Network Strategy" activities box to the "Implementing the laboratory network strategy" outputs box and then an arrow downwards from this outputs box to the "Laboratories share scientific information" outputs box.

There is an arrow pointing downwards from the "Develop research papers on rapid testing methods" to the "Validate and embed rapid testing methods into laboratory operations" outputs box.

There is an arrow pointing left from the "Validate and embed rapid testing methods into laboratory operations" outputs box to the "collection of listeria samples" outputs box.

There is an arrow pointing downwards from the "Verifying industry compliance with Health Canada's revised listeria policy" projects/sub-initiatives box to the "Collection of listeria samples" outputs box.

There is an arrow pointing downwards from the "Develop refresher training program" activities box to the "Implement refresher training program" outputs box.

There is an arrow pointing downwards from the "Develop core (PREP) training program" activities box to the "Implement core (PREP) training program outputs box.

There is an arrow pointing downwards from the "Develop national recruitment strategy" activities box to the "Implement national recruitment strategy" outputs box.

There is an arrow pointing downwards from the "Develop new food inspection delivery model" and "Identify IM/IT solutions for the new food inspection delivery model" activities boxes to the "Implement new food inspection delivery model" outputs box.

There is an arrow pointing downwards from the "Develop ESDP system" activities box to the "Implement ESDP system" outputs box.

There is an arrow pointing from the "Implement new food inspection delivery model" outputs box to the "Implement ESDP system" outputs box and an arrow pointing in the other direction.

There is an arrow pointing downwards from the "Pilot remote inspector IM/IT tools" activities box to the "Implement remote inspector IM/IT tools" outputs box.

There is an arrow pointing downwards from the "Pilot new and standardized IM/IT tools" activities box to the "Implement new and standardized IM/IT tools" outputs box.

Outcomes

The outcomes section is made up of seventeen boxes and they are organized horizontally and in three vertical rows. The top row has eleven boxes organized horizontally, two are blue and nine are yellow. The middle row has five boxes organized horizontally, two are blue and three are yellow. The bottom row has one horizontal box that stretches across the width of the entire diagram, it is yellow.

From left to right, the first four boxes on the top row, and the first box on the middle row are below the "Enhanced Science Capacity" vertical legend box. The following text is written within the boxes from left to right on the top row:

  • More efficient collection and use of scientific information, in a blue box
  • More efficient laboratory instillations and equipment, in a yellow box
  • Greater laboratory capacity, in a yellow box
  • More efficient testing procedures, in a yellow box

The following text is written within the box on the middle row:

  • Improved detection and response, in a yellow box

Continuing horizontally from left to right, the next seven boxes are under the "Inspection System Modernization" vertical legend box. There are four boxes on the top row, one is blue and three are yellow. There are three boxes on the middle row, two are blue and one is yellow.

The following text is written within the boxes from left to right on the top row:

  • Earlier identification of contamination in high-risk RTE non-meat, in a blue box
  • Existing inspectors have consistent inspection skills and knowledge, in a yellow box
  • Newly hired inspectors have consistent inspection skills and knowledge, in a yellow box
  • Consistent and efficient approach for all commodity types, in a yellow box

The following text is written within the boxes from left to right on the middle row:

  • Consistent and more efficient food inspection approach applied across commodity types, in a yellow box
  • Integrated and improved industry access to CFIA services, in a blue box
  • Facilitated pre-clearance decisions and improved market access, in a blue box

Continuing horizontally from left to right, the next four boxes are under the "Improved IM/IT" vertical legend box. There are three boxes on the top row and one box on the middle row. All boxes are yellow.

The following text is written within the boxes from left to right on the top row:

  • Consistent and efficient inspection activities, in a yellow box
  • Tools in place to ensure CFIA program delivery consistency, in a yellow box
  • Improved Agency data collection and data management systems, in a yellow box

The following text is written within the box on the middle row:

  • Improved Agency wide data collection and management, in a yellow box

The box on the bottom outcome row is yellow and is underneath all three vertical legend boxes, the "Enhanced Science Capacity" vertical legend box, the "Inspection System Modernization" box and the "Improved IM/IT" vertical legend box. The text within this box is as follows:

  • Strengthened federal food safety system

There is an arrow pointing downwards from the "Laboratories share scientific information" outputs box to the "More efficient collection and use of scientific information" outcomes box.

There is an arrow pointing downwards from the "modernize laboratory instillations and equipment" activities box to the "More efficient laboratory instillations and equipment" outcomes box.

There is an arrow pointing downwards from the "Hire new laboratory staff" activities box to the "Greater laboratory capacity" Outcomes box.

There is an arrow pointing downwards from the "Validate and embed rapid testing methods into laboratory operations" activities box to the "More efficient testing procedures" outcomes box.

There is an arrow pointing downwards from the "Collection of listeria samples" outputs box to the "Earlier identification of contamination in high-risk RTE non-meat" outcomes box.

There is an arrow pointing downwards from the "More efficient collection and use of scientific information," the "More efficient laboratory instillations and equipment," "Greater laboratory capacity," the "More efficient testing procedures" and the "Earlier identification of contamination in high-risk RTE non-meat" outcomes boxes to the "Improved detection and response" outcomes box.

There is an arrow pointing downwards from the "Implement refresher training program" outputs box to the "Existing inspectors have consistent inspection skills and knowledge" outcomes box.

There is an arrow pointing downwards from the "Implement core (PREP) inspector training program" and the "Implement national recruitment strategy" outputs boxes to the "Newly hired inspectors have consistent inspection skills and knowledge" outcomes box.

There is an arrow pointing downwards from the "Implement new food inspection delivery model" outputs box to the "Consistent and efficient approach to inspection for all commodity" outcomes box.

There is an arrow pointing downwards from the "Exchange of electronic information between CFIA, industry, and foreign authorities" outputs box to the "Integrated and improved industry access to CFIA services" and the "Facilitated preclearance decisions and improved market access" outcomes boxes.

There is an arrow pointing downwards from the "Implement remote inspector IM/IT tools" outputs box to the "Consistent and efficient inspection activities" outcomes box.

There is an arrow pointing downwards from the "Existing inspectors have consistent inspection skills and knowledge," the "Newly hired inspectors have consistent inspection skills and knowledge," the "Consistent and efficient approach to inspection for all commodity," and the "Consistent and efficient inspection activities" outcomes boxes to the "Consistent and more efficient food inspection approach applied across commodity types" Outcomes box.

There is an arrow pointing downwards from the "Implement new and standardized IM/IT tools" outputs box to the "Tools in place to ensure CFIA program delivery consistency" Outcomes box.

There is an arrow pointing downwards from the "Develop data standards for information integration" and the "Acquire additional data storage and foundational infrastructure" activities boxes to the "Improved Agency data collection and data management systems" Outcomes box.

There is an arrow pointing downwards from the "Improved detection and response," the "Consistent and more efficient food inspection approach applied across commodity types," the "Integrated and improved industry access to CFIA services," the "Facilitated preclearance decisions and improved market access," and the "Improved Agency data collection and data management systems" outcome boxes to the "Strengthened federal food safety system" outcomes box.

2.7 FSMI resources

A total of $139.8 million was allocated to the FSMI for fiscal year 2011-12 to fiscal year 2015-16. This includes $40 million that was reallocated from existing CFIA resources. Details on the CFIA internal reallocation are provided in Appendix C.

From the total of $139.8 million, $22.6 million was allocated to verifying industry compliance with Health Canada's Revised Listeria Policy. The Listeria project is not considered in the evaluation. In addition, $3 million was allocated to Health Canada as part of the initiative.

In summary, minus these allocations, the CFIA's portion of FSMI funding was $114.2 million.

Table 4 shows how the $114.2 million was distributed across the seven remaining projects in each of the five fiscal years from 2011-12 to 2015-16. The evaluation focussed on the funding and associated activities undertaken during fiscal years 2011-12 to 2014-15.

Table 4: FSMI Original Funding Allocations (millions)
Activity 2011-2012 2012-2013 2013-2014 2014-2015 2015-2016 Total
Inspection System Modernization
Improved food inspection model (IFIM) $3.0 $5.9 $10.3 $12.4 $5.6 $37.2
Electronic services delivery platform (ESDP) $1.0 $3.1 $6.5 $6.4 $5.5 $22.5
Recruitment and training of inspectors $0.6 $2.4 $5.1 $4.9 $4.9 $17.9
Subtotal $4.6 $11.4 $21.9 $23.7 $16.0 $77.6
Enhanced Science Capacity
Developing a laboratory network strategy $0.2 $1.1 $1.2 - - $2.5
Modernizing equipment and laboratories - $0.5 $2.2 $3.8 $5.4 $11.9
Enhancing laboratory response capacity $0.7 $0.8 $1.3 $1.3 $1.3 $5.4
Subtotal $0.9 $2.4 $4.7 $5.1 $6.7 $19.8
Improved IM/IT
Increased efficiency through improved IM/IT - $3.8 $4.8 $4.1 $4.1 $16.8
Subtotal - $3.8 $4.8 $4.1 $4.1 $16.8
Total $5.5 $17.6 $31.4 $32.9 $26.8 $114.2

3.0 Evaluation Objectives

In accordance with the Treasury Board Policy on Evaluation (2009) and its associated Directive and Standard, the evaluation assessed the following core issues:

This evaluation was atypical of Government of Canada evaluations, as it did not attempt to directly assess an established and ongoing program. Rather, it examined a suite of time-limited projects and investments meant to improve the way in which the FSP - and more broadly, the Agency - operates.

Given the nature of the FSMI and its relationship to both the FSP and broader change at the CFIA, the evaluation also needed to:

Therefore, this evaluation was designed to meet both TB requirements and the information needs of the Agency.

3.1 Evaluation team and support

The CFIA's Evaluation Directorate managed the evaluation and conducted it with the assistance of PRA Inc. The evaluation was guided by an Advisory Committee and a Working Group, which reviewed and provided feedback on the evaluation plan, evaluation matrix, logic model, findings, report and Management Response and Action Plan.

The evaluation team developed and conducted the following:

3.2 The evaluation matrix and FSMI logic model

To guide its work, the evaluation team developed an evaluation matrix, which included a list of the evaluation questions, organized according to the core TB issues (Appendix B). Each question was aligned with indicators and data collection methods. Aligning evaluation questions with the evaluation methods ensured the matrix maintain the evaluation scope. It also helped ensure that the evaluation methods were designed to address all of the identified evaluation questions.

Given the absence of an integrated performance measurement strategy and an approved logic model for FSMI, the evaluation team worked closely with members of the evaluation Working Group to finalize the logic model (Figure 3). The team modified a draft version of an FSMI logic model that was developed at the time of the initiative's inception with information gathered during preliminary discussions with members of the Working Group. This model outlined not only the immediate goals of the FSMI, but also highlighted their intended downstream impacts on Agency operations and ultimately strengthening the food safety system. This model was subsequently used to guide the evaluation, as it presented an overall view of the FSMI's expected impacts.

3.3 Data collection activities

The following data collection activities were conducted as part of the evaluation:

3.3.1 Document and literature review

The document and literature review leveraged existing documentation from within the Agency, along with academic and professional literature related to food safety.

A standardized template was used for the systematic review of this documentation and literature, and facilitated an overall understanding of these works. Information gathered through this review was valuable in addressing nearly all of the evaluation questions. It was particularly important in identifying FSMI project details and impacts, as well as changes to projects over time.

3.3.2 Data review and cost analysis

The data review and cost analysis generated quantitative financial information and, where possible, output and outcome information on FSMI projects. This information provided insight into the efficiency and economy of FSMI expenditures, specifically:

3.3.3 Interviews

Thirty-seven interviews were conducted with representatives from Policy and Programs Branch, Operations Branch, Science Branch, Human Resources Branch, and IM/IT Branch. Interviewees were chosen based on their knowledge about the FSMI and related projects. The interviews were conducted after the document and literature review, in order to supplement existing information with contextual insight and interpretation on the part of program delivery staff and stakeholders. Further, it allowed the evaluation team to identify gaps and inconsistencies in information.

Eight interview guides were developed to facilitate semi-structured interviews - one for individuals associated with each of the seven FSMI projects, and a final guide for individuals involved with the initiative as a whole. These are provided in Appendix D.

3.4 Analysis and reporting

Information from the various data collection activities was analyzed in order to make conclusions regarding each of the questions included in the evaluation matrix. This triangulation process allowed the evaluation to use the strengths of each of the data collection activities to their best possible advantage. It also allowed the team to compare or confirm findings across data collection activities, as well as to supplement and contextualize the findings from one data collection activity to another.

The triangulation completed during the evaluation analysis also provides an opportunity to identify key information gaps. Not all questions identified in the evaluation matrix had an equivalent amount of data or information to support associated conclusions.

3.5 Limitations and challenges

The evaluation limitations and challenges, and the corresponding mitigation strategies, are described in Table 5 below.

Table 5: Evaluation Limitations and Challenges and Mitigating Strategies
Limitations/Challenges Mitigation Strategy Implications
Partially developed logic model - The FSMI had only a partially developed logic model at the start of the evaluation. This made it difficult to develop an initial understanding of the initiative's collective goals. The evaluation team developed an FSMI logic model specifically for the evaluation. The newly developed logic model used for the evaluation outlined the expected outputs and outcomes for the FSMI at its inception.
Ongoing change at the CFIA - Throughout the implementation of the FSMI, the CFIA has been undergoing significant change. This has influenced the FSP. At the same time, the FSMI forms part of these change activities and is meant to affect the way the Agency works overall. During the evaluation, the team took into consideration the FSMI's place within broader change activities at the Agency. The evaluation focused on the FSMI's influence on the Agency's work as a whole and not only on its benefits to the FSP.
The need to examine multiple projects - This evaluation needed to examine multiple projects with differing approaches to management, varying data collection practices, and different amounts of available data. This made it challenging to maintain consistency and manage the scope of the evaluation. The evaluation team worked closely with the Working Group to focus the evaluation on the most relevant aspects of each project. The team then used a findings table in the final report to support aggregate reporting across all projects. A findings table in the FSP Modernization Evaluation - Part 1 provides for the simplification of many of the observations regarding individual FSMI projects and the alignment across FSMI project observations.
Limited information on food safety outcomes - Projects generally tracked their own activities and outputs well, but the evaluation team encountered very limited tracking of project effects on food safety. The evaluation team made recommendations for measuring the influence of FSMI projects and more general Agency change activities on CFIA programming and outcomes. Recommendations from the evaluation extend beyond the FSMI to track the link between projects and the FSP or other Agency programming.
Projects in early stages of development and implementation - Many of the FSMI projects remain at an early stage of implementation, with limited effects on the FSP to date. Rather than leveraging existing outcomes data related to FSP changes, the evaluation team focussed on establishing whether the expected outcomes of the projects appeared likely given the available theory, business cases, best practices, and other evidence. Recommendations from the evaluation focus on ensuring that the Agency has an efficient way of establishing the effects of the FSMI on FSP in the future.

4.0 Findings

The evaluation identified a number of project-specific observations, as well as observations that were similar across projects.

This section uses consolidated findings tables to present findings from the evaluation (Table 6 and Table 7). The intent is to allow readers to quickly identify the similarities and differences among the projects, and to facilitate general statements about the FSMI as a whole. A summary of observations precedes each table.

Observations included in the consolidated findings tables were compiled from information collected through all of the evaluation's data collection activities. A list of the documentation used to support the development of the tables is included in Appendix A.

4.1.1 Need and Objective: There is a continued need for FSP and a demonstrated need for the FSMI.

4.1.2 Project Design: The design of individual FSMI projects is aligned with overall project objectives.

4.1.3 Priorities, roles, and responsibilities: The FSMI supports government-wide and CFIA priorities. It will enhance how the CFIA carries out its activities.

4.2.1 Implementation and Outputs: Despite some delays, project activities are producing their respective outputs, but there is a lack of performance measurement to track the initiative's effects on the FSP.

4.2.1.1 Dependencies: Project delays are largely a reflection of associated FSMI dependencies.

4.2.1.2 Challenges: Communication and stakeholder buy-in are common challenges across all FSMI projects.

4.2.2 Economy: Financial data is in line with project delays, but it is projected most funding will be spent within the five-year timeframe ending in fiscal year 2015-16. Remaining funding has been extended to fiscal year 2017-18.

4.2.2.2 Efficiency: There is minimal evidence to support the efficiency of FSMI projects, in part due to delays; however the fact that all projects were either implemented or are scheduled to be completed without significant overages provides reasonable evidence of efficiency.

4.2.3 Outcomes: Generally, most FSMI initiatives are at too early a stage to report on outcomes. However, there is no evidence of a plan to track to track the initiative's effects on Agency programming.

4.1 Relevance

4.1.1 Need and objectives

Overall, the evaluation found there is a continuing need for the FSP, as it supports the CFIA's strategic outcome and plays a key role under food safety legislation. The need for FSMI supports the need for modernizing FSP. As well, the needs and the objectives of each of the FSMI projects are well aligned. However, the information available about the FSMI projects suggests a lack of or limited emphasis on linking the FSMI objectives with the broader needs for the FSP.

The activities of the FSP strive to ensure Canadians have access to safe food (including imported food), and that the food we export is safe. The continued need for FSP is supported by evidence of the need for a stronger focus on prevention, a stronger focus on compliance, increased efficiency, and improved consistency for FSP. These needs point to broad outcomes, such as modernizing in order to strengthen the FSP, which the FSMI is striving to achieve.

This is supported by the Government of Canada's commitment to strengthen food safety in Canada by allocating funds directly for the purpose of modernizing the food safety system through inspection and science (Government of Canada, 2011). The CFIA's Long Term Strategic Plan and Corporate Risk Profile further support the need for the FSP, and place significant importance on modernizing the program.

While the FSP plays an important role in ensuring safe food in Canada, a number of factors continue to impact the effectiveness of its activities. These include:

The CFIA's need to address these risks supports the relevance of the FSMI, which focusses on inspection effectiveness, scientific capabilities, IM/IT infrastructure, and transparency and leveraging relationships with consumers and industry. Under this initiative, the CFIA is modernizing the FSP through, a stronger focus on prevention and compliance, a citizen-centered service culture, optimizing performance, building capacity, and modernizing tools (CFIA, 2013j).

In many instances it is possible to understand the needs targeted by FSMI projects as the intended individual project outcomes. These projects share similar collective outcomes that support the overarching needs for FSP. For example, the need for a consistent approach to conducting inspections is being addressed by the design and implementation of a single inspection approach and a consistent approach to recruitment and training across all food commodities. Together, these activities address the need for consistency in the approach of the FSP. The relationship between the need for FSP and the need for FSMI, as observed during the evaluation process, underpins the relationship between the implementation of the FSMI projects and their eventual influence on the FSP and further Agency programming.

Table 6 demonstrates that the needs and objectives of each of the FSMI projects are closely aligned. For example, in the case of the Improved Food Inspection Model (IFIM) project, the need for consistency of inspection through a systems-based approach aligns well with the project's objective of shifting to an audit-based inspection ap proach that is based on the implementation of hazard control plans.

4.1.2 Project Design

The evaluation found individual FSMI projects were designed to meet overall objectives and were based on existing evidence, theories and experience.

In some cases, as with ESDP and IFIM, the Agency has looked to other jurisdictions, such as the United States, the Netherlands, Australia, and New Zealand for evidence of project similarity and success. These reviews examined not only the applicability of other IM/IT solutions and approaches, but also the success of alternative approaches to inspection in other jurisdictions. In the case of IFIM, proven industry practices have also been incorporated into project design. This includes moving toward using established health and safety plan approaches, as well as International Organization for Standardization (ISO) standards for control procedures.

Others, like the FSIN project, have leveraged concepts from similar projects that have been implemented domestically. The FSIN project is a clear example of how theory provides a strong justification for the project. Specifically, the project's approach rests on the idea that building a laboratory network will allow better information sharing, will lead to better and more readily-available food safety data. This will support a more preventative approach to food safety. In certain cases, such as with the Modernizing Equipment and Laboratories (MEL) project, activities represent an extension of the ongoing work of the Agency, allowing the project to leverage this existing experience. It is clear that a strong theoretical justification for the work remains.

Table 6 indicates that individual FSMI projects have all leveraged existing evidence, theories, or experience bases in their design.

4.1.3 Priorities, roles, and responsibilities

The evaluation found FSMI contributes to a variety of federal government and Agency priorities such as ensuring a healthy Canadian population and the Government of Canada's Blueprint 2020 vision. Evidence from the evaluation also suggests the FSMI could improve the FSP's ability to support the Agency's strategic outcome. The mandate for providing safe food will not change; however, under FSMI the way inspectors conduct their business will be different.

Overall, FSMI projects align with federal priorities. For instance, all projects support the federal priority of a healthy Canadian population. Many FSMI projects also support priorities outlined in the Government's Blueprint 2020 vision, which calls for a modern and service-oriented federal government. For example, the investments in IM/IT and ESDP represent a significant step toward better use of technology and improving service delivery. Individually and collectively, FSMI projects directly support the Agency's ability to meet its strategic outcome.

The FSMI projects will not substantially change the Agency's roles and responsibilities with respect to food safety; however, some initiatives may result in subtle changes in the way the CFIA interacts with industry, and scientific and international partners. For example, inspection modernization is intended to provide greater clarification around the role of industry vs. the role of the CFIA with respect to food safety. Industry has the principal role in food safety, while the CFIA is responsible for overseeing industry compliance with regulations.

Table 6 presents observations regarding the need, federal government and Agency priorities, project objectives and project design that support the FSP and its modernization through FSMI. This information supports the analysis presented in Section 4.1.

Table 6: Relevance
Description Improved Food Inspection Model (IFIM) Electronic Services Delivery Platform (ESDP) Recruitment and Training Information Management and Information Technology (IM/IT) Developing a Laboratory Network Strategy (CFSIN) Table Note 7 Modernizing Equipment and Laboratories (MEL) Enhanced Laboratory Response Capacity (ELRC)

Need - This describes how each of the individual FSMI projects is intended to strengthen the FSP.

Analysis: Section 4.1.1

  • A stronger focus on prevention.
  • A stronger focus on compliance.
  • A systems- based approach to inspection.
  • Ensuring inspection consistency through a single inspection approach across commodities.
  • Reducing overlap, duplication, and financial burden for industry.
  • Maintaining international obligations and more readily adapting to emerging global and scientific trends.
  • Enhancing service delivery, optimizing performance, and increasing transparency.
  • Moving away from a paper-based record keeping system.
  • Improving inspection delivery service and information exchange with stakeholders.
  • Moving away from a system of independent, commodity-specific inspector training.
  • Optimizing diverse talent supported by modern electronic tools for frontline inspectors.
  • Consistent and standardized training and competencies for inspectorate across Areas and commodities.
  • Targeted recruitment and retention.
  • Timely and continued training in order to keep up with operational changes and an evolving industry.
  • Optimizing program performance.
  • Providing electronic tools.
  • Access to CFIA's secure network for Industry and trading partners.
  • Access to the CFIA network from remote areas for CFIA staff.
  • Increasing speed of connectivity.
  • Increasing the volume of accessible information .
  • Increasing the analytical capabilities of the Agency.
  • Enabling electronic services for industry/ trading partners.
  • A stronger focus on prevention and responsiveness.
  • Enhancing overall capacity and information sharing.
  • Maintaining and enhancing scientific capabilities.
  • Leveraging advances in technology and science.
  • Managing the risks of contaminants and chemical effects in food.
  • Supporting an integrated and multi- jurisdictional food safety surveillance network .
  • Leveraging advances in technology and science.
  • Managing risks associated with new foods.
  • Maintaining and enhancing scientific capabilities.
  • Improving quality management controls within laboratories.
  • Maintaining and replacing equipment near or beyond its life expectancy.
  • Improving food safety through additional science capacity.
  • Managing risks associated with new foods.
  • Maintaining and enhancing scientific capabilities.
  • Leveraging advances in technology and science.
  • Managing an increased volume of global trade.
  • Implementing mandatory scientific and technical training across all food commodities.

Objectives – This row presents the objectives of individual FSMI projects.

Analysis: Section 4.1.1

  • Develop a consistent (single food) inspection approach across all food commodities.
  • Reduce the need for multiple inspections at a single facility.
  • Use an audit-based inspection approach based on hazard control plans.
  • Ensure clear and consistent inspection requirements for industry.
  • Develop a web-based electronic portal for industry to access Agency services such as registration.
  • Ongoing collection and tracking of inspection data and laboratory samples.
  • Enable managers to track inspection activity and to target inspection resource allocation.
  • Make inspector worksheets and daily tasks directly accessible to inspectors in the field.
  • Leverage existing training material to develop an inspector training curriculum aligned with the Agency's new inspection approach.
  • Build a mechanism to support the new consistent training curriculum.
  • Provide consistent training to CFIA inspectors (new and existing) and supervisors that reflects new competencies required by the new inspection approach.
  • Foster a new operational culture.
  • Improve data management through data consolidation.
  • Provide inspectors with tools to support their work and connectivity for remote areas.
  • Develop a plan for a network that would connect/ centralize information from food safety laboratories across Canada.
  • Improve facilities and equipment at Scarborough and St. Hyacinthe laboratories.
  • Purchase additional laboratory equipment to improve response time and laboratory capability.
  • Increase the number of highly-skilled scientists working at CFIA food safety laboratories.
  • Collaborate with academia and universities to enhance knowledge of new technologies and scientific methods.
  • Develop new testing methods.

Design foundations – This row provides information on how individual FSMI projects were designed and developed.

Analysis: Section 4.1.2

  • Established Hazard Analysis and Critical Control Point (HACCP) and International Organization of Standardization (ISO) standards.
  • The successful use of internationally recognized standards for inspection from other jurisdictions.
  • Commercially available off-the-shelf software products with associated with industry standard business processes.
  • Existing CFIA meat processing inspection curriculum.
  • International Food Protection Training Institute in the United States curriculum.
  • Commercially available off-the-shelf software packages with industry standard business processes.
  • The Canadian Animal Health Surveillance Network, and the Canadian Public Health Network as a basis for network planning.
  • Pan- Canadian food safety community and concepts, processes and mechanisms to support a country wide network.
  • Experience in ongoing equipment purchases for the laboratory equipment purchases.
  • Formal assessment of alternative renovation approaches for the two laboratory renovations.
  • Enhancements of ongoing science modernization activities at the Agency.

Alignment with Federal priorities – This row demonstrates how individual FSMI projects link with both federal and CFIA priorities.

Analysis: Section 4.1.3

  • A Canadian population that is healthy.
  • Ensuring that Canadians have access to safe food.
  • Blueprint 2020's goal of a whole government approach to service delivery.
  • A Canadian population that is healthy.
  • Ensuring that Canadians have access to safe food.
  • Blueprint 2020's goal of an open and networked environment that engages stakeholders.
  • Blueprint 2020's goal of a modern workplace that leverages technology.
  • A Canadian population that is healthy.
  • Ensuring that Canadians have access to safe food.
  • Blueprint 2020's goal of a capable, confident, and high- performing workforce.
  • A Canadian population that is healthy.
  • Ensuring that Canadians have access to safe food.
  • Blueprint 2020's goal of a modern workplace that leverages technology.
  • A Canadian population that is healthy.
  • Ensuring that Canadians have access to safe food.
  • Blueprint 2020's goal of a whole government approach to service delivery, an open and networked environment that engages stakeholders.
  • A Canadian population that is healthy.
  • Ensuring that Canadians have access to safe food.
  • Blueprint 2020's goal of a modern workplace that has smarter new technologies.
  • A Canadian population that is healthy.
  • Ensuring that Canadians have access to safe food.
  • Blueprint 2020's goal of a modern workplace that leverages technology.
  • Blueprint 2020's goal of a capable, confident, and high- performing workforce.

Roles and responsibilities and strategic outcome – This row links individual FSMI projects to the Agency's strategic outcome, and describes how projects will impact stakeholder relations.

Analysis: Section 4.1.3

  • Improved ability to meet the strategic outcome.
  • Improved clarity between the Agency and industry with regards to inspection requirements.
  • Improved ability to meet the strategic outcome.
  • Improved working relationship with existing industry stakeholders and international partners.
  • Improved ability to meet the strategic outcome.
  • Improved ability to meet the strategic outcome.
  • Improved ability to meet the strategic outcome.
  • Improved working relationship with existing laboratory and scientific partners.
  • Improved ability to meet the strategic outcome.
  • Improved ability to meet the strategic outcome.

Table Notes

Table Note 7

Developing a laboratory network strategy project is now known as the Canadian Food Safety Information Network

Return to table note 7  referrer

4.2 Performance

4.2.1 Implementation and outputs

The evaluation found that despite some delays, all project activities are progressing along their intended plans and are producing outputs. The alignment between individual project designs and their expected outcomes suggest that projects will eventually meet their outcomes. From the analysis the evaluation infers that delayed projects will be successfully and fully implemented as planned. However, the evaluation did not find evidence of performance measures to track how individual projects will impact the FSP once they are fully implemented.

There were a number of activities planned under each of the FSMI projects. For example, in the case of the IFIM project, planned activities included:

Progress was made in all FSMI projects, and measurable outputs were produced. For example:

Certain aspects of other projects have yet to be completed. For example, the design and implementation of ESDP has been delayed by about two years. As well, IFIM - now being expanded to include commodities beyond the FSP - was only in the early stages of implementationFootnote 9 at the time of the evaluation. The standardized inspection approach had been validated in 125 facilities across the fish, feed, dairy, and greenhouse sectors, and the Agency was in the process of implementing it nationally for those commodities. Subsequent sectors are expected to follow a similar pattern, until all commodity areas have been transitioned over to the common inspection approach.

4.2.1.1 Dependencies

The evaluation found project delays to be in part a reflection of the number of dependencies associated with the FSMI projects.

Delays in the implementation of the IFIM [which includes its IT component Food Inspection Modernization System (FIMS)] and the ESDP are good examples of projects being interdependent.

The FIMS and the ESDP are closely related, and progress in one is often dependent on the other. This dependency led to the decision to merge these two projects under the FSMI. However, both also had associated dependencies outside the scope of the FSMI. For example, effective implementation of the IFIM is supported by the regulatory change underway at the Agency.

Other dependencies, such as buy-in and participation with respect to enhancing science capacity, had fewer consequences, as projects are on track to be completed within the original five-year timeframe. For example, the MEL project was dependent on approval by Health Canada for extra space, the capital equipment procurement process, and PWGSC project management.

4.2.1.2 Challenges

The evaluation found communication and stakeholder buy-in to be common challenges for the FSMI projects. This poses difficulties around fostering a culture of change and ensuring an integrated and measurable approach to support the momentum of long-term project and program goals.

Interviewees were asked to speak to the challenges faced by each of the FSMI projects. Many of the challenges identified in the consolidated findings table are common across other ongoing change activities within the CFIA. For example, common challenges include:

These challenges, along with project delays and observations made by interviewees during the evaluation, point to the need for:

Building momentum appears to be an important aspect of successful change activities, and the need to do so appears even more acute given implementation delays in some of the FSMI projects noted above.

Table 7 presents observations regarding the implementation, production of outputs, achievement of outcomes, and efficiency of FSMI projects. This information supports the analysis presented in Section 4.2.

Table 7: Performance
Description Improved Food Inspection Model (IFIM) Electronic Service Delivery Platform (ESDP) Recruitment & Training Information Management & Information Technology (IM/IT) Developing a Laboratory Network Strategy (CFSIN) Table Note 10 Modernizing Equipment & Laboratories (MEL) Enhanced Laboratory Response Capacity (ELRC)

Planned activities - This row described the planned activities for individual FSMI projects. These activities link directly to the needs and objectives of FSMI.

Analysis: Section 4.2.1

  • Identify common food safety objectives, strategies, and processes.
  • Develop a strategy and engage internal and external stakeholders and subject matter experts.
  • Complete draft improved inspection delivery model.
  • Explore IM/IT options and solutions to support the inspection business process.
  • Develop preliminary project plan.
  • Seek expenditure authority and begin execution of IM/IT component.
  • Develop implementation strategy and implement the improved inspection delivery model.
  • Completion of supporting IM/IT solutions.
  • Perform a project risk and complexity assessment.
  • Validate business requirements, including IM/IT infrastructure.
  • Develop preliminary project plan.
  • Develop detailed project management plan and seek expenditure authority.
  • Develop ESDP system.
  • Inform industry of new system.
  • Train staff on ESDP.
  • Adapt existing training materials into a six-week training program.
  • Build mechanisms for implementing core and refresher training.
  • Update six-week core training program (PREP) to reflect the competencies required by the improved inspection delivery model and all new and existing inspectors will receive training.
  • Core training delivered to new inspection staff: ongoing.
  • Refresher training to existing staff: ongoing.
  • Access e-learning specialists and contractors and ensure capacity to support demand for online training and learning.
  • Initial planning and implementation of technology foundation pieces for increased connectivity; more modern tools; clarification of required support for new inspection model, and business foundations.
  • Negotiate and acquire additional data storage and backup.
  • Improve data and information management capabilities.
  • Pilot technology that can be implemented immediately.
  • Deploy new tools and devices.
  • Implement plan for ever- greening of technology and business foundations, given continual advances in technology.
  • Assemble small team and work in collaboration with partners to establish support for the creation and operation of a network.
  • Explore with experts the concepts, processes, and mechanisms available to conduct a lab systems analysis.
  • Engage IM/IT.
  • Analyse laboratory systems with partners: examine and profile potential Canadian laboratories as contributors for the Laboratory Network across food safety authorities.
  • Develop a strategy by combining common vision and value proposition.
  • Develop a plan for a food safety information network initiative.
  • Consultations to undertake a functional program assessment and options analysis.
  • Formal selection process for identifying the laboratory re-design and purchase decisions.
  • Purchase new microbiology equipment.
  • Complete construction.
  • Hire new scientists to respond to demands for food safety testing.
  • Develop academic and university partnerships to enhance training and knowledge of new technologies and scientific methods.
  • Develop research reports on novel rapid testing methods.

Measurable outputs produced - This row provides evidence of outputs achieved by individual FSMI projects.

Analysis: Section 4.2.1

  • Completed new inspection model - single food program.
  • Wave 1 implementation of the IFIM underway with limited mobile tools support.
  • System planning complete with development and implementation forecasted for 2017 - 2018 completion.
  • Core and refresher training implemented.
  • Implementation of supervisory schools.
  • Created IM/IT business function.
  • Enhanced data storage, backup, and information management capabilities.
  • Put in place remote hardware tools to support IFIM.
  • Completed plan for FSIN.
  • Laboratory renovations set for completion in March 2016.
  • Completed planned equipment purchases.
  • Completed planned hiring.
  • Testing development ongoing.

Dependencies - This row provides evidence of what each project's success is dependent on.

Analysis: Section 4.2.1.1

  • Development and implementation of electronic workbooks on handheld devices and remote connectivity for inspectorate on-site access.
  • ESDP capacity and capability to house inspection data and target inspection need.
  • Implementation of new food regulations to support the IFIM.
  • Development and implementation of the IFIM.
  • Development of core IM/IT functionality and business processes.
  • The roll out (waves) of the new inspection model.
  • Development of an enhanced IM/IT business function.
  • Buy-in and participation by provincial and territorial partners.
  • Participation from Co-location partner in shared departmental facilities.
  • Ongoing technology change and improvements in scientific approaches and testing procedures.

Challenges to project success - This row describes individual project challenges.

Analysis: Section 4.2.1.2

  • Significant project management requirements.
  • Reliance on other FSMI project completion and compatibility..
  • IM/IT privacy and security standards.
  • Acquiring buy-in from all inspectors and Agency staff.
  • Implementation of new regulations to better support the requirements of the new inspection model - e.g., audit-based recordkeeping by industry.
  • Maintaining the necessary momentum to fully implement the model.
  • Implementation is a long-term activity with success affected by attrition among key staff present during the IFIM development.
  • Expectation that ESDP could be developed independently of without underlying processes - e.g., IFIM.
  • Compliance with modern standards of privacy, business continuity.
  • Need for a continual working relationship with other governments regarding acceptance of ESDP system as information transfer system for international trade.
  • Integration of ESDP with other business lines.
  • Scope creep Table Note 11 during project implementation.
  • Business processes to appropriately design elements of ESDP.
  • Integration of work with other projects such as IFIM.
  • Confusion between the ESDP "project" and ESDP "platform".
  • Adopting Agency-wide culture change to support transformation.
  • Developing a new hiring process without additional funding for hiring.
  • Releasing inspectors from duties to obtain training - all levels.
  • Integrating previous work with new systems - LMS with ESDP.
  • Lacking an enhanced IM/IT business function at the start of the project.
  • Getting buy-in from all of the necessary stakeholders.
  • Using non-custodial space for laboratories and its impact on renovation planning.
  • Continual modernization activity.

Measureable project outcomes - This row provides evidence of FSMI outcomes.

Analysis: Section 4.2.3

  • None measured to date.
  • None measured to date.
  • Measured skill consistency among new and existing inspectors.
  • None measured to date.
  • None measured to date.
  • None measured to date.
  • None measured to date.

Forecasted project outcomes - This row outlines the expected outcomes of individual FSMI projects based on their respective needs, objectives and outputs.

Analysis: Section 4.2.3

  • Consistent and efficient approach to inspection for all commodities.
  • Improved information exchange between CFIA and stakeholders.
  • Support for IFIM implementation (all waves).
  • Consistent skills among new and existing CFIA inspectors.
  • Consistent skills among CFIA inspector supervisors.
  • Tools in place to support consistent program delivery at the CFIA.
  • Improved CFIA data collection and management systems.
  • More efficient collection, collaboration, and use of scientific information across Canadian laboratories.
  • More efficient laboratory installations and equipment.
  • More efficient testing procedures.
  • Greater laboratory capacity.

Measured influence on FSP - This row outlines the lack of performance measurement for FSMI benefits on FSP.

Analysis: Section 4.2.3

  • None measured to date.
  • None measured to date.
  • None measured to date.
  • None measured to date.
  • None measured to date.
  • None measured to date.
  • None measured to date.

Expected Influence on FSP - This row outlines the forecasted benefits of FSMI projects on the FSP.

Analysis: Section 4.2.3

  • Consistent and efficient approach to inspection across all commodities - single food inspection system.
  • Strengthened food safety system.
  • Consistent and efficient approach to inspection across all commodities.
  • Integrated and improved industry access to CFIA services.
  • Facilitated pre-clearance decisions and improved market access.
  • Strengthened food safety system.
  • Consistent and efficient approach to inspection across all commodities - single food inspection system.
  • Strengthened food safety system.
  • Improved Agency-wide data collection and management.
  • Consistent CFIA program delivery.
  • Strengthened food safety system.
  • Improved detection and response.
  • Integrated and improved information sharing network.
  • Strengthened food safety system.
  • Improved detection and response.
  • Strengthened food safety system.
  • Improved detection and response.
  • Strengthened food safety system.

Need beyond the FSMI - This row provides evidence of on-going and long-term FSMI project goals.

Analysis: Section 4.2.3

  • Expansion beyond the food program.
  • Expansion beyond the food program.
  • Integration of other functionality - Agency-wide.
  • Continued recruitment and training development.
  • Continued culture development.
  • Integration of CFIA training and recruitment needs with national educational programming.
  • Ongoing IM/IT development.
  • Expansion beyond the food program.
  • Expansion beyond the food program.
  • Expansion beyond the food program.

Table Notes

Table Note 10

Developing a laboratory network strategy project is now known as the Canadian Food Safety Information Network

Return to table note 10  referrer

Table Note 11

Scope creep is defined as "the tendency of a project to include more tasks than originally specified which may lead to higher project costs and/or possible missed deadlines"

Return to table note 11  referrer

4.2.2 Efficiency and economy

4.2.2.1 Economy

Financial data collected during the evaluation supports the evidence of delays in initial project implementation, as expenditure figures are below end of year/adjusted budgets during the early fiscal years of the FSMI.

The evaluation also found funding for most FSMI projects is anticipated to be spent within the five-year project timeframe ending fiscal year 2015-16; remaining funding has been extended until fiscal year 2017-18.

Table 8 (below) provides information on budgets and expenditures, by project, for the four initially planned years of the FSMI and forecasts subsequent years. The figures include:

The reinvestment of specific lapsed (the amount the variance captures) initiative funding back into the initiative is an Agency decision, not a TB requirement. These funds are pooled with other Agency variances and carried forward into subsequent years based on annual CFIA priorities and senior management decision making.

It is important to note FSMI allocations provided by TB were not considered fenced funding. The Agency independently decided to fence all funds associated with the FSMI, in order to track expenditures over time and ensure those funds remained within the initiative. Decisions to reallocate funding are made through the CFIA's governance structure. Since the Agency elected to track all FSMI funding, the overall level of funding for the initiative remained largely unchanged, despite changes to individual project allocations. The ESDP project, however, was tracked slightly differently, as it fit the federal government's requirements for tracking under the ePMFFootnote 12 structure.

Table 8 demonstrates funding allocations for the FSMI. This information supports the analysis presented in Section 4.2.2.

Table 8: FSMI Funding Allocations - Overal Snapshot

I. Fiscal Years 2011-2012 to 2014-2015
II. Project III. Initial Funding Allocation IV. Actual Budget V. Expenditures VI. Variance (Variance is calculated by subtracting expenditures for a given year from the actual budget in the same year.)
IFIM $13,653,537 $18,099,031 $18,509,330 ($410,299)
(2.3%)
ESDP (FIMS) $34,946,463 $16,969,065 $10,431,681 $6,537,384
38.5%
Recruitment & Training $13,000,000 $13,202,666 $6,233,591 $6,969,075
52.8%
IM/IT $12,700,000 $10,444,438 $8,409,507 $2,034,931
19.5%
FSIN $2,500,000 $2,907,200 $2,235,669 $671,531
23.1%
MEL $6,500,000 $7,112,000 $3,378,427 $3,733,573
52.5%
ELRC $4,100,000 $4,100,000 $3,921,551 $178,449
4.4%
Initial 4 Year Sub-total $87,400,00 $72,834,400 $53,119,756 $19,714,644
27.1%
Less Authorities Carried Forward to next Fiscal year ($16,647,499)
Actual 4 Year Sub-total $56,186,901 $53,119,756 $3,067,145
5.5%
I. Fiscal Years 2015-2016
II. Project III. Initial Funding Allocation IV. Actual Budget V. Expenditures VI. Variance (Variance is calculated by subtracting expenditures for a given year from the actual budget in the same year.)
IFIM $2,558,000 $2,558,000
ESDP (FIMS) $8,542,000 $9,385,815
Recruitment & Training $4,900,000 $4,900,000
IM/IT $4,100,000 $3,954,541
FSIN $0 $0
MEL $5,400,000 $8,643,400
ELRC $5,400,000 $1,300,000
2015-16 Total $26,800,000 $30,741,756 TBD TBD
I. Fiscal Years 2016-17 to 2017-18
I. Fiscal Years II. Project III. Initial Funding Allocation IV. Actual Budget V. Expenditures VI. Variance (Variance is calculated by subtracting expenditures for a given year from the actual budget in the same year.)
Funding transferred to SSC responsibility $0 $3,187,409
Reprofiled
to
2016-2017
ESDP + FIMS $0 $21,377,743
Reprofiled
to
2016-2017
IM/IT $0 ($37,172)
Reprofiled
to
2017-2018
ESDP + FIMS $0 $2,743,363
Reprofiled
to
2017-2018
IM/IT $0 ($29,016)
I. Fiscal Years II. Project III. Initial Funding Allocation IV. Actual Budget V. Expenditures VI. Variance (Variance is calculated by subtracting expenditures for a given year from the actual budget in the same year.)
2011-12 to 2017-18 Total Initiative $114,200,000 $114,170,984 TBD TBD

Source: Corporate Management Branch, CFIA

Table 8 shows most FSMI projects are expected to spend their adjusted funding during the five-year period; however, some notable exceptions exist.

Recruitment and Training, for example, had a total spending variance of 52.8 per cent during fiscal years 2011-12 through 2014-15. This is a result of an early end to the planned refresher programming, along with an interruption to the planned recruitment and training activities due to delays in the implementation of IFIM. That project is currently implementing Wave 1 of a multi-wave roll out, and training needs to be aligned with the final operational model. At the moment, it does not appear that much of the variance in funding for Recruitment and Training will be reallocated to future years of the project; however, it will remain within the FSMI.

Another project with significant variances (38.5 per cent) over the initial four years was ESDP (FIMS). The merger of ESDP and FIMS in 2013 may have contributed to this. Delays in ESDP implementation have also resulted in a significant portion of its budget being re-profiled into later fiscal years through 2017-18. These re-profiled amounts are currently forecasted to be spent within that timeframe.

The IM/IT variance of 19.5 per cent is forecasted to be spent in the fifth year of the FSMI. In addition, reallocations from the IM/IT project took place as certain IM/IT functions were consolidated under Shared Services Canada (SSC), as a change in government structured responsibilities occurred during the FSMI period.

The development of a laboratory network strategy - although initially planned to be completed in three years - had an overall variance of 23.1 per cent for the four-year period. The funds have been reinvested in the project for fiscal year 2015-16.

Finally, there was about $3 million in variances (52.5 per cent) associated with the modernizing laboratories and equipment project during the initial four years of FSMI. However, much of this funding is forecasted to be reallocated to the project for the 2015-16 fiscal year.

In summary, changes in the funding allocations for the FSMI, from 2011-12 to 2014-15, came through either re-profiling or carry forward of funding based on senior management decisions. While the overall variance shown in Table 8, Column VI, indicates 27.1 per cent of FSMI funds were not spent, it does not take into account this funding that was authorized to be carried forward to the next fiscal year. When you consider the authorized carried forward amount of $16,647,499 (Column IV), the overall variance is reduced to 5.5 per cent. If the funds forecasted fiscal year 2015-16 are spent as planned, close to 95 per cent of the budgeted FSMI funds will have been spent.

Considering re-profiled amounts that were pushed out to fiscal years 2016-17 and 2017-18, along funding transferred to SSC, it is possible to forecast the $114.2 million initially allocated for the five-year FSMI project will be spent.

4.2.2.2 Efficiency

The evaluation found it difficult to establish efficiency for FSMI project expenditures. However, the fact that most projects were either implemented or are scheduled to be completed as planned without significant overages provides reasonable evidence of efficiency.

It is difficult to establish efficiency of project expenditures under FSMI due to lack of a similar, fully-costed and implemented alternative to each of the projects. Even projects that examined very close alternatives at their early stages - such the laboratory modernization and equipment renovation projects - involved estimates for planning purposes, rather than implemented alternatives. Other projects, like the IM/IT and ESDP projects, looked at established best practices, similar past work and alternative strategies prior to implementation.

4.2.3 Outcomes

The evaluation found that while measures were in place to track the immediate outcomes of FSMI projects data, outcomes have largely not been collected as many of the projects are at too early a stage to expect outcomes to be realized.

The evaluation was also unable to find evidence of a clear plan to track the initiative's effects on Agency programming in the future indicating a lack of performance measurement in place for measuring the outcomes of FSMI on the FSP and the overall Agency.

When examining the FSMI logic model, immediate outcomes relate more directly to individual projects. However, the ultimate intent of all FSMI projects is to enhance the FSP and ultimately to strengthen the food safety system.

With the exception of the Recruitment and Training project, the evaluation was unable to find evidence of measured outcomes for the projects examined. In the case of recruitment and training, concerted efforts were taken to demonstrate core and refresher training among inspectors, along with the acquisition of skills, resulted in successful inspection work. Evidence also suggests that more such tracking is planned for the future.

The lack of a performance measurement strategy is particularly problematic, given many of the Agency's current FSP performance measures relate to stakeholder compliance with program requirements. Without additional information, these will be insufficient to assess the impact of the FSMI on the FSP and broader Agency programming.

For example, compliance measures used to track program requirements under the previous inspection model are different from compliance measures tracking program requirements under the new inspection model. Since these two forms of measuring compliance are different, it will not be possible to compare them in order to determine which form of compliance measurement is more effective or efficient. The reality is, compliance measurement is being modernized to align with the new audit-based inspection approach that is risk-based and designed to enhance the Agency's inspection system.

The Agency's plan to use a risk-based approach to determine inspection activities will further complicate the use of compliance measures as a tool for understanding improvements in food-related risks to the Canadian public. Under this model, appropriate targeting of resources to high-risk producers and industries will likely be associated with identifying higher levels of non-compliance. This does not necessarily reflect a systematic increase in food safety risk.

Both of these complications suggest more direct measures of food safety risks and other aspects of CFIA program performance will be necessary to determine FSMI effectiveness and efficiency.

The need for better performance measurement becomes even more acute when examining the need for each FSMI project outside the context of the initiative itself. As Table 7 notes, there are clear plans to build on all of the FSMI projects to support change beyond the FSP. In all likelihood, this will require not only concerted effort on the part of the Agency, but substantial additional funding. Without the means to establish the effectiveness of past change initiatives such as the FSMI, accessing future funding could become difficult.

5.0 Conclusions

This evaluation found individual projects under the FSMI were well designed to meet established program needs and objectives. Despite some delays, the performance of the FSMI is upheld by the evidence that individual projects are on track to be completed with only minor setbacks.

These projects represent the beginning of long-term change activities at the Agency; therefore, ongoing efforts will be required to fully realize their intended benefits on CFIA programs, including the FSP. Without this, the effectiveness of these investments could be undermined.

Perhaps more importantly, there is a lack of a performance measurement approach for FSMI projects and their impacts on the FSP, as well as broader Agency programming. This could pose challenges in establishing the effectiveness of FSMI investments and, therefore, justifying future investments in Agency change initiatives.

5.1 Recommendations

Recommendation 1:

The Agency should establish an internal and external communication process to share ongoing information about the FSMI projects and their benefits to those involved.

Recommendation 2:

The Agency should develop and implement a performance measurement strategy to track how FSMI projects are affecting the Food Safety Program. The strategy should include:

  • Indicators directly linked to overall Food Safety Program outcomes
  • Indicators to measure the effects of FSMI investments on program efficiency

Relevance: Need, Alignment with Government Priorities, and Alignment with Federal Roles and Responsibilities

Overall, the evaluation found the FSMI to be relevant and necessary for modernizing the FSP. Furthermore, FSMI projects are in line with the recommendations from the Weatherhill report, federal priorities, and the Government of Canada's Blueprint 2020 vision. The evaluation found all FSMI projects were well designed to meet established program needs and objectives.

These projects represent the beginning of long-term change activities at the Agency; therefore, ongoing efforts will be required to fully realize their intended benefits on CFIA programs. Without this, there is a risk the effectiveness of the initiative's investments will be undermined. Ensuring appropriate levels of effort rests in part on maintaining CFIA staff and external stakeholder buy-in for change.

The following recommendation is meant to establish a culture of change and support Agency program improvements.

Recommendation 1:

The Agency should establish and monitor an internal and external communication process to share ongoing information about the FSMI projects and their benefits to those involved.

Performance: Achievement of Outcomes, and Demonstration of Efficiency and Economy

Despite some implementation delays, FSMI performance is supported by the fact individual projects are on track to be completed with minor setbacks.

Of particular concern, the evaluation demonstrates there is a lack of an established and effective means of measuring the influence of FSMI projects and their impacts on the FSP, as well as broader Agency programming. Without an established and effective means for measurement, there is a challenge in establishing the effectiveness of FSMI investments and, therefore, justifying future investments in Agency change initiatives.

The following recommendation is meant to establish the basis for measuring the effectiveness and efficiency of change initiatives affecting the FSP.

Recommendation 2:

The Agency should develop and implement a performance measurement strategy to track how FSMI projects are affecting the Food Safety Program. The strategy should include:

Appendix A – References

References (cited in final report)

CFIA. (2013). Blueprint 2020: Building tomorrow's CFIA together: Getting started - getting your ideas.

CFIA. (2014a). Canadian Food Inspection Agency 2014-15 Report on Plans and Priorities.

CFIA. (2014b). Canadian Food Inspection Agency Food Program Framework.

CFIA, & SFCA. (2012). Safe Food for Canadians Act. Retrieved February 12, 2016, from http://www.inspection.gc.ca/about-the-cfia/acts-and-regulations/regulatory-initiatives/sfca/eng/1338796071420/1338796152395

Crawford, J. (2015). Strengthening Canada's Food Safety System - CASA 99th Annual Educational & Training Seminar April 20-13, 2015.

Government of Canada. (2011). The Next Phase of Canada's Economic Action Plan: A Low-Tax Plan for Jobs and Growth. Retrieved November 17, 2015, from http://www.budget.gc.ca/2011/plan/Budget2011-eng.pdf

Government of Canada. (2013a). 2012-2013 Departmental Performance Report. Retrieved August 10, 2015, from http://www.inspection.gc.ca/about-the-cfia/accountability/reports-to-parliament/2012-2013-dpr/eng/1377176926809/1377177134114?chap=3;%202013-14%20DPR%20:%20http://www.inspection.gc.ca/about-the-cfia/accountability/reports-to-parliament/2013-2014-dpr/eng/1409769354767/1409769355486?chap=4#s7c4

Government of Canada. (2013b). Section II - 2011-2012 Departmental Performance Report. Retrieved August 10, 2015, from http://www.inspection.gc.ca/about-the-cfia/accountability/reports-to-parliament/2011-2012-dpr/eng/1348777953917/1348778053447?chap=4

Government of Canada. (2014). 2013-2014 Departmental Performance Report. Retrieved August 10, 2015, from http://www.inspection.gc.ca/about-the-cfia/accountability/reports-to-parliament/2013-2014-dpr/eng/1409769354767/1409769355486?chap=3#s5c3

Government of Canada. (2016). 2014-2015 Departmental Performance Report. Retrieved February 3, 2016, from http://www.inspection.gc.ca/about-the-cfia/accountability/reports-to-parliament/2014-2015-dpr/eng/1442253072937/1442253073921?chap=2

References (to inform findings)

Available upon request

Appendix B – Evaluation Matrix

Evaluation of the CFIA's FSP-PART 1 Evaluation Matrix

Issue #1: Continued Need for the Foods Safety Program (FSP) and Food Safety Modernization Initiative (FSMI)

Evaluation Questions - What demonstrated need(s) do the FSP and FSMI address?
Indicators FSP FSMI Data collection

Identified needs for modernizing FSP

x x
  • Interview
  • Site Visits
  • Data review/ cost analysis
  • Literature review
  • Document review
Identified needs for food safety in Canada x
  • Data review/
    cost analysis
  • Literature review
  • Document review
Evaluation Questions - What evidence suggests that the FSP and FSMI will address demonstrated need(s)?
Indicators FSP FSMI Data collection
Examples of the effectiveness of programming similar to that offered under FSP x
  • Literature review
  • Document review
Examples of the effectiveness of initiatives similar to FSMI x
  • Literature review
  • Document review
Anticipated impact of FSMI on FSP in whole or in part x x
  • Survey
  • Interviews
  • Literature review
  • Document review
Evaluation Questions - Are there plausible alternatives, in whole or in part, to the sub-initiatives under the FSMI?
Indicators FSP FSMI Data collection
Alternatives to the FSMI sub-initiatives to address need for modernizing FSP, namely, alternatives to the:
  • Laboratory network strategy
  • Modernization of equipment and laboratories
  • Enhanced laboratory response capacity
  • Electronic service delivery platform
  • New food inspection model
  • Recruitment and training of inspectors
  • IM/IT development
x
  • Interviews
  • Literature review
  • Document review

Issue #2: Alignment with Government Priorities/data

Evaluation Question - Does the FSP, as currently delivered, support the CFIA's strategic outcome? Is this support for the strategic outcome expected to change as a result of the FSMI, and if so, how?
Indicators FSP FSMI Data collection
Demonstrated linkages between the Agency's strategic outcome and the FSP's design x
  • Interviews
  • Literature review
  • Document review
Whether or not FSP's support for the strategic outcome is expected to change as a result of the FSMI x x
  • Survey
  • Interviews
  • Literature review
  • Document review
Evaluation Question - Other than the CFIA's strategic outcome, to which federal priorities does the FSP contribute? Is this contribution expected to change as a result of the FSMI, and if so, how?
Indicators FSP FSMI Data collection
Identified other federal priorities considered in the FSP design x
  • Survey
  • Interview
  • Literature review
  • Document review
Whether or not contribution(s) of FSP to other federal priorities will change as a result of the FSMI x x
  • Survey
  • Interview
  • Document review

Issue #3: Alignment with Federal Roles and Responsibilities

Evaluation Question - Are the food safety roles and responsibilities of the CFIA clear and communicated?
Indicators FSP FSMI Data collection
Nature and means of communication regarding the current and future Agency role and responsibilities under the FSP x
  • Interviews
  • Data review/ cost analysis
  • Document review
Evaluation Question - Are FSP activities within the scope of CFIA's responsibilities for food safety? What changes, if any, are expected as a result of the FSMI?
Indicators FSP FSMI Data collection
Itemization of activities assessed against Agency responsibilities for food safety x
  • Data review/cost analysis
  • Document review
Perceptions of program design and delivery x x
  • Survey
  • Interviews
FSP activities expected to change as a result of the FSMI x x
  • Survey
  • Interviews
  • Document review

Issue #4: Delivery of FSMI Expected Outputs

Evaluation Question - Were the designs of the FSMI sub-initiatives based on evidence of success/good practices?
Indicators FSP FSMI Data collection

The FSMI sub-initiatives are grounded in theory and/or good practices, including the:

  • Laboratory network strategy
  • Modernization of equipment and laboratories
  • Enhanced laboratory response capacity
  • Electronic service delivery platform
  • New food inspection model
  • Recruitment and training of inspectors
  • IM/IT development
x
  • Interviews
  • Literature review
  • Document review
Evaluation Question - What challenges and emerging issues affected the implementation of the FSMI sub-initiatives, and how have these been addressed?
Indicators FSP FSMI Data collection

Challenges or issues that affected the implementation of the FSMI sub-initiatives, including the:

  • Laboratory network strategy
  • Modernization of equipment and laboratories
  • Enhanced laboratory response capacity
  • Electronic service delivery platform
  • New food inspection model
  • Recruitment and training of inspectors
  • IM/IT development
x
  • Interviews
  • Document review
Evaluation Question - Were the outputs of the FSMI sub-initiatives delivered as planned?
Indicators FSP FSMI Data collection

Outputs of the FSMI sub-initiatives delivered as planned, including those related to:

  • Laboratory network strategy
  • Modernization of equipment and laboratories
  • Enhanced laboratory response capacity
  • Electronic service delivery platform
  • New food inspection model
  • Recruitment and training of inspectors
  • IM/IT development
x
  • Interviews
  • Data review/ cost analysis
  • Document review

Issue #5: Achievement of FSMI Expected Outcomes

Evaluation Question - To what extent is there consistency in inspector recruitment and training?
Indicators FSP FSMI Data collection

Existence and use of a standardized approach to inspector recruitment and training for new and existing inspectors across regions

  • common tools and materials (e.g. manuals)
  • common/routine processes
  • common structure /uniform schedule/plan (frequency and regularity of training and recruitment, including continuous training)
  • coverage (e.g., proportion of inspectors who receive mandatory training)
  • Common approach to delivery
x
  • Survey
  • Interviews
  • Data review/ cost analysis
  • Document review
Evaluation Question - How has the approach to recruitment and training affected the composition of the CFIA workforce?
Indicators FSP FSMI Data collection

Proportion of inspectors who possess skills and knowledge aligned with the requirements of the new inspection model

  • Knowledge of new industry practices
  • Approach to maintain level of enhanced skills
x
  • Survey
  • Interviews
  • Document review

Proportion of science personnel possess skills/knowledge related to new science equipment and technology

  • new sampling methods/aligned with enhanced science capacity
x
  • Survey
  • Interviews
  • Document review

Proportion of inspectors who possess skills and knowledge to apply the new technological advancements:

  • Use of IM/IT tools (e.g. hardware and software; i.e., wireless technologies)
x x
  • Survey
  • Interviews
  • Document review
Planned vs. actual staff complement x
  • Data review/ cost analysis
  • Document review
Agency vs. inspectors' confidence levels in inspector capacity and targets over time x
  • Survey
  • Interviews
  • Document review
Evaluation Question - Has the FSMI led to, or will the FSMI lead to improved sharing of laboratory information; improved laboratory efficiencies; improved exchange of information between the CFIA, industry, and foreign authorities; inspectors being able to implement the new food safety inspection model; inspectors being able to access CFIA networks remotely; and effective Inspector use of new IT tools (new capital and equipment)?
Indicators FSP FSMI Data collection
Ongoing development of laboratory information sharing relative to pre-FSMI period x x
  • Survey
  • Document review
Progress to date in increasing efficiency of laboratory services and increasing laboratory capabilities x x
  • Survey
  • Document review
Progress in information exchange between CFIA, industry, and foreign authorities x x
  • Survey
  • Document review
Implementation of the new food safety inspection model x x
  • Survey
  • Document review
Progress toward gaining remote access to CFIA networks x x
  • Survey
  • Document review
IT tools implemented and used by Inspectors x x
  • Survey
  • Document review
Evaluation Question - Will the FSMI improve the FSP, and if so, how? In particular, will it: Improve the FSP's detection and response to outbreaks; Improve industry's access to CFIA services; Facilitate pre-clearance decisions for market access; and Allow for a consistent and efficient approach to the Agency's new inspection model?
Indicators FSP FSMI Data collection
Progress in improving laboratory services and capacity to respond to emergencies x x
  • Interviews
  • Document review
Expected industry use of CFIA services related to food safety x x
  • Interviews
  • Document review
Expected increases pre-clearance decisions x x
  • Interviews
  • Document review
Progress toward a consistent application of the new food safety inspection approach and improved efficiency of food safety inspection x x
  • Interviews
  • Document review
Evaluation Question - What factors have, or are expected to, influence FSMI success?
Indicators FSP FSMI Data collection
Perceptions of the impact of internal and external factors on sub-initiative success x
  • Survey
  • Interviews
  • Literature review
  • Document review
Evaluation Question - Is FSMI performance monitored on an ongoing basis? Has performance information been used to support decision-making regarding the FSMI sub-initiatives?
Indicators FSP FSMI Data collection
Availability, reliability, and usability of performance information x
  • Survey
  • Interviews
  • Data review/ cost analysis
  • Document review
Use of performance information to support decision-making x
  • Interviews
  • Document review

Issue #6: Demonstration of Efficiency and Economy

Evaluation Question - What are the costs of delivering FSMI? What are these costs relative to sub-initiative outputs and outcomes?
Indicators FSP FSMI Data collection
Costs of Initiative-related outputs x
  • Data review/ cost analysis
  • Literature review
  • Document review
Evaluation Question - Were FSMI resources expended as planned?
Indicators FSP FSMI Data collection
Planned-to-actual resource use / spending (budget vs. expenditures) for FSMI and explanation of variance x
  • Interviews
  • Data review/ cost analysis
  • Document review
Evaluation Question - How are the FSMI sub-initiatives expected to affect the efficiency of the FSP?
Indicators FSP FSMI Data collection
Anticipated sub-initiative impacts on the efficiency of the FSP x
  • Survey
  • Interviews
  • Data review/ cost analysis
  • Literature review
  • Document review
Planned-to-actual resource use / spending for FSP and explanation of variance x
  • Interviews
  • Data review/ cost analysis
  • Document review

Appendix C – CFIA Internal Reallocation

Table 9 presents the distribution of the internal reallocations within the FSMI in each of the fiscal years 2011-2012 through 2015-2016.

Table 9: CFIA Internal Funding Reallocations to FSMI (millions)
Activity 2011–2012 2012–2013 2013–2014 2014–2015 2015–2016 Total
Reclassification loan efficiencies - $5.00 $5.00 $5.00 $500 $20.00
Rent efficiencies - - $1.77 $1.77 $1.77 $5.33
Headquarters fit up efficiencies - - - - $1.60 $1.60
Internal administrative audit efficiencies - - $3.25 $3.25 $6.60 $13.10
Total - $5.00 $10.00 $10.00 $15.00 $40.00

Appendix D – Interview Guides

Food Safety Modernization Initiative - Overall

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

Since you have been involved directly or indirectly with one or more of these projects / sub-initiatives, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the projects/sub-initiatives will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

These issues relate to FSMI and FSP generally, and we would like you to reflect on them from a higher level or Agency strategic perspective when answering the questions on the subsequent pages of this interview guide.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA?

  1. Please briefly describe your involvement with FSMI and FSP.

Need for Programming

2. From your perspective, what needs does/will FSMI address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically, does FSMI address? Could you please explain how it is intended to do so?

Alignment with Government Priorities

4. From your perspective, will the implementation of FSMI affect the way that FSP supports the Agency's strategic outcome (A safe and accessible food supply)?

5. Did/will the implementation of FSMI affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

6. Has/will the implementation of FSMI affect the roles and responsibilities of the CFIA in any way? If so, what changes do you anticipate?

7. Has/will FSMI change(d) the scope of the CFIA's work or its involvement with other federal departments?

8. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

9. Thinking about FSMI, have there been challenges to implementation and if so, what were they? What has been the impact (positive or negative)?

10. Are there key dependencies that have/had to take place for FSMI to be a success? Please elaborate.

11. Were there any unexpected benefits from the implementation of FSMI?

12. Considering the expected outcomes of FSMI, have any been realized? Do you anticipate others to be realized in the future?

13. Please describe the performance monitoring and reporting approach for FSMI. How is it used to support the implementation of the initiative? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

14. What have been the implications of any deviation(s) from planned spending that took place during the implementation of FSMI?

15. Thinking about the FSP, in what way(s) has/will FSMI affect(ed) the efficiency of the program?

Conclusion

16. Are there any other points regarding FSMI that you would like to discuss and that would be relevant to the evaluation work currently underway?

17. In addition, are there documents that could help the evaluation team better understand the points we discussed today?

Thank you.

Improved Food Inspection Model

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

Since you have been involved with the Improved Food Inspection Model, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

Although these issues relate to FSMI and FSP generally, we would like you to reflect on the Improved Food Inspection Model specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean the Improved Food Inspection Model.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does/will your project address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome (A safe and accessible food supply)?

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way? If so, how?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that had/ will have to take place for your project to be a success? Please elaborate.

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of any the following?

16. Please describe the performance monitoring and reporting approach for your project. How is it used to support the continued development and implementation of your project over time? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviation(s) from planned spending that took place during the implementation of your project?

18. Thinking about the FSP, in what way has/will your project affect(ed) the efficiency of the program?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there documents that could help the evaluation team better understand the points we discussed today?

Thank you.

Electronic Service Delivery Platform

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

Since you have been involved with the ESDP, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

Although these issues relate to FSMI and FSP generally, we would like you to reflect on the ESDP specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean the ESDP.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does/will your project address.

3. What aspect of Food Safety Modernization or Agency Transformation specifically does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome (A safe and accessible food supply)?

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way? If so, what changes do you anticipate?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that had/will have to take place for your project to be a success? Please elaborate.

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of the following:

16. Please describe the performance monitoring approach to your project. How is it used to support the continued development and implementation of your project over time? Would you continue to use this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviations(s) from planned spending that took place during the implementation of your project?

18. Thinking about the FSP, how has or is your project likely to affect the efficiency of the program?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there are other documents about your project that could help the evaluation team understand the points we discussed today?

Thank you.

Recruitment and Training of Inspectors

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

Since you have been involved with the Recruitment and Training of Inspectors, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

Although these issues relate to FSMI and FSP generally, we would like you to reflect on the Recruitment and Training of Inspectors specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean the Recruitment and Training of Inspectors.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does/will your project address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically, does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome (A safe and accessible food supply)?

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way - particularly with regards to food safety? If so, what changes do you anticipate?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that had/will have to take place in order for your project to be a success? Please elaborate

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of any of the following?:

16. Please describe the performance monitoring and reporting approach for your project. Is this type of performance monitoring taking place on an ongoing basis? How is it used to support the continued development and implementation of your project over time? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviation(s) from planned spending that took place during the implementation of your project?

18. Thinking about the FSP, in what way(s) has/will your project affect(ed) the efficiency of the program?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there documents that could help the evaluation team understand the points we discussed today?

Thank you.

Laboratory Network Strategy

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

Since you have been involved with Developing a Laboratory Network Strategy, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

Although these issues relate to FSMI and FSP generally, we would like you to reflect on Developing a Laboratory Network Strategy specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean Developing a Laboratory Network Strategy.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does or will your project address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically, does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome (A safe and accessible food supply)?

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way - particularly with regards to food safety? If so, what changes do you anticipate?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that had/will have to take place for your project to be a success? Please elaborate.

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of any of the following?:

16. Please describe the performance monitoring and reporting approach for your project. Is this type of performance monitoring taking place on an ongoing basis? How is it used to support the continued development and implementation of your project over time? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviation(s) from planned spending that took place during the implementation of your project?

18. Thinking about the FSP, in what way(s) has/will your project affect(ed) the efficiency of the program?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there documents that could help the evaluation team understand the points we discussed today?

Thank you.

Modernization of Equipment and Laboratories

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

Since you have been involved with Modernizing Equipment and Laboratories, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

Although these issues relate to FSMI and FSP generally, we would like you to reflect on Modernizing Equipment and Laboratories specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean Modernizing Equipment and Laboratories.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does/will your project address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome (A safe and accessible food supply)?

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way - particularly with regards to food safety? If so, what changes do you anticipate?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that have/had to take place for your project to be a success? Please elaborate.

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of any of the following?:

16. Please describe the performance monitoring and reporting approach for your project. Is this type of performance monitoring taking place on an ongoing basis? How is it used to support the continued development and implementation of your project over time? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviation(s) from planned spending, if any, that took place during the implementation of your project?

18. Thinking about the FSP, in what way(s) has/will your project affect(ed) the efficiency of the program?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there documents that could help the evaluation team understand the points we discussed today?

Thank you.

Enhancing Laboratory Response Capacity

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

Since you have been involved with Enhancing Laboratory Response Capacity, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

Although these issues relate to FSMI and FSP generally, we would like you to reflect on Enhancing Laboratory Response Capacity specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean Enhancing Laboratory Response Capacity.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does/will your project address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome (A safe and accessible food supply)?

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way? If so, how?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that had/will have to take place for your project to be a success? Please elaborate.

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of the following:

16. Please describe the performance monitoring approach to your project. How is it used to support the continued development and implementation of your project over time? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviations(s) from planned spending that took place during the implementation of your project?

18. Thinking about the FSP, in what way(s) has/will your project affect(ed) the efficiency of the program ?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there documents about your project that could help the evaluation team understand the points we discussed today?

Thank you.

Improved IM/IT

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch (AEB), with the assistance of PRA Inc., is currently conducting a number of interviews in support of its Evaluation of the Food Safety Program (FSP) - Part 1. The evaluation focuses on seven of the eight projects, or sub-initiatives, developed as part of the Food Safety Modernization Initiative (FSMI). These include:

Since you have been involved with Increased Efficiency through IM/IT, we are asking you to participate in one of these one-hour interviews. We hope that your familiarity and close connection to the project/sub-initiative will allow you to provide us with valuable information for the evaluation.

With that in mind, the evaluation is examining six main issues related to FSP and FSMI. These include:

Although these issues relate to FSMI and FSP generally, we would like you to reflect on Increased Efficiency through IM/IT specifically when answering the questions on the subsequent pages of this interview guide. When the term "your project" is used, please take this to mean Increased Efficiency through IM/IT.

It is important to note that your participation in this interview is completely voluntary, and you can choose not participate at any time. In addition, if there are specific questions that you would prefer not to answer or that you feel you do not have the information to address, please let us know and we will continue on to the next question. Responses provided will be administered in accordance with the Privacy Act and other applicable privacy laws. In addition, all reporting will be written to provide aggregate results only, and no comments will be linked back to you, individually.

Introduction

1. Could you please introduce yourself and briefly describe the work you do at the CFIA? Although we may have reviewed this in an earlier discussion, for the purposes of this interview, could you briefly describe your involvement with your project?

Need for Programming

2. From your perspective, what needs does/will your project address?

3. What aspect of Food Safety Modernization or Agency Transformation specifically does/will your project address? Could you please explain how it is intended to do so?

4. Was your project based on another similar and/or successful initiative undertaken in another jurisdiction? If so, can you provide some detail about this initiative?

5. When your project was first being developed, were there alternative approaches to its implementation that were examined? If so, why were they rejected in favour of the current project approach?

Alignment with Government Priorities

6. From your perspective, did/will the implementation of your project affect the way that the Food Safety Program supports the Agency's strategic outcome

7. Did/will the implementation of your project affect the way in which the Agency is able to support other government priorities?

Alignment with Federal Roles and Responsibilities

8. Has/will the implementation of your project affect the roles and responsibilities of the CFIA in any way - particularly with regards to food safety? If so, what changes do you anticipate?

9. Has/will your project affect the scope of the CFIA's work or its involvement with other federal departments?

10. Have/will the changes discussed in the last two questions, if any, been communicated broadly throughout the Agency and to relevant external stakeholders?

Outputs and Outcomes

11. What theory or evidence informed the development of your project? For example, have there been studies that would suggest that implementation of your project as designed will produce anticipated outputs and outcomes?

12. Have there been challenges to the implementation of your project, and if so, what were they? What has been the impact (positive or negative)?

13. Are there key dependencies that had/will have to take place in order for your project to be a success? Please elaborate.

14. Were there any unexpected benefits from the implementation of your project?

15. To date, the data collected as part of the evaluation has pointed to your project's realization of many anticipated outputs and outcomes. Would you like to offer some additional comments on the achievement of any of the following?:

16. Please describe the performance monitoring and reporting approach for your project. Is this type of performance monitoring taking place on an ongoing basis? How is it used to support the continued development and implementation of your project over time? Would you continue with this approach in the future? Why or why not?

Efficiency and Economy

17. What have been the implications of any deviation(s) from planned spending that took place during the implementation of your project?

18. Thinking about the FSP, in what way(s) has/will your project affect(ed) the efficiency of the program?

Conclusion

19. Are there any other points regarding your project that you would like to discuss and that would be relevant to the evaluation work currently underway?

20. In addition, are there documents that could help the evaluation team understand the points we discussed today?

Thank you.

Date modified: