Evaluation of the CFIA's Food Safety Program Modernization - Part 1
3.0 Evaluation Objectives
In accordance with the Treasury Board Policy on Evaluation (2009) and its associated Directive and Standard, the evaluation assessed the following core issues:
- continued need for the program (relevance)
- alignment with government priorities (relevance)
- alignment with federal government roles and responsibilities (relevance)
- achievement of expected outcomes (performance)
- demonstration of efficiency and economy (performance)
This evaluation was atypical of Government of Canada evaluations, as it did not attempt to directly assess an established and ongoing program. Rather, it examined a suite of time-limited projects and investments meant to improve the way in which the FSP - and more broadly, the Agency - operates.
Given the nature of the FSMI and its relationship to both the FSP and broader change at the CFIA, the evaluation also needed to:
- assess whether the FSMI projects represent an appropriate direction for Agency change; and
- assess whether the FSMI projects' collective impacts will effectively contribute to the Agency's operations beyond the completion of FSMI funding.
Therefore, this evaluation was designed to meet both TB requirements and the information needs of the Agency.
3.1 Evaluation team and support
The CFIA's Evaluation Directorate managed the evaluation and conducted it with the assistance of PRA Inc. The evaluation was guided by an Advisory Committee and a Working Group, which reviewed and provided feedback on the evaluation plan, evaluation matrix, logic model, findings, report and Management Response and Action Plan.
The evaluation team developed and conducted the following:
- the evaluation Matrix
- a document and literature review
- a data review and cost analysis
- analysis and reporting
3.2 The evaluation matrix and FSMI logic model
To guide its work, the evaluation team developed an evaluation matrix, which included a list of the evaluation questions, organized according to the core TB issues (Appendix B). Each question was aligned with indicators and data collection methods. Aligning evaluation questions with the evaluation methods ensured the matrix maintain the evaluation scope. It also helped ensure that the evaluation methods were designed to address all of the identified evaluation questions.
Given the absence of an integrated performance measurement strategy and an approved logic model for FSMI, the evaluation team worked closely with members of the evaluation Working Group to finalize the logic model (Figure 3). The team modified a draft version of an FSMI logic model that was developed at the time of the initiative's inception with information gathered during preliminary discussions with members of the Working Group. This model outlined not only the immediate goals of the FSMI, but also highlighted their intended downstream impacts on Agency operations and ultimately strengthening the food safety system. This model was subsequently used to guide the evaluation, as it presented an overall view of the FSMI's expected impacts.
3.3 Data collection activities
The following data collection activities were conducted as part of the evaluation:
3.3.1 Document and literature review
The document and literature review leveraged existing documentation from within the Agency, along with academic and professional literature related to food safety.
A standardized template was used for the systematic review of this documentation and literature, and facilitated an overall understanding of these works. Information gathered through this review was valuable in addressing nearly all of the evaluation questions. It was particularly important in identifying FSMI project details and impacts, as well as changes to projects over time.
3.3.2 Data review and cost analysis
The data review and cost analysis generated quantitative financial information and, where possible, output and outcome information on FSMI projects. This information provided insight into the efficiency and economy of FSMI expenditures, specifically:
- whether FSMI funds were used as planned
- whether expenditures were reasonable
- whether projects were achieving their intended outcomes
Thirty-seven interviews were conducted with representatives from Policy and Programs Branch, Operations Branch, Science Branch, Human Resources Branch, and IM/IT Branch. Interviewees were chosen based on their knowledge about the FSMI and related projects. The interviews were conducted after the document and literature review, in order to supplement existing information with contextual insight and interpretation on the part of program delivery staff and stakeholders. Further, it allowed the evaluation team to identify gaps and inconsistencies in information.
Eight interview guides were developed to facilitate semi-structured interviews - one for individuals associated with each of the seven FSMI projects, and a final guide for individuals involved with the initiative as a whole. These are provided in Appendix D.
3.4 Analysis and reporting
Information from the various data collection activities was analyzed in order to make conclusions regarding each of the questions included in the evaluation matrix. This triangulation process allowed the evaluation to use the strengths of each of the data collection activities to their best possible advantage. It also allowed the team to compare or confirm findings across data collection activities, as well as to supplement and contextualize the findings from one data collection activity to another.
The triangulation completed during the evaluation analysis also provides an opportunity to identify key information gaps. Not all questions identified in the evaluation matrix had an equivalent amount of data or information to support associated conclusions.
3.5 Limitations and challenges
The evaluation limitations and challenges, and the corresponding mitigation strategies, are described in Table 5 below.
|Partially developed logic model - The FSMI had only a partially developed logic model at the start of the evaluation. This made it difficult to develop an initial understanding of the initiative's collective goals.||The evaluation team developed an FSMI logic model specifically for the evaluation.||The newly developed logic model used for the evaluation outlined the expected outputs and outcomes for the FSMI at its inception.|
|Ongoing change at the CFIA - Throughout the implementation of the FSMI, the CFIA has been undergoing significant change. This has influenced the FSP. At the same time, the FSMI forms part of these change activities and is meant to affect the way the Agency works overall.||During the evaluation, the team took into consideration the FSMI's place within broader change activities at the Agency.||The evaluation focused on the FSMI's influence on the Agency's work as a whole and not only on its benefits to the FSP.|
|The need to examine multiple projects - This evaluation needed to examine multiple projects with differing approaches to management, varying data collection practices, and different amounts of available data. This made it challenging to maintain consistency and manage the scope of the evaluation.||The evaluation team worked closely with the Working Group to focus the evaluation on the most relevant aspects of each project. The team then used a findings table in the final report to support aggregate reporting across all projects.||A findings table in the FSP Modernization Evaluation - Part 1 provides for the simplification of many of the observations regarding individual FSMI projects and the alignment across FSMI project observations.|
|Limited information on food safety outcomes - Projects generally tracked their own activities and outputs well, but the evaluation team encountered very limited tracking of project effects on food safety.||The evaluation team made recommendations for measuring the influence of FSMI projects and more general Agency change activities on CFIA programming and outcomes.||Recommendations from the evaluation extend beyond the FSMI to track the link between projects and the FSP or other Agency programming.|
|Projects in early stages of development and implementation - Many of the FSMI projects remain at an early stage of implementation, with limited effects on the FSP to date.||Rather than leveraging existing outcomes data related to FSP changes, the evaluation team focussed on establishing whether the expected outcomes of the projects appeared likely given the available theory, business cases, best practices, and other evidence.||Recommendations from the evaluation focus on ensuring that the Agency has an efficient way of establishing the effects of the FSMI on FSP in the future.|
- Date modified: