Evaluation of the CFIA's Food Safety Program Modernization - Part 1
Final Report

Appendix B – Evaluation Matrix

Evaluation of the CFIA's FSP-PART 1 Evaluation Matrix

Issue #1: Continued Need for the Foods Safety Program (FSP) and Food Safety Modernization Initiative (FSMI)

Evaluation Questions - What demonstrated need(s) do the FSP and FSMI address?
Indicators FSP FSMI Data collection

Identified needs for modernizing FSP

x x
  • Interview
  • Site Visits
  • Data review/ cost analysis
  • Literature review
  • Document review
Identified needs for food safety in Canada x
  • Data review/
    cost analysis
  • Literature review
  • Document review
Evaluation Questions - What evidence suggests that the FSP and FSMI will address demonstrated need(s)?
Indicators FSP FSMI Data collection
Examples of the effectiveness of programming similar to that offered under FSP x
  • Literature review
  • Document review
Examples of the effectiveness of initiatives similar to FSMI x
  • Literature review
  • Document review
Anticipated impact of FSMI on FSP in whole or in part x x
  • Survey
  • Interviews
  • Literature review
  • Document review
Evaluation Questions - Are there plausible alternatives, in whole or in part, to the sub-initiatives under the FSMI?
Indicators FSP FSMI Data collection
Alternatives to the FSMI sub-initiatives to address need for modernizing FSP, namely, alternatives to the:
  • Laboratory network strategy
  • Modernization of equipment and laboratories
  • Enhanced laboratory response capacity
  • Electronic service delivery platform
  • New food inspection model
  • Recruitment and training of inspectors
  • IM/IT development
x
  • Interviews
  • Literature review
  • Document review

Issue #2: Alignment with Government Priorities/data

Evaluation Question - Does the FSP, as currently delivered, support the CFIA's strategic outcome? Is this support for the strategic outcome expected to change as a result of the FSMI, and if so, how?
Indicators FSP FSMI Data collection
Demonstrated linkages between the Agency's strategic outcome and the FSP's design x
  • Interviews
  • Literature review
  • Document review
Whether or not FSP's support for the strategic outcome is expected to change as a result of the FSMI x x
  • Survey
  • Interviews
  • Literature review
  • Document review
Evaluation Question - Other than the CFIA's strategic outcome, to which federal priorities does the FSP contribute? Is this contribution expected to change as a result of the FSMI, and if so, how?
Indicators FSP FSMI Data collection
Identified other federal priorities considered in the FSP design x
  • Survey
  • Interview
  • Literature review
  • Document review
Whether or not contribution(s) of FSP to other federal priorities will change as a result of the FSMI x x
  • Survey
  • Interview
  • Document review

Issue #3: Alignment with Federal Roles and Responsibilities

Evaluation Question - Are the food safety roles and responsibilities of the CFIA clear and communicated?
Indicators FSP FSMI Data collection
Nature and means of communication regarding the current and future Agency role and responsibilities under the FSP x
  • Interviews
  • Data review/ cost analysis
  • Document review
Evaluation Question - Are FSP activities within the scope of CFIA's responsibilities for food safety? What changes, if any, are expected as a result of the FSMI?
Indicators FSP FSMI Data collection
Itemization of activities assessed against Agency responsibilities for food safety x
  • Data review/cost analysis
  • Document review
Perceptions of program design and delivery x x
  • Survey
  • Interviews
FSP activities expected to change as a result of the FSMI x x
  • Survey
  • Interviews
  • Document review

Issue #4: Delivery of FSMI Expected Outputs

Evaluation Question - Were the designs of the FSMI sub-initiatives based on evidence of success/good practices?
Indicators FSP FSMI Data collection

The FSMI sub-initiatives are grounded in theory and/or good practices, including the:

  • Laboratory network strategy
  • Modernization of equipment and laboratories
  • Enhanced laboratory response capacity
  • Electronic service delivery platform
  • New food inspection model
  • Recruitment and training of inspectors
  • IM/IT development
x
  • Interviews
  • Literature review
  • Document review
Evaluation Question - What challenges and emerging issues affected the implementation of the FSMI sub-initiatives, and how have these been addressed?
Indicators FSP FSMI Data collection

Challenges or issues that affected the implementation of the FSMI sub-initiatives, including the:

  • Laboratory network strategy
  • Modernization of equipment and laboratories
  • Enhanced laboratory response capacity
  • Electronic service delivery platform
  • New food inspection model
  • Recruitment and training of inspectors
  • IM/IT development
x
  • Interviews
  • Document review
Evaluation Question - Were the outputs of the FSMI sub-initiatives delivered as planned?
Indicators FSP FSMI Data collection

Outputs of the FSMI sub-initiatives delivered as planned, including those related to:

  • Laboratory network strategy
  • Modernization of equipment and laboratories
  • Enhanced laboratory response capacity
  • Electronic service delivery platform
  • New food inspection model
  • Recruitment and training of inspectors
  • IM/IT development
x
  • Interviews
  • Data review/ cost analysis
  • Document review

Issue #5: Achievement of FSMI Expected Outcomes

Evaluation Question - To what extent is there consistency in inspector recruitment and training?
Indicators FSP FSMI Data collection

Existence and use of a standardized approach to inspector recruitment and training for new and existing inspectors across regions

  • common tools and materials (e.g. manuals)
  • common/routine processes
  • common structure /uniform schedule/plan (frequency and regularity of training and recruitment, including continuous training)
  • coverage (e.g., proportion of inspectors who receive mandatory training)
  • Common approach to delivery
x
  • Survey
  • Interviews
  • Data review/ cost analysis
  • Document review
Evaluation Question - How has the approach to recruitment and training affected the composition of the CFIA workforce?
Indicators FSP FSMI Data collection

Proportion of inspectors who possess skills and knowledge aligned with the requirements of the new inspection model

  • Knowledge of new industry practices
  • Approach to maintain level of enhanced skills
x
  • Survey
  • Interviews
  • Document review

Proportion of science personnel possess skills/knowledge related to new science equipment and technology

  • new sampling methods/aligned with enhanced science capacity
x
  • Survey
  • Interviews
  • Document review

Proportion of inspectors who possess skills and knowledge to apply the new technological advancements:

  • Use of IM/IT tools (e.g. hardware and software; i.e., wireless technologies)
x x
  • Survey
  • Interviews
  • Document review
Planned vs. actual staff complement x
  • Data review/ cost analysis
  • Document review
Agency vs. inspectors' confidence levels in inspector capacity and targets over time x
  • Survey
  • Interviews
  • Document review
Evaluation Question - Has the FSMI led to, or will the FSMI lead to improved sharing of laboratory information; improved laboratory efficiencies; improved exchange of information between the CFIA, industry, and foreign authorities; inspectors being able to implement the new food safety inspection model; inspectors being able to access CFIA networks remotely; and effective Inspector use of new IT tools (new capital and equipment)?
Indicators FSP FSMI Data collection
Ongoing development of laboratory information sharing relative to pre-FSMI period x x
  • Survey
  • Document review
Progress to date in increasing efficiency of laboratory services and increasing laboratory capabilities x x
  • Survey
  • Document review
Progress in information exchange between CFIA, industry, and foreign authorities x x
  • Survey
  • Document review
Implementation of the new food safety inspection model x x
  • Survey
  • Document review
Progress toward gaining remote access to CFIA networks x x
  • Survey
  • Document review
IT tools implemented and used by Inspectors x x
  • Survey
  • Document review
Evaluation Question - Will the FSMI improve the FSP, and if so, how? In particular, will it: Improve the FSP's detection and response to outbreaks; Improve industry's access to CFIA services; Facilitate pre-clearance decisions for market access; and Allow for a consistent and efficient approach to the Agency's new inspection model?
Indicators FSP FSMI Data collection
Progress in improving laboratory services and capacity to respond to emergencies x x
  • Interviews
  • Document review
Expected industry use of CFIA services related to food safety x x
  • Interviews
  • Document review
Expected increases pre-clearance decisions x x
  • Interviews
  • Document review
Progress toward a consistent application of the new food safety inspection approach and improved efficiency of food safety inspection x x
  • Interviews
  • Document review
Evaluation Question - What factors have, or are expected to, influence FSMI success?
Indicators FSP FSMI Data collection
Perceptions of the impact of internal and external factors on sub-initiative success x
  • Survey
  • Interviews
  • Literature review
  • Document review
Evaluation Question - Is FSMI performance monitored on an ongoing basis? Has performance information been used to support decision-making regarding the FSMI sub-initiatives?
Indicators FSP FSMI Data collection
Availability, reliability, and usability of performance information x
  • Survey
  • Interviews
  • Data review/ cost analysis
  • Document review
Use of performance information to support decision-making x
  • Interviews
  • Document review

Issue #6: Demonstration of Efficiency and Economy

Evaluation Question - What are the costs of delivering FSMI? What are these costs relative to sub-initiative outputs and outcomes?
Indicators FSP FSMI Data collection
Costs of Initiative-related outputs x
  • Data review/ cost analysis
  • Literature review
  • Document review
Evaluation Question - Were FSMI resources expended as planned?
Indicators FSP FSMI Data collection
Planned-to-actual resource use / spending (budget vs. expenditures) for FSMI and explanation of variance x
  • Interviews
  • Data review/ cost analysis
  • Document review
Evaluation Question - How are the FSMI sub-initiatives expected to affect the efficiency of the FSP?
Indicators FSP FSMI Data collection
Anticipated sub-initiative impacts on the efficiency of the FSP x
  • Survey
  • Interviews
  • Data review/ cost analysis
  • Literature review
  • Document review
Planned-to-actual resource use / spending for FSP and explanation of variance x
  • Interviews
  • Data review/ cost analysis
  • Document review
Date modified: