Archived - Evaluation of Daily Shift Inspection Presence

June 2014

Table of Contents

• Executive Summary
• 1. Introduction
  • 1.1 Description
  • 1.2 Context
  • 1.3 Evaluation Objective and Scope
• 2. Methodology
  • 2.1 Description
  • 2.2 Analysis and Integration of Data
  • 2.3 Limitations
• 3. Findings
  • 3.1 Relevance: Continued need for the program
  • 3.2 Relevance: Alignment with federal government priorities
  • 3.3 Relevance: Alignment with federal government roles and responsibilities
  • 3.4 Performance: Achievement of expected outcomes
• 4. Conclusions
• 5. Recommendation
• A. Logic Model
• B. Bibliography
• C. Interview Guides
• D. Evaluation Matrix

Executive Summary

Background

The 2009 United States Department of Agriculture, Food Safety and Inspection Service audit found that the frequency of inspections in Canada's federally registered meat processing plants was not equivalent to the frequency of inspections in American plants. Canada subsequently adopted daily shift inspection presence in order to meet U.S. standards for import of processed meat products and thus to maintain access to the U.S. market.

Evaluation

The Treasury Board Secretariat requested an evaluation of the Daily Shift Inspection Presence (DSIP) initiative in order to inform its potential renewal. This evaluation was added to the Canadian Food Inspection Agency's 2013-2014 Evaluation Plan, approved by the Evaluation Committee in June 2013.

The evaluation focused on the issues of relevance and performance of the DSIP. DSIP is not a typical evaluation subject as it is an augmentation to an existing program, of the same activity, which forms a small part of a much broader food safety program. The evaluation's scope was limited to the additional effort to accomplish the increased frequency of inspection, from a risk based frequency to a daily shift frequency. Due to this, there is no performance measurement strategy for DSIP or indicators tailored for DSIP, separate from the existing food safety inspection program.

Conclusions

Daily shift inspection presence is an important component of CFIA's inspection program as it meets U.S. standards for the import of processed meat products and ensures continued exports of Canadian meat to the U.S. Given its alignment with federal trade priorities and with CFIA's regulatory responsibilities, it continues to remain relevant to the mandate of the CFIA.

Although DSIP is achieving its primary outcome, to ensure continued exports to the U.S., and while the program is part of the CFIA's overall food safety regime, the evaluation was unable to conclude on its impact on food safety.

While the DSIP initiative appears to be cost effective in achieving the trade objective of keeping the border to the U.S. open to meat exports, it is not a cost-effective means of addressing food safety because it is not risk-based. However, U.S. trade requirements prevent implementation of a purely risk-based approach. A risk-based approach may still lead to inspection presence at each shift, but this would only be for those establishments with the highest risk, whereas the U.S. requirement is for inspection presence at each shift of all establishments, even for those with very low food safety risk.

Current requirements, for the number of inspectors to cover all plant shifts, appear to remain in line with past levels, suggesting that the funding needs for DSIP remain stable. However, a change in one or more environmental factors (e.g., U.S. readiness to examine DSIP, consumer understanding and acceptance of a risk-based approach and a dual system for domestic and international markets) could create opportunities to discontinue or modify DSIP to achieve greater efficiencies. For example, ceasing the requirement for daily shift inspections at domestic plants would provide potential cost savings, though also potentially challenge the public's perception of food safety. It is important to maintain an awareness of these environmental factors and opportunities to be able to leverage them and maximize cost-efficiency.

The Agency does not have effective reporting capability to demonstrate that the DSIP requirement was met. There are records in regional offices and at meat establishments, that, if collected and analysed, would provide point in time data required to create such a report. Generating this report would be a significant undertaking. Interview evidence has shown that some resource reallocation is occurring, particularly on the domestic side (e.g., to respond to food safety emergencies). Information was not available (without investment of considerable resources) to demonstrate the level of coverage and reallocation.

Recommendation: The Operations Branch should identify and track performance measurement indicators that would illustrate that the necessary number of inspections is undertaken each year.

1. Introduction

1.1 Description

The United States Department of Agriculture, Food Safety and Inspection Service (USDA-FSIS) conducts regular audits of the Canadian meat inspection system to ensure that Canada continues to maintain a food safety system for meat and poultry that is equivalent to that of the United States. The 2009 USDA-FSIS audit ^Footnote 1 found the frequency of inspections in Canadian meat processing plants was not equivalent to the frequency of inspections in American plants. U.S. meat and poultry legislation requires inspection of a meat establishment on a continuous basis, which is interpreted by the USDA as once per shift, regardless of the level of food safety risk. This is referred to as "daily shift inspection presence" and means establishments operating two shifts every 24 hours would be inspected twice daily. The FSIS audit identified Canada's lack of a daily shift inspection presence regime as a deficiency to the eligibility requirement to ship processed meat to the U.S. ^Footnote 2

In response to the 2009 USDA-FSIS audit, Canada adopted daily shift inspection presence (DSIP) at all federally registered meat processing establishments to meet U.S. standards for the import of processed meat products, thus ensuring continued exports of Canadian meat to the U.S. In order to maintain consistent standards, Canada adopted the same system for all federally registered meat processing plants, including those that do not export to the U.S.

CFIA received the following funding to increase the frequency of inspections at federally registered meat processing plants.

The response to the 2009 USDA-FSIS audit involved adding an additional 108 full-time equivalent (FTE) front-line inspectors to carry out DSIP, approximately eleven additional FTE supervisors and program specialists, five additional FTE program management staff to support the increased front-line inspection, and one additional FTE for training inspectors. At the time of the transition to DSIP there were 613 registered establishments ^Footnote 3 of which 388 (63%) were eligible to export to the U.S. Of the 388 U.S. eligible registered establishments, 165 (42.5%) operated with multiple shifts, thereby requiring additional inspections to satisfy DSIP. By comparison, of the 225 domestic registered establishments, only 31 (13.8%) had multiple shifts. ^Footnote 4 Thus, the impact of DSIP on the number of required inspectors has been more pronounced for export establishments which have a greater percentage of multiple shifts.

A logic model showing the relationships between the activities, outputs and outcomes of increased inspection frequency is provided in Appendix A.

1.2 Context

Under the previous risk-based system, the frequency of inspection visits was based on a plant's risk profile, including its history of compliance. Plants with a high risk profile would be inspected daily, while plants with a low risk profile would be inspected as little as once per week. With the introduction of DSIP, the overall frequency of inspections increased, regardless of a processing plant's risk.

CFIA's Compliance Verification System (CVS), a tool that inspectors use to verify industry compliance, was implemented in all of Canada's federally-registered meat establishments on April 1, 2008. The CVS includes verification tasks that are used by the CFIA inspection staff to assess compliance with regulatory requirements. Each verification task includes detailed procedures for the inspection staff to follow when conducting verifications, as part of the food safety inspection system. The CVS tool allows data to be captured and analysed for: patterns of compliance; indications of systemic problems; compliance with Canadian trading partner's regulations; and uniformity of program delivery. ^Footnote 5

Under DSIP, a typical visit takes approximately 90 minutes (1.5 hours) ^Footnote 6 comprised of about 45 minutes for the CVS task(s), 15 minutes to clean up and 30 minutes to travel to the next plant. ^Footnote 7 By comparison, under the previous risk-based system, a typical visit would take about three to four hours, permitting the CFIA inspection staff to cover a larger portion of tasks. Over a one week period, the amount of time CFIA inspection staff spent at a high risk plant was about the same under DSIP as in the previous system, because inspectors are essentially splitting up the time previously expended in one visit. In contrast, the amount of time spent in a low risk plant has increased under DSIP as inspectors now must attend every shift. In some circumstances, during the previous system, an inspector visited a plant as little as once per week. Attending every shift, even at the minimum of 1.5 hours, is cumulatively more hours than once per week could ever have been.

1.3 Evaluation Objective and Scope

The Treasury Board Secretariat (TBS) requested an evaluation of the DSIP initiative in order to inform its potential renewal. This evaluation was added to the CFIA's 2013-14 Evaluation Plan, approved by the Evaluation Committee in June 2013. The evaluation's focus was limited to the relevance and impact of the additional effort required to accomplish the increased frequency of inspection, from a risk based frequency to a daily shift frequency. In accordance with the Treasury Board Secretariat Policy on Evaluation (2009), and its supporting Directive and Standard, the evaluation assessed DSIP as per the following evaluation issues:

1. Continued need for the program (Relevance);
2. Alignment with government priorities (Relevance);
3. Alignment with federal government roles and responsibilities (Relevance);
4. Achievement of expected outcomes (Performance); and
5. Demonstration of efficiency and economy (Performance).

The evaluation was conducted over a four month period from mid-June to mid-October 2013 and covers the period from April 2010 to September 2013.The evaluation team was composed of evaluators from a consulting firm and CFIA's Audit and Evaluation Branch.

2. Methodology

2.1 Description

A number of methodologies were applied to collect data from multiple sources, as outlined below. Each of these methodologies is presented in the evaluation matrix, included as Appendix D, and is linked to the evaluation questions and indicators.

Document and Data Review

A review was conducted of relevant databases and documents including the Compliance Verification System and trade data on exports and imports of meat products between Canada and the U.S. and other countries.

Literature Review

A literature review was conducted to develop an overview of Canadian meat exportation, including related food safety issues, with a focus on events that have both inhibited and promoted this exportation and to compare the meat inspection processes of a number of other meat exporting countries including Australia, New Zealand, and Mexico.

Interviews

Interviews were conducted with representatives from the CFIA, Canadian Secretariat for the Canada-U.S. Regulatory Cooperation Council (RCC), and plant managers at meat establishments, all of whom were identified as key stakeholders through the research conducted in the planning for this evaluation. Interviewees were asked to provide their perspective on the outcomes of the DSIP, both intended and unintended. The number of interviews by interview group is shown in Figure 2-1 below. The interview guides used in the evaluation are provided in Appendix C.

The planned number of interviews with representatives from the Regulatory Cooperation Council (RCC)/initiative working group members was not completed as one had recently retired and could not be reached. The planned interviews with market access stakeholders were not completed because it was determined that they did not have sufficient familiarity with DSIP to comment. The planned number of interviews with representatives from meat processing establishments was not completed because of the limited time available (further details provided below in Section 2.3). Time limitations required the evaluation to focus only on those interviewees with the most experience and knowledge about DSIP.

2.2 Analysis and Integration of Data

Although the interviews were not designed or used to collect quantitative data, the interpretation of findings takes into account the extent to which certain perceptions or views were expressed or shared by the interviewees. Figure 2-2 defines terms used in this report to quantify the proportion of interviews who expressed similar experiences, views and opinions.

Multiple methodologies were applied to collect data from multiple sources. Data was validated within and across methods, an approach that helps to increase the validity of the evaluation findings.

2.3 Limitations

The evaluation challenges and limitations and the corresponding mitigation strategies are described in Figure 2-3 below.

3. Findings

Findings for each of the five core evaluation issues are provided below.

3.1 Relevance: Continued need for the program

The majority of interviewees, including interviewees from CFIA and meat processing plants, believe that trade is the main issue addressed by DSIP, and that there is continued need for this presence in order to comply with requirements imposed by the U.S. At the same time, the majority of interviewees were of the view that relying on a risk-based approach to inspection would provide sufficient oversight for ensuring food safety. A risk-based approach to food safety is also aligned with CFIA's approach to inspection outlined in its inspection modernization initiative. ^Footnote 8 Alternatives to DSIP are not feasible at this time as it is required to maintain access to the U.S. market.

Both CFIA and RCC interviewees noted that there was no appetite to explore options relative to DSIP at this time and in the context of the current regulatory cooperation initiative. In the meantime, in the absence of changes to the U.S. DSIP requirements, or a bilateral agreement with Canada, DSIP continues to be relevant and fulfill a recognised need for Canada. The effect of DSIP on food safety is discussed in section 3.4 below.

3.2 Relevance: Alignment with federal government priorities

Facilitating the lawful movement of people, goods and services between Canada and the U.S. is a federal government priority. The June 3, 2011 Speech from the Throne recognizes the U.S. as Canada's most important trading partner and commits to strengthening collaboration to streamline and secure the border and enhance regulatory cooperation to ensure that goods can flow freely between the two countries. ^Footnote 9 The 2013 Speech from the Throne committed the Government to continue to implement the work of the Regulatory Cooperation Council. ^Footnote 10 The Canada-U.S. Regulatory Cooperation Council was created in 2011 to better align Canada and the U.S. regulatory approaches, where possible. ^Footnote 11 Canada adopted DSIP to align with U.S. import requirements for processed meat products. As indicated in section seven of the Meat Inspection Act, a meat product will not be exported outside of Canada unless it meets the requirements of the country to which it is being exported. ^Footnote 12

The majority of interviewees indicated that DSIP is aligned with the government's trade priority, although they did not consider it to be specifically addressing a food safety priority. It was a commonly held view that Canada's adoption of the initiative supports market access for products from Canadian meat processing establishments. The majority of interviewees noted, however, that the Agency's approach to food safety and the inspection modernization initiative focus on "food inspection activities that target the highest risk areas (domestic or imported)" ^Footnote 13, a focus that is not consistent with DSIP, which makes no differentiation based on level of risk.

However, some interviewees believe that DSIP is aligned with government food safety priorities, by providing increased confidence by American and Canadian consumers in Canada's food safety system. There are no polls or surveys to support this conclusion, although one respondent noted that "there is a natural appeal in a simplistic analysis, saying more inspectors, more safety; it resonates with consumers."

While there is insufficient evidence to conclude on a specific link between DSIP and food safety priorities, the evidence supports a strong link between DSIP and the government's trade objectives.

3.3 Relevance: Alignment with federal government roles and responsibilities

The Meat Inspection Act states that meat that is transported between provinces or exported out of the country must be inspected in a federally registered establishment; it is therefore clear that implementation of the DSIP initiative is directly aligned with federal government's stated roles and responsibilities. More broadly, the U.S. objective in requiring DSIP is based on ensuring food safety, the primary responsibility of the CFIA. As the United States requires meat products imported into the country be produced under the daily shift inspection of government inspectors, DSIP is, therefore, required to be implemented by all countries wishing to export their product to the U.S.

3.4 Performance: Achievement of expected outcomes

See Appendix A for the logic model for DSIP which lists all expected outcomes, including activities and outputs that support them.

3.4.1 Expected Outcomes

Canadian Meat Products Exported to the U.S. Unhindered

The border has remained open, unhindered to meat product shipments, which is the primary goal of DSIP. Since the implementation of DSIP, the U.S. has not hindered meat imports from Canada because of any deficiency in meat plant shift inspection presence. The value of the meat trade that this open border facilitates is substantial, as shown by the high value of meat exports. A review of export data for all meat, including meat from processing plants (as defined by Harmonized System ^Footnote 14 – HS-02) over the 2003 to 2012 period indicates that the value of exports to the U.S. has remained more or less constant at around $2 billion since 2009 (see Figure 3-1).

Figure 3-1: Value of Exports: HS 02 - Meat and Edible Meat Offal, 2003-2012 ^Footnote 15

Description for Figure 3-1

Figure 3.1 outlines the value of exports for HS 02 – Meat and Edible Meat Offal from 2003 to 2012

The value of total meat exports to the USA (in billions) was:

• $2.42 billion in 2003
• $2.88 billion in 2004
• $2.72 billion in 2005
• $2.10 billion in 2006
• $1.92 billion in 2007
• $1.90 billion in 2008
• $1.83 billion in 2009
• $2.03 billion in 2010
• $1.99 billion in 2011
• $1.89 billion in 2012

The value of total meat exports to other countries (in billions) was:

• $1.36 billion in 2003
• $1.72 billion in 2004
• $2.04 billion in 2005
• $1.84 billion in 2006
• $1.90 billion in 2007
• $2.44 billion in 2008
• $2.26 billion in 2009
• $2.39 billion in 2010
• $2.75 billion in 2011
• $2.70 billion in 2012

The value of total meat exports to all countries (in billions) was:

• $3.78 billion in 2003
• $4.60 billion in 2004
• $4.76 billion in 2005
• $3.94 billion in 2006
• $3.82 billion in 2007
• $4.33 billion in 2008
• $4.09 billion in 2009
• $4.42 billion in 2010
• $4.74 billion in 2011
• $4.59 billion in 2012

Maintained International and Domestic Confidence

Most interviewees felt that DSIP has helped maintain international and domestic confidence in Canada's food safety and food safety systems. These interviewees noted that while DSIP does not necessarily produce safer food, being at the plant every day and every shift makes a positive impression on consumers. The results from Canadian federal government consultations with Canadians on regulatory cooperation would tend to support efforts to align standards and approaches between Canada and the United States. The consultations found that Canadians want "common approaches to food safety requirements, biotechnology product approvals, crop protection products, labelling, packaging and product content information..." ^Footnote 16 On the other hand, many interviewees felt it was difficult to determine whether DSIP has affected consumer confidence, particularly in other countries to which Canada exports meat products. No known survey information exists to establish any linkage between DSIP and consumer confidence.

Enhanced Food Safety Culture

Federally registered meat processing establishments are responsible for producing a safe product. ^Footnote 17 For meat products exported to the U.S., they are obliged to be compliant with both Canadian and U.S. laws and regulations irrespective of whether a risk-based or DSIP approach is used. ^Footnote 18

As noted above, the primary objective of the DSIP is to align domestic requirements with those in the United States to support Canadian trade objectives.

However, all plant manager interviewees, albeit limited in number (n=6), indicated their view that DSIP also reinforces the importance of food safety on plant employees and has had a positive impact on food safety culture.

Food Safety

Most interviewees, including the majority of inspection managers, noted that there are apparent advantages and disadvantages from a food safety perspective with the DSIP. These interviewees noted that there is insufficient scientific evidence to indicate whether the previous risk-based system was better or worse from a food safety point of view than the current DSIP system. Based on their personal experience, most interviewees were of the view that increasing inspection frequency, without basing this on food safety risk, did not directly enhance food safety. To explore differences between the systems and the degree to which they generated safety events, Corrective Action Requests (CARs) from 2009-10 to 2012-13 were reviewed. There was little change in the number of CARs issued over this period, with 11 CARs issued in 2009-10, 10 in 2010-11, 9 in 2011-12 and 11 in 2012-13. Due to the lack of any significant changes, no conclusions can be made. One interviewee suggested that the reason for the consistency in the number of events, despite changes in the frequency of inspection, is that only a few plants are the cause of CARs. In other words, if a few plants are consistently the source of problems, increasing inspections at all plants is not going to change the compliance rates when most were complying in the first place.

It was noted by some interviewees that there is never a guarantee that every possible safety issue will always be identified, even with increased inspector presence. Under the DSIP model, inspectors make shorter, more frequent visits to plants while, under the previous risk-based system, each inspection took longer, allowing the inspector to cover more tasks during the visit.

Some inspection managers noted that the rigidity of the DSIP model can be a disadvantage as it can hinder the re-assignment of staff in the event of a food safety emergency. In a 2012 audit report by the United States Department of Agriculture (USDA), the U.S. identified this lack of flexibility as an issue in the U.S. as well. ^Footnote 19 Another disadvantage cited by a majority of inspection managers of the DSIP initiative is that the increased repetitiveness of plant work flows and practices that result from the short task visits could lead to complacency for some inspectors. To mitigate against this risk, inspectors are rotated to provide a set of "fresh eyes"; a practice that is easier to introduce in urban regions with many plants within close proximity. The Agency also employs a comprehensive Quality Management System "to continuously improve the consistency and overall quality of activities delivered by CFIA Operations staff, by incorporating modern quality management principles and practices." ^Footnote 20

Most recently, Minister Ritz announced the government's intent to establish Inspection Verification Teams (IVTs) to oversee the performance of Canada's entire food inspection system. IVTs are expected to undertake spot checks of plants across the country to help ensure that the overall food safety system is effective and that food safety rules and standards are consistently and thoroughly followed and enforced.

Most interviewees from both government and processing plants noted that the responsibility for ensuring the food they produce is safe rests with the meat processors. One respondent summarized the view of most interviewees by noting that "there are about 30 billion meals consumed each year in Canada and the number of foodborne illnesses has gone down. It's not because we have more inspectors, but because we have companies who accept responsibility and adopt better practices." Nonetheless, most interviewees identified that as a result of continuous inspector presence in facilities, there is a risk that, over time, the line between CFIA inspector and processing plant "staff" could become blurred. Some processing plants could inappropriately come to regard the inspector as "staff" and ask the inspector to help with operational decisions.

Food Safety Culture

Possible linkages between DSIP and enhanced food safety culture were explored. Some interviewees noted that, through increased frequency, inspectors deepen their familiarity with work flow and food safety practices at a particular plant. This in turn, can provide for increased "rigour" and issue identification during the inspection process, creating a greater awareness of food safety and thus enhancing the food safety culture. A contrasting view is that spending more time in the plants at any one visit, as was previously the case, can provide the opportunity for a more robust inspection and more time to perceive a broader view of the entire operation, instead of the daily task oriented approach now.

The Hawthorne Effect, or Observer Effect, is a well-documented and tested theory whereby productivity is improved with oversight. ^Footnote 21 In essence – the more that people are watched, the better they perform. Applied here it posits that the inspectors' more regular presence increases plant employee compliance. A contrasting viewpoint is that when the inspector is not at the plant, which is most of the time, the employees are not as compliant. This paternalistic perspective suggests that by not leaving it to the private sector to develop their own oversight, they will have a weak safety culture, reliant on CFIA inspection and only bolstered when inspectors are present. The Observer Effect suggests that it is not the act of watching that will ensure compliance all of the time. Whether or not the inspector is there once a shift verses twice a week still leaves the meat plant without an inspector present the vast majority of the time.

In summary, while a range of theories and opinions exist, there is no clear indication that food safety culture or food safety itself is either improved or diminished by increased inspection frequency through DSIP.

3.4.2 Other Outcomes

Some interviewees felt that DSIP in Canada has improved access for meat products to other countries. They added that most countries have adopted DSIP as a requirement for exporting their products to the U.S., and if Canada did not implement DSIP it would most likely hinder meat exports not only to the U.S. but to these other countries as well. No other data was available to further validate this claim.

The implementation of DSIP and the associated requirement for more inspectors has increased the cost of meat inspection to the CFIA. The requirement for 108 additional front-line inspectors is evidence of the additional cost associated with DSIP, a fact that many of the interviewees also pointed out. Many CFIA and plant manager interviewees indicated that DSIP has also increased inspection costs to companies. While it is not required, most plants have a Quality Assurance (QA) manager or HACCP (Hazard Analysis Critical Control Point ^Footnote 22) coordinator accompany the CFIA inspector in order to respond to any questions. ^Footnote 23 In particular, it was observed that many plants operating a midnight shift have hired an additional QA manager or HACCP coordinator in order to liaise with the CFIA inspector. This, together with the additional record keeping associated with DSIP increases inspection costs which naturally represents a relatively greater burden for smaller firms. Larger firms we spoke to indicated that inspection costs are not high relative to other operating costs of meat processing establishments.

Most inspection managers expressed concern that the meat sector receives a disproportionate amount of the Agency's budget when compared to other agricultural commodities such as dairy, fish, fruit and vegetables, and DSIP exacerbates this imbalance. For example, for 2011-12, the Agency allocated 35.2% of its budget to Meat and Poultry, and about 4% to Fresh Fruit and Vegetables. Recent US studies by the Centres for Disease Control ^Footnote 24 and the Centre for Science in the Public Interest ^Footnote 25 recognize produce as a major cause of foodborne disease outbreaks. The latter study also identifies seafood as a major source of foodborne disease. While it is recognized that ensuring daily presence is a non-discretionary requirement to maintain access to US export markets, the expressed concern raises a question as to whether the Agency is under-invested in some programs in comparison to meat.

3.5 Performance: Demonstration of efficiency and economy

3.5.1.1 Adequacy of Human and Financial Resources

Most CFIA interviewees indicated that current resources allocated to DSIP are adequate for ensuring an inspector is present at every shift; i.e., the additional 108 front-line inspectors, eleven additional supervisors and program specialists, and five additional management staff. Some CFIA interviewees noted, however, that unexpected contingencies such as illness, adverse weather or a food safety emergency, have, on occasion, forced the Agency to reallocate resources. An identified coping strategy is to cut back on the need for a DSIP on every shift in federally registered processing plants serving the domestic market only.

In a risk-based model, inspections of meat processing facilities occur on a regular basis. The DSIP initiative is intended to fund the additional inspections required to comply with a U.S. requirement. This distinction makes tracking spending on DSIP challenging, namely the inherent limitations in funding enhancements to an existing program. Add to this that CFIA inspectors carry out inspections in a variety of facilities; and inspectors may carry out a range of activities within a given facility – some of which are for DSIP purposes, as well as for other purposes. This is compounded by the fact that several Areas limit the number of programs to which they code inspectors in order to reduce the administrative burden and complexity of coding.

While some positions and activities are coded specifically to DSIP, due to the limitations expressed above, it is not unexpected that the amounts recorded do not necessarily align with the funding provided. According to the CFIA financial tracking system, for DSIP there was an approximate $4.35 million gap in planned versus actual spending in 2010-11, $1.19 million in 2011-12 and $2.1 million in 2012-13. Despite the apparent under-spending, due to the limitations outlined above, there are some within the Operations Branch that believe the actual amount spent is higher than what is reported in the CFIA financial tracking system and the Agency is instead under-reporting.

In order to better understand how much of the planned and allocated funding was spent, data was requested to demonstrate the number of inspections that were completed within relevant facilities over the evaluation period. Unfortunately, as per the aforementioned limitations, the information is not tracked in a retrievable fashion and we were unable to confirm how many total inspections were required to satisfy DSIP for domestic and international facilities. Similarly, data was unavailable to demonstrate how many inspections were conducted to satisfy daily shift requirements. Part of the challenge lies in the fact that new facilities are opened or some closed, and shifts added or removed regularly, making the total number of inspections required relatively fluid over time.

In order to assess current DSIP requirements against those that existed when the funding was granted, the Agency undertook an analysis of its human resource (inspector) capacity needs to satisfy DSIP based on the number of processing plants and shift coverage required (see Table 3.3 below). The underlying assumptions used to estimate the number of inspectors required (available inspector time, average travel time and average time per inspector task) received third party validation from Pricewaterhouse Coopers LLP. These assumptions were developed for a costing of the meat program. The analysis outlines the number of processing plants with single shifts and the number with multiple shifts, determines what the FTE requirement would have been without DSIP and calculates the FTE requirement with the DSIP requirement.

As of September 2013, the number of FTEs needed to maintain DSIP was 108.5. The number will decline or rise slightly as the number of registered processing plants increases and decreases as plants open and close, or the number of shifts expands and contracts. From 2010 to September 2013, the annual median number of registered establishments was 563. ^Footnote 26 Information is reportedly available to show that an inspection occurred at all shifts, however it is not maintained in an easily retrievable format as it is not collected for such a reporting purpose. Records are kept at processing facilities and regional CFIA offices that together would provide point in time data necessary to confirm inspectors were present at all shifts. A U.S. FSIS audit conducted of seven Canadian meat exporting facilities in 2012 found that verification worksheets showed that for all seven facilities the CFIA maintained daily inspection when product was produced for the U.S. ^Footnote 27

Although the financial tracking system indicates that funding was unspent each year, the number of human and financial resources needed to deliver DSIP has remained relatively consistent since 2010. The inspection managers interviewed indicated that an inspector is present at all shifts, with very few exceptions. These include situations where a replacement cannot be found for an inspector off due to illness or hampered by severe weather conditions. On occasion, another inspector might be found but an overtime cost would be involved.

3.5.1.2 Cost Recovery

The objectives of the Agency's cost recovery policy are to:

1. encourage effective and responsive service delivery;
2. promote a fair and consistent approach to program funding;
3. foster an understanding that user fees may be charged for services provided by the Agency, the use of facilities provided by the Agency, or the products, rights and privileges provided by the Agency; and
4. communicate the requirements of the User Fees Act (March 2004), the Treasury Board Policy on Service Standards for External Fees (November 2004), and the Treasury Board's Guide to Costing. ^Footnote 28

No data was available on the amount recovered from user fees for processing plants specific to DSIP; however, a number of interviewees indicated that CFIA is not recovering the full cost of administering DSIP. ^Footnote 29 Some suggested the Agency explore the feasibility of increasing the fee charged to plants, particularly on the weekend and overtime shifts. The CFIA is currently reviewing its service standards and fees to verify that fees are in line with the actual cost of delivering the service.

3.5.1.3 Cost-Effectiveness

The majority of interviewees indicated that DSIP is not a cost-effective approach in terms of food safety. It would be more effective to allocate resources according to the level of risk (in their view), but clearly to do so would jeopardize meat exports to the U.S. As noted above, an alternative to the current approach that would improve cost effectiveness would be to confine DSIP to exports to the U.S. and to use a risk-based approach for the domestic market. Such a change could, however, risk compromising Canadian consumer confidence in Canada's food safety system and would establish what could be perceived as higher requirements for international vs domestic markets.

It is the domestic DSIP costs, both to the CFIA and to industry, that were identified in particular by interviewees as unnecessary because they are not in line with the Agency's risk-based approach. Examples provided by interviewees include the cost of having an inspector in a plant every day and for every shift leading to increased transportation costs (travel between plants), more overtime costs (e.g., to cover a third shift, holiday shifts), and having to send an inspector to a plant even if the shift is short (e.g., 3 hours). For example, one plant indicated that they operate 13 to 14 hours each day, and because they are just over the twelve hour shift, under DSIP, CFIA is required to send a second inspector for the last 1-2 hours the plant is operating.

4. Conclusions

Daily shift inspection presence is an important component of CFIA's inspection program as it satisfies U.S. standards for the import of processed meat products and ensures continued exports of Canadian meat to the U.S. Given its alignment with federal trade priorities and with CFIA's regulatory responsibilities, it continues to remain relevant to the mandate of the CFIA.

Although DSIP is meeting its primary outcome, to ensure continued exports to the U.S., and while the program is part of the CFIA's overall food safety regime, the evaluation was unable to conclude on its impact on food safety.

The CFIA financial tracking system does not provide an accurate financial profile due to limitations inherent in the funding of enhancements to an existing program. Using past analysis methodologies, the coverage requirements to implement DSIP appear relatively unchanged since the initiative was last renewed. Information on user fees recovered specifically for processing plants was not available; however it seems unlikely that full costs are being recovered.

Although there was limited financial information, DSIP is clearly not cost-effective from a purely food-safety perspective. A risk-based approach to inspection would be more cost-effective and in-line with the principles of inspection modernization. However, DSIP is required to maintain access to the U.S. market, which it has achieved. The value of maintaining open trade and collaborative relationships with the US cannot be tangibly measured. Optimizing cost-effectiveness would remove the DSIP requirement in domestic plants, returning to a risk-based approach in line with current CFIA direction. However, implementing a dual system that could appear less onerous for domestic products brings associated challenges and could undermine confidence in Canada's food safety system.

Based on the financial costing model and analysis undertaken by CFIA, current resources allocated to DSIP appear to be in line with previous levels. The lack of availability of data to confirm the number of inspections required and conducted, coupled with a lack of specific costs for delivery, prevents further validation.

5. Recommendation

There are no Agency reports that demonstrate that the DSIP coverage requirement was met. Some resource reallocation is occurring, particularly on the domestic side. Information was not available (without investment of considerable time) to demonstrate DSIP coverage and when and where reallocation is occurring.

Recommendation: The Operations Branch should identify and track performance measurement indicators that would illustrate that the necessary number of inspections is undertaken each year.

Appendices

A) Logic Model

B) Bibliography

C) Interview Guides

D) Evaluation Matrix

A. Logic Model

Activities

Five main activities associated with daily shift presence were identified based on the initial document review as follows.

1. Front-line inspection capacity increased: 108.0 FTE front-line inspectors added, at a cost of $21.2 million over two years
2. Supervisory and program specialists' capacity increased: 10.6 FTE supervisors and program specialists added, at a cost of $2.4 million over two years
3. Program management capacity increased: 5.0 FTE program management staff added at national headquarters and the four operational areas (Atlantic, Quebec, Ontario, and Western), at a cost of $1 million over two years
4. Training for new inspectors: 1.0 FTE added for inspector training of additional inspectors, at a cost of $1.0 million over two years
5. Inspection of federally registered meat processing facilities
6. Inspection fee collected

Outputs

Four main outputs were identified as follows.

1. Increased inspection frequency (or daily shift presence) maintained at federally registered meat processing facilities certified to export to the US
2. Supervisory, expert specialist and management capacity maintained to coordinate and support increased inspection frequency activities performed by front-line inspectors
3. Complement of well-trained inspection staff maintained at increased inspection frequency requirements
4. Additional costs associated with daily shift presence recovered

Immediate Outcomes

Three immediate outcomes were identified as follows.

1. Gaps between FSIS and CFIA meat inspection equivalency requirements addressed
2. Requirements and procedures related to equivalency of meat inspection systems streamlined
3. Canadian meat products compliant with Canadian and US laws and regulations

Intermediate Outcome

Three intermediate outcomes were identified as follows.

1. Food safety culture enhanced in federally registered meat processing facilities
2. International and domestic confidence in Canada's food safety systems maintained
3. Canadian meat products exported to the US unhindered

Ultimate Goal

The ultimate goal of increased inspection frequency is as follows.

• A safe and accessible food supply and plant and animal resource base

Increased Inspection Frequency Logic Model

A logic model showing the logical relationships between the activities, outputs and outcomes of increased inspection frequency is provided in Figure A-1 below.

Figure A-5-1: Daily Shift Presence Logic Model

Description for Figure A-5-1: Daily Shift Presence Logic Model

B. Bibliography

Agriculture and Agrifood Canada, Announcement on Chile: http://www.agr.gc.ca/cb/index_e.php?s1=n&s2=2013&page=n130111

Acheson David, Determining the "High Risk Foods": Another Analysis Sees Things Differently: Food Safety News (April 8, 2013): http://www.foodsafetynews.com/2013/04/determining-the-high-risk-foods-another-analysis-sees-things-differently/

Australian Department of Agriculture, Fisheries and Forestry, AEMIS Meat Inspection: http://www.daff.gov.au/biosecurity/export/meat/elmer-3/meat-inspection-aemis-package

Australian Law, Export Controls Legislation: http://www.comlaw.gov.au/Details/F2013L00502

Baker Library, Hawthrone Effect: http://www.library.hbs.edu/hc/hawthorne/09.html#nine

Berkeley Rice, Hawthrone Defect: Persistence of a Flawed Theory: Psychology Today (1992), 2

Bloomberg, Meat Producers Sue Over Country-of-Origin Labeling Rules: http://www.bloomberg.com/news/2013-07-09/meat-industry-groups-sue-u-s-over-country-of-origin-labeling.html

Canada-U.S. Regulatory Cooperation Council: http://actionplan.gc.ca/en/page/rcc-ccr/regulatory cooperation-council

Canadian Food Inspection Agency, Cost Recover Policy and Framework: http://www.inspection.gc.ca/about-the-cfia/accountability/other-activities/sound-agency-management/service-standards-and-user-fees/cost-recovery-policy/eng/1378073377319/1378073471053

Canadian Food Inspection Agency, Inspection Modernization: http://www.inspection.gc.ca/about-the-cfia/transforming-the-cfia/inspection-odernization/eng/1337025084336/1376482277925

CBC, Canada-EU Trade Discussion: http://www.cbc.ca/news/canada/story/2013/06/03/pol-beef-canada-europe-trade-talks.html

CBC, Canada-US Meat Labelling: http://www.cbc.ca/news/business/story/2013/05/24/business-meat-labelling.html

CBC, Meat producers sue over U.S. labelling laws: http://www.cbc.ca/news/canada/story/2013/07/10/meat-labelling.html

Financial Post, Canadian Imposition of Tariffs on US: http://business.financialpost.com/2013/06/07/canada-threatens-to-impose-tariffs-on-u-s-foods-furniture-in-meat-labelling-dispute/

George Morris Centre, Country of Origin Labeling: The Damage Done and the Fight Underway: http://www.georgemorris.org/publications/file.aspx?id=ddba3cca-4061-418f-b8dc-6dae0f3cf1b7

Government of Canada, Foreign Affairs and International Trade Canada: What Canadians Told Us: A Report on Consultations on Regulatory Cooperation between Canada and the United States (2011): http://www.actionplan.gc.ca/grfx/psec-scep/pdfs/RCC-CCR_Eng.pdf

Government of Canada, Speech from the Throne, June 3, 2011: http://www.speech.gc.ca/eng/media.asp?id=1390

Government of Canada Speech from the Throne, October 16, 2013: http://www.speech.gc.ca/eng/full-speech

Levitt, Steven D. & List, John: Was there Really a Hawthorne Effect at the Hawthorne Plant? An Analysis of the Original Illumination Experiments: American Economic Journal: Applied Economics (Jan 2011).

Mexico, SENASICA: http://www.senasica.gob.mx/?Idioma=2&id=745

Mexico, Legislation on Meat Inspection: http://legismex.mty.itesm.mx/normas/zoo/zoo009.pdf

Mexican Embassy in Canada, Certified Mexican Plants to Export to Canada: http://embamex.sre.gob.mx/canada_eng/index.php?option=com_content&view=article&id=5373:canada-authorizes-meat-shipments-from-eight-mexican-federal-inspection-level-plants&catid=556:june-2013&Itemid=149

Mexico, TIF: http://www.anetif.org/

New Zealand, Code of Hygienic Practice: http://www.foodsafety.govt.nz/elibrary/industry/code-of-hygienic-practise-meat.pdf

New Zealand, Food Safety FAQ about New Inspection Requirements: http://www.foodsafety.govt.nz/elibrary/industry/meat-inspection-change-q-a.pdf

New Zealand, Manual 16 on Meat Inspection: http://www.foodsafety.govt.nz/industry/sectors/meat-ostrich-emu-game/meatman/manual-16/manual-16-post-mortem-examination-procedures.pdf

New Zealand, Manual 16 Amendment 17: http://www.foodsafety.govt.nz/industry/sectors/meat-ostrich-emu-game/meatman/manual-16/manual-16-amendment-17.pdf

New Zealand Food Safety Authority, Animal Products Notice 2006: http://www.foodsafety.govt.nz/elibrary/industry/animal_products-specifications_that.pdf

Painter John A., Hoekstra, Tracy, Tauxe Robert, Braden, Christopher et al: Attribution of Foodborne Illnesses, Hospitalizations, and Deaths to Food Commodities by using Outbreak Data, United States, 1998-2008: Emerging Infectious Diseases Journal (March 2013), Volume 19, Number 3.

StatsCanada, Trade Information on Meat and Edible Meats: http://www5.statcan.gc.ca/cimt-cicm/chapters-chapitres?countryId=999&usaState=0&dataTransformation=1&save=null&trade=null&freq=12&lang=eng&refYr=2013&sectionId=1&sectionLabel=I+-+Live+animals+and+animal+products&retrieve=Retrieve&provId=1&refMonth=5

StatsCanada, Trade Information on Meat and Edible Meats: http://www5.statcan.gc.ca/cimt-cicm/chapters-chapitres?lang=eng&sectionId=1&dataTransformation=1&refYr=2012&refMonth=5&freq=12&countryId=999&usaState=0&provId=1&retrieve=Retrieve&save=null&trade=null&sectionLabel=I%20-%20Live%20animals%20and%20animal%20products.

RéseauSPAANetwork, Differential Impacts of Country of Origin Labeling: COOL Econometric Evidence from Cattle Markets: http://www.spaa-network.fsaa.ulaval.ca/uploads/tx_centrerecherche/Cool_econometrics_-_SPAA_WP_2012-14.pdf

Reuters, Removal of Ban on Canadian Meats from Chile: http://www.reuters.com/article/2013/05/30/beef-canada-chile-idUSL2N0EB0WQ20130530

Reuters, Canadian Imposition of Tariffs on US: http://www.reuters.com/article/2013/06/07/canada-us-trade-idUSL1N0EJ16420130607

United States Department of Agriculture, Assessment of Food Safety and Inspection Service Inspection Personnel Shortages in Processing Establishments (April 2012): http://www.usda.gov/oig/webdocs/24601-0011-HY.pdf

United States Department of Agriculture, Food Safety and Inspection Service, Final Report of an Audit Conducted in Canada, August 25 through October 1, 2009: Evaluating the Food Safety Systems Governing the Production of Meat and Poultry Products Intended for Export to the United States of America (October 28, 2010).

United States Department of Agriculture, Foreign Agricultural Service: http://www.fas.usda.gov/gats/ExpressQuery1.aspx

United States Department of Agriculture: http://www.fsis.usda.gov/wps/portal/fsis/topics/international-affairs/import-and-export-program

United States Office of International Affairs, Import Data: http://www.fsis.usda.gov/wps/wcm/connect/b52a03cc-a0ef-4679-ac45-afccb57ffc23/import_summary_2011.pdf?MOD=AJPERES

WHO Codex Alimentarius: http://www.codexalimentarius.org/

C. Interview Guides

Non-plant Interviewees

Background

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch has contracted Hickling Arthurs Low Corporation to conduct an evaluation of the increased frequency of food inspections in meat processing establishments. Increased inspection frequency was implemented in meat processing establishments in response to the 2009 United States Department of Agriculture, Food Safety and Inspection Service audit recommendation.

You have been identified as an individual who can speak about key issues related to the increased frequency of food inspections; therefore, we would like to conduct a 30 minute interview with you by telephone to gather your perspectives. Your responses will be reported in aggregate only and without attribution.

Questions

1. What is your role in connection with daily shift inspection presence?

• How long have you been in this position?

Relevance

2. What issues does the daily shift presence address, for example:

• food safety;
• food safety culture;
• trade, regulatory;
• or other?

3. Is there continued need for daily shift presence to address these issues?

4. Is daily shift inspection presence consistent with government-wide priorities and the CFIA mandate?

5. Is there a necessary role for the federal government in daily shift inspection presence?

Performance

6. Is the daily shift presence achieving its intended outcomes including:

• Producing Canadian meat products compliant with Canadian and US laws and regulations
• Maintaining international and domestic confidence in Canada's food safety and systems
• Providing unhindered access for Canadian meat products to the US?

7. Have there been other outcomes of the daily shift presence, both positive and negative, for example:

• Improved access for meat products to other countries
• Improved access to the US in other trade areas
• Extra cost and/or inconvenience to companies?

8. Have the allocated financial resources and human resources been adequate for daily presence?

9. Is daily presence cost-effective as currently delivered?

• Are there any improvements or efficiencies that could be made?

10. Are there alternatives to daily shift presence?

Other Comments

11. Any other aspects of daily shift presence that might be examined in the evaluation?

12. Who are the key individuals in CFIA and outside that should be consulted in this study?

Plant Interviewees

Background

The Canadian Food Inspection Agency's (CFIA) Audit and Evaluation Branch has contracted Hickling Arthurs Low Corporation to conduct an evaluation of the increased frequency of food inspections in federally registered meat processing establishments (also, referred to as "daily shift inspection presence"). Increased inspection frequency was implemented in meat processing establishments in response to the 2009 United States Department of Agriculture, Food Safety and Inspection Service audit recommendation.

You have been identified as an individual who can speak about key issues related to the increased frequency of food inspections; therefore, we would like to conduct a 30 minute interview with you by telephone to gather your perspectives. Your responses will be reported in aggregate only and without attribution.

Questions

1. What is your role in connection with daily shift inspection presence?

• How long have you been in this position?

Relevance

2. What issues does the daily shift inspection presence address, for example:

• food safety;
• food safety culture;
• trade;
• regulatory; or
• other?

3. Is there continued need for daily shift inspection presence to address these issues?

4. Is there a necessary role for the federal government in daily shift inspection presence?

Performance

5. Is the daily shift inspection presence achieving its intended outcomes including:

• Producing Canadian meat products compliant with Canadian and US laws and regulations;
• Maintaining international and domestic confidence in Canada's food safety and systems;
• Providing unhindered access for Canadian meat products to the US?

6. Have there been other outcomes of the daily shift inspection presence, both positive and negative, for example:

• Improved access for meat products to other countries;
• Improved access to the US in other trade areas;
• Extra cost; and/or
• Inconvenience to companies?

7. Are there any improvements or efficiencies that could be made to daily shift presence?

8. Are there alternatives to daily shift presence?

Other Comments

9. Any other aspects of daily shift presence that might be examined in the evaluation?

D. Evaluation Matrix