The Establishment-based Risk Assessment model

The Canadian Food Inspection Agency (CFIA), within the scope of the modernization of its inspection system, developed an Establishment-based Risk Assessment (ERA) model that aims to identify the different risks associated with food establishments. This risk assessment takes into consideration typical food safety hazards, and will be used to determine the level of oversight required to appropriately manage the risks. The ERA model has been developed by CFIA staff in collaboration with experts from academia, industry and other government departments. The development of the model also drew upon the experience of other countries that have used a similar approach to risk assessment, and considered scientific literature and leading edge modelling technology.

The ERA model is being implemented in a step-wise approach, using an electronic data collection method as part of the regular inspection activities. In order to implement the ERA model, CFIA inspectors will collect additional tombstone data related to the establishments' food products and their production volume, mitigation strategies, and compliance factors as part of the regular inspection. The model output for a specific food establishment will be expressed as an estimated annual number of Disability Adjusted Life Years (DALY's). After a cycle of data collection and analysis by the ERA model, the ERA results will provide input into the Agency's risk-based approach to managing food safety risks, including the prioritization of inspection, oversight strategies and priorities, laboratory capacity mobilization and annual work plans. After the ERA is fully implemented in all registered sectors (by the end of 2017), the Agency will implement the ERA model in the currently non-federally registered sectors, when the Safe Food for Canadians Regulations (SFCR) come into force. The ultimate objective of this initiative is to produce a near real time risk assessment result for individual food producing establishments that will assist the Agency with its overall Risk Management Strategy.

This report summarizes the scientific approach i.e. the development steps since 2013, and the practical application and performance assessment. Annex 1 presents the Scientific Advisory Committee members and the ERA technical team.

Scientific Approach

Expert elicitation 1: Identification of important risk factors for the ERA model

The objective was to assess the importance and significance of 155 risk factors identified in the literature that could potentially be used in a risk assessment model. Expert selection was based on nomination by the CFIA scientific advisory committee (SAC). Overall, 126 Canadian experts were invited to participate; 75 participated (60%) by completing an electronic survey in June and July 2013. A scale from 1 to 10 was used to score the risk factors, 1 having the least impact on food safety and 10 having the highest impact. Experts attributed a high score to most risk factors. One risk factor had a median score of 10, 43 had a median score of 9, 75 had a median score of 8 and 36 had a median score of less than 8.

Respondent profile did not have a strong influence on the attributed score. There was a general and uniform thinking between experts. As this survey did not completely meet the objective of discriminating risk factors, a process was needed to refine and select risk factors to be included in the model. The criteria used for the selection and refinement of risk factors included: the availability of data sources, the clarity and precision (i.e. definition) of the selected factors, the elimination of lower-rated risk factors (from the first expert elicitation), the grouping of risk factors with similar focus, and how measurable the selected factors were (e.g. possibility to be objectively assessed during an audit process). The final list of risk factors is presented in Figure 1. Risk factors are grouped based on three different groups: the inherent risks factors, mitigation factors and compliance factors.

  • Inherent risk factors represent the risks associated with a specific food commodity, operation or manufacturing process. These risks take into account the type of product, volume, and its direct distribution to a vulnerable population, such as residents of nursing homes, hospitals or daycares.
  • Mitigation factors represent the measures or strategies that a food establishment is using to reduce the inherent risk and therefore reduce the risk of a food safety issue. Examples of these strategies include the implementation of an internationally recognised private certification scheme (i.e. preventive control plan), having a full time employee responsible for quality assurance and food safety on site, and the application of specific risk-prevention processes (e.g., use of antimicrobials).
  • Compliance factors refer to a food establishment's track record with respect to how well it has complied with its own preventive control measures and with regulatory requirements. This is assessed using the food establishment's historical and current data such as information pertaining to recalls, inspection reports and enforcement actions taken.

Figure 1: Final list of risk factors included in the ERA model

Figure 1: Final list of risk factors included in the Establishment-based Risk Assessment model. Description follows.
Description for image – Final list of risk factors included in the ERA model

This figure depicts the final list of risk factors included in the ERA model. The yellow box represents the inherent risk factors which are the risks associated with a specific food commodity, operation or manufacturing process. Note that the factors with an asterisk are related to the health impact (unit in DALYs) attributed to these risk factors. The blue box represents the mitigation factors which are the measures or strategies that a food establishment has implemented to control the inherent risks and reduce the overall risk of a food safety issue. The red box represents the compliance factors which refer to a food establishment's track record on how well it has complied with its own preventive control measures and with regulatory requirements. The list under the inspector assessment is the list of preventive control plan (PCP) sub-elements that the ERA considers under the CFIA's integrated Agency Inspection Model (iAIM).

Expert elicitation 2: Risk factor weighting for the ERA model

The objective was to estimate the relative weight of each risk factor included in the ERA model. Expert selection was based on nomination by the SAC. Overall, 50 experts were invited to participate; 29 participated (58%) to a face to face Delphi process with two rounds on December 17th 2014. The first round occurred in the morning by using a web-based questionnaire. During lunch, results from all experts were summarized in a report and printed. The report presented the number of answers, the minimum and maximum values and the 3 quartiles (Q1, Q2/median, Q3) for each question. The second round occurred during the afternoon. Experts were asked to compare, adjust and discuss their results. There was a good consensus on the weighing given to most risk factors. Respondent profile did not have a strong influence on the risk factor weighting. No expert expressed formal opposition for the inclusion of any risk factor. However, most of them recommended more consideration for risk factors at the farm-level in the ERA model such as using suppliers that are participating to an on-farm food safety programs recognised by the CFIA, surveillance programs and biosecurity program at the farm-level. Since experts used different scales to weight each risk factor, results from the second round were standardised on a scale of 1 to 25. Median scores of rescaled data are used in the ERA model.

Expert elicitation 3: Attribution of sources to food sub-products

The objective was to estimate the contribution of different food sub-products to the level of human illness in the Canadian population. A web-based questionnaire was used to determine the attribution of sources for the main food safety pathogens in all commodities. Expert selection was based on nomination by the SAC and the technical committee. Then, a snow-ball approach was used to nominate other experts. Overall, 119 experts were invited to participate in completing the expert elicitation survey; 64 completed the questionnaire. The survey was composed of 4 sections including the expert profile, the distribution of risks along the supply chain, the contribution of different types of hazards (chemical, microbiological and allergens) to the Canadian foodborne health burden and the source attribution at the sub-product level for 32 pathogen-commodity combinations. For each pathogen-commodity combination, experts were asked to provide their level of certainty on a scale from 1 to 10 (1 being less confident and 10 being the most confident). Experts were offered two versions of the questionnaire, depending on their choice. The first version grouped questions based on the type of pathogen, while the second version grouped questions based on the type of commodity. A Food Safety Hazards Background document was provided to experts before completing the survey. There was a good consensus on the attribution of sources to food sub-products. The distribution of risk at the food supply chain was mainly attributed at the federal establishments in the dairy and fish & seafood sectors, and at the farm-level for the produce and egg sectors. The majority of the experts attributed a high relative contribution of risk to microbiological hazards compared with chemical hazards and allergens. For most of the pathogen-commodity combinations, respondent profile did not have a significant influence on the source attribution at the sub-product level. Since experts provided different levels of certainty for each pathogen-commodity combination, a weighted average has been used to calculate the source attribution at the sub-product level.

Practical Application and Performance Assessment

Meat, poultry and dairy pilot project

The objectives of the meat/poultry and dairy products pilot project were to test and simplify the ERA model prototype, to obtain ERA results and to validate the gathering tools. Forty-nine meat and poultry establishments and 29 dairy establishments located in Quebec and Ontario participated to the pilot project between March and June 2014. As a result of the pilot project data analysis, as well as feedback from establishments and inspectors, a simplified and improved version of the data collection tools was developed. Based on data obtained during the pilot project, summaries were created for each establishment for the comparison of the ERA model results and the risk assessment inspector results. Figures 2 and 3 present the ERA results which incorporate inherent risk, mitigating factors and compliance factors for meat/ poultry, and dairy establishments respectively. Figure 4 presents the distribution of the final risk results across all the establishments that participated in the pilot. From these figures, we can conclude that most of the risk related to food safety is attributed to very few establishments. Figure 4 shows that the top 5 riskiest establishments are responsible for 70% of the cumulative risk.

Figure 2: ERA results of 49 meat and poultry establishments

Click on image for larger view
Figure 2: Establishment-based Risk Assessment results of 49 meat and poultry establishments. Description follows.

Description for image – ERA results of 49 meat and poultry establishments

Figure 2 depicts the ERA results measured in DALYs for the 49 meat and poultry establishments that participated in the pilot. The blue bar represents the inherent risk result which takes into consideration only the information related to the inherent risk factors. The red bar represents the mitigated risk result which is the inherent risk adjusted with the mitigation factors. The green bar represents the final risk result which combines the inherent and mitigated result with the compliance factors.

Figure 3: ERA results of 29 dairy establishments

Click on image for larger view
Figure 3: Establishment-based Risk Assessment results of 29 dairy establishments. Description follows.

Description for image – ERA results of 29 dairy establishments

Figure 3 depicts the ERA results measured in DALYs for the 29 dairy establishments that participated in the pilot. The blue bar represents the inherent risk result which takes into consideration only the information related to the inherent risk factors. The red bar represents the mitigated risk result which is the inherent risk adjusted with the mitigation factors. The green bar represents the final risk result which combines the inherent and mitigated result with the compliance factors.

Figure 4: Cumulative risk attributed to meat/poultry and dairy establishments from the pilot

Click on image for larger view
Figure 4: Cumulative risk attributed to meat/poultry and dairy establishments from the pilot. Description follows.

Description for image – Cumulative risk attributed to meat/poultry and dairy establishments from the pilot

Figure 4 depicts the cumulative risk attributed to the meat/poultry and dairy establishments from the pilot. The blue line represents the total percentage of final risk results based on the percentage of establishments that participated in the pilot. The first vertical red line shows that over 80% of the establishments are responsible for only 5% of the total risk. The second vertical red line demonstrates that the top 5 establishments with the highest risk (i.e. only 6% of the establishments) are responsible for 70% of the total risk.

Performance evaluation of ERA model outputs from the meat/poultry and dairy pilot project

The objectives were to estimate the agreement between the risk assessment provided by the ERA model and by CFIA senior inspectors, and to refine the model based on the identification of major discrepancies. Sixty-five CFIA senior inspectors participated to the project between December 2014 and January 2015 (71 were identified and 67 followed the information sessions). For each establishment that participated to the meat/poultry and dairy pilot project, information related to the risk factors used as inputs in the ERA model was summarized in one page. Each expert categorized 10 establishments for their risk to the health of Canadian consumers.

Each expert received 8 randomly selected establishments and 2 controls: one with the lowest risk and one with the highest risk. The controls were created by the ERA technical committee.

Results showed there was a good correlation between the scores assigned by the experts and those obtained with the model for the dairy establishments (r=0.63, p<0.01). For the meat/poultry establishments, there was a lower (r=0.48, p<0.01) correlation. However, the correlations were good for slaughter plants (r=0.53, p= 0.05), storage plants (r=0.70, p=0.01), and large size meat/poultry establishments (r=0.72, p=0.02). Further statistical tests were used to explore predictors of the difference between the scores assigned by the ERA model and the experts, considering various characteristics of the establishments (i.e. type of establishment, volume, number of recalls, number of non-compliances, enforcement actions). In this regard, storage plants and slaughterhouses were scored systematically higher (i.e. riskier) by the model than by experts, compared to ready-to-eat (RTE) products' manufacturing plants. By scoring RTE product establishments higher, experts seem to take into consideration the fact that no further step will be taken by consumers to decrease the microbiological risks. In addition, establishments manufacturing products that fall under the Health Canada Listeria category 1 policy were scored systematically higher by experts than by the model. Furthermore, establishments with good inspection results and track records received higher scores by the model compared to experts. In other words, compared to the model, CFIA senior inspectors gave more credit to the good results obtained for variables that are related to inspection activities and track records, while the model seems to take into account other risk factors like volume or the subtype of product. This effect is supported by the relative weight assigned by experts (academia, government and industry) to each individual risk factor (second expert elicitation).

By the end of 2017, the ERA model will be tested and its performance assessed in all other registered commodities, such as fish and seafood and eggs and egg products. The ERA technical team is currently adapting the ERA model for importers (Importer Risk Assessment model) and is developing a Hatchery Risk Assessment model considering food safety risks. More details on the Establishment-based Risk Assessment model will be available through peer-reviewed scientific articles. All questions can be sent to RAModel-ModeleER@inspection.gc.ca.

Annex 1 – The Scientific Advisory Committee members and the ERA technical team

Scientific Advisory Committee members
Name Affiliation
Sylvain Quessy Université de Montréal
Julie Arsenault Université de Montréal
Ann Letellier Université de Montréal
Mansel Griffiths University of Guelph
Art Hill University of Guelph
Jeff Farber University of Guelph
Sylvain Charlebois Dalhousie University
Tom Gill Dalhousie University
Rick Holley University of Manitoba
Danielle Brule Health Canada
Aamir Fazil Public Health Agency of Canada
Aline Dimitri Canadian Food Inspection Agency
Anna Mackay Canadian Food Inspection Agency
ERA technical team
Name Affiliation
Manon Racicot Canadian Food Inspection Agency
Romina Zanabria Canadian Food Inspection Agency
Hargun Chandhok Canadian Food Inspection Agency
Irina Frenkel Canadian Food Inspection Agency
Alexandre Leroux Canadian Food Inspection Agency
Geneviève Comeau Canadian Food Inspection Agency
Raphaël Plante Canadian Food Inspection Agency
Eva Pietrzak Canadian Food Inspection Agency
Eva Kaminski Canadian Food Inspection Agency
Alain Pilote Canadian Food Inspection Agency
Suzanne Savoie Canadian Food Inspection Agency
Cécile Ferrouillet Université de Montréal
Marie-Lou Gaucher Université de Montréal
Date modified: