Ethics and evidence generation: Steps to guide public health evaluations and promote ethical reflection throughout the lifecycle of an evaluation

Contents

I  Introduction
II What guides ethical reflection and practice?
III Three steps to incorporate ethical reflection into an evaluation
IV Lifecycle of an Evaluation: Ethical Considerations and Strategies
V Conclusion

--Submitted by Kim Bergeron, Charoula Tsamis, Allison Meserve

I  Introduction

Evidence generation plays a pivotal role in informing public health practice and provides numerous benefits, including access to more high-quality information about what works and what doesn’t, increased likelihood of successful programs and policies and more efficient use of public health resources. [1] Common public health evidence-generating initiatives, like surveillance, evaluation, quality improvement and research, involve the systematic collection of data about individuals or communities, their environments and the health and social services they receive or provide. [2] Ethical issues may arise at any time throughout the lifecycle of evidence generation, from inception to knowledge dissemination. With increasing demands for evidence-informed public health practice, public health practitioners engaged in evaluation, in particular, face unique and sometimes challenging ethical dilemmas. [3]

The purpose of this article is to offer three steps to guide public health evaluations and highlight ethical considerations during an evaluation. The intent is to encourage ethical reflection, throughout the lifecycle of an evaluation and to offer strategies to increase ethical integrity.
 

II What guides ethical reflection and practice?

Take a moment and think about the frameworks or principles that guide your professional practice. What do you take into consideration when planning to collect or use data about individuals or communities and protect and promote their best interests? Is it your organizations’ policies and procedures? If you are a registered professional, is it your professional standards? There has been limited attention paid to the development of ethical guidance addressing the context of public health evaluation. For example, the Canadian Evaluation Society Competencies for Canadian Evaluation Practice [4] is limited in its consideration of the potential harms to participants, communities and populations. Additionally, existing research ethics guidelines, such as the Tri-Council Policy Statement 2, Ethical Conduct of Research (TCPS 2) [5] tend to focus on protecting the welfare of the individual whereas public health initiatives focus on protecting the welfare of the public as a group or population. To assist public health practitioners in operationalizing the TCPS 2 within their public health practice, Public Health Ontario (PHO) has developed a framework for the ethical conduct of public health evidence-generating initiatives. This framework provides a public health lens to the traditional core research ethics principles of respect for persons, concern for welfare, and justice and can be applied to any public health evidence-generating initiative, not just research. [2]

III Three steps to incorporate ethical reflection into an evaluation

Your first step to incorporate ethical reflection into an evaluation is to complete the TCPS 2 Tutorial. This is a free, self-paced, online course that includes interactive multi-disciplinary examples and once completed, provides a certificate of completion. [6] To complement the TCPS 2, your second step should be to  refer to and understand PHO’s Framework for the Ethical Conduct of Public Health Initiatives.2  Finally, consult PHO’s ten steps for conducting an evaluation [7] (Figure 1). These resources will help you conduct a rigorous and useful public health evaluation that also protects the welfare of evaluation participants, communities and populations.    

Figure 1: Public Health Ontario’s ten steps for conducting an evaluation. [7]
Figure 1: Public Health Ontario’s ten steps for conducting an evaluation.

 

 

 

 

 

 

 

 

 

 

 

IV Lifecycle of an Evaluation: Ethical Considerations and Strategies

This section of the article will discuss how you can apply ethical principles to identify potential ethical issues and strategies to mitigate these issues. It highlights the three phases of the lifecycle of an evaluation: 1) planning, 2) implementation; and 3) utilization. The intent is to increase ethical reflection and minimize implementation challenges.

Planning Phase: The purpose of this phase is to plan how an evaluation will be implemented by considering the ten guiding questions outlined in PHO’s Framework for the Ethical Conduct of Public Health Initiatives. [2] In this phase, all ten ethics questions should be considered to appropriately plan and ensure the ethical implementation of the evaluation. Ethical issues and questions may arise during the implementation phase, particularly regarding participant selection, recruitment, consent, privacy and confidentiality; however, considering these issues and questions during the planning phase will help anticipate issues and provide mitigation strategies in the event that they occur. PHO’s ten guiding ethical questions are:

  1. What are the objectives for the initiative? How are they linked to potential improvements in public health?
  2. Can the objectives be achieved using the proposed methods?
  3. Who are the expected beneficiaries of the knowledge gained or other benefits?
  4. What are the burdens and potential harms associated with the proposed initiatives? Who bears them?
  5. Are burdens and potential harms justified in light of the potential benefits to participants and/or to society?
  6. Is selection of participants fair and appropriate?
  7. Is individual informed consent warranted? Is it feasible? What level of engagement is appropriate?
  8. Is community engagement warranted? Is it feasible? What level of engagement is appropriate?
  9. What are the social justice implications of this initiative?
  10. What are the potential longer-term consequences? [2]

The first two questions set the stage for the evaluation plan. For example, during the initial stages of planning an evaluation, it is important to ensure that there is a clear link between the proposed evaluation and potential public health benefits. The expected benefits will serve as an anchor throughout the evaluation plan in terms of justifying the potential risks and costs associated with the evaluation. [3] Furthermore, proposed methods that lack methodological rigor and that are not linked to the evaluation objectives could lead to harms by wasting resources, affecting the delivery of programs, disrupting program staff, allowing harmful programs to continue and beneficial programs to be terminated. [3]

The third question helps identify how different stakeholders can benefit from the evaluation. Within the public health context, stakeholders can include program clients such as individuals and/or communities, program staff and government or boards of health. These stakeholders may have shared or competing interests in the evaluation results [2] therefore it is important to consider different stakeholder interests as this can help inform the planning, implementation and the utilization phase of an evaluation. [8] When competing interests arise, refer to the guiding principles of evaluation integrity (e.g., being faithful to the original evaluation plan) and protecting the welfare of evaluation participants to aid you in your decision making process. [3]

The fourth and fifth questions focus on considering burdens and risks associated with the proposed evaluation, not the program or service being evaluated. From an ethics perspective, burdens and risks involve the potential negative effects or harms that an evaluation can have on the welfare of participants or communities. Consideration of burdens and risks includes identifying physical, mental and social harms for participants or communities and taking into account the differing contexts of those who may be affected in relation to the design of the evaluation. To be considered ethical the benefits should be proportionate to, or outweigh the burdens and risks. [2] It is understood that with any evaluation there will be some degree of risk; however, it is the responsibility of those conducting the evaluation to ensure participants are not exposed to unnecessary harm and, if there are potential harms, attempt to minimize or mitigate them. [3]

Questions six and seven also consider burdens and risks in light of the benefits in the context of participant selection, recruitment and consent. Question six focuses on the selection and recruitment of participants to ensure a fair distribution of burdens, risks and benefits. For example, it is important to ensure that particular populations such as marginalized populations are not inappropriately excluded from the benefits of evaluation or inappropriately included when there is greater potential for burdens or risks. [2] Determining whether voluntary, informed and ongoing consent is warranted should also be considered, as well as how it should be appropriately implemented (i.e., written, verbal, implied, opt out approach)  and whether it is even feasible (e.g., secondary use of data from transient populations). [2]  When in doubt about when consent is warranted, ask yourself whether the participant has a choice to participate or provide you with information about themselves, because consent is about respecting an individual’s freedom to choose whether they want to participate or not. [3]

Question eight relates to the role of community engagement. Benefits for engaging the community include validating and informing the evaluation plan and dissemination strategy, and identifying benefits and risks as well as mitigation strategies. [2] For example, engaged community stakeholders can identify whether data collection tools are written in a language understandable by evaluation participants and whether you have identified the potential burdens and risks of the evaluation. [8]

Questions nine and ten support the consideration of implications of the evaluation plan within larger and long-term contexts. For example, considering the social justice implications, whether the evaluation plan reinforces existing inequities and considering the long-term consequences related to the impact of the evaluation. [2] These impacts could include closure of a program, loss of trust in the community or stigmatization of specific populations and/or communities. [8]

Implementation Phase: This phase involves participant or community recruitment, obtaining consent as well as collecting or accessing data, and processing and analyzing the results. [7,8] When recruiting participants, factors such as how, when and where participants are approached can affect the voluntariness of participation. For example, the issue of undue influence may arise when individuals are recruited by a person who is in a position of authority. In some instances, community approval may need to be sought before individuals can be approached. In terms of the consent process, issues that can compromise consent such as undue influence, temporary, permanent, diminished or developmental incapacity and language and cultural barriers need to be addressed and in some circumstances, accommodated. [2] During this phase, evaluators also have the ethical duty to maintain confidentiality, respect privacy and ensure the security of information collected and used as a result of the evaluation. [4] Consideration needs to be given to the privacy and confidentiality risks associated with the collection, use and disclosure of information throughout the lifecycle of an evaluation. [4,7,8] For example, privacy and confidentiality risks depend on two interrelated factors: (1) the identifiability of the information and (2) the degree and potential for harm; [2] if the data used for the evaluation are identifiable and sensitive in nature (e.g., survey responses about risky behaviors), then these two factors can increase the risk of the project. Finally, mitigation strategies should be in place during data collection to ensure the welfare of participants, e.g., if collecting sensitive information about an individual (e.g., depression scales, illegal behaviors, intimate partner violence, sexual practices) evaluators should have resources and protocols in place to deal with unanticipated events (e.g., suicidal ideation, requests for help).

Utilization Phase: The final phase of the evaluation involves the dissemination and uptake of the evaluation findings. [7,8] Potential risks associated with this phase include publically presenting findings that may be stigmatizing or negatively impact the reputation of a community, omitting or downplaying negative findings and misuse or misrepresentation of findings. [2] Strategies to mitigate these risks include ensuring that the initial analytical plan is followed, proposed benefits are respected, and that stakeholders or communities are engaged throughout the utilization phase to ensure validity of findings and appropriate use of the evaluation findings in future decision making. [8]  Engagement could also help address potential harms associated with dissemination.

V Conclusion

Including ethics principles and strategies into public health evaluations will improve evaluation quality and rigor, increase the credibility of the evaluation and the evaluator, and engender community and public trust in the evaluation. Through the use of the TCPS 2, [5]  the ten questions in PHO’s Framework for the Ethical Conduct of Public Health Initiatives, [2] and consulting PHO’s  ten steps for conducting an evaluation, [7] evaluators can have increased confidence in their ethical conduct in all three phases of an evaluation.
 

References

  1. Brownson RC, Fielding JE, Maylahn, CM. Evidence-based public health: A fundamental concept for public health practice. Annu. Rev. Public Health. 2009:30;175-201. Available from: www.astho.org/Programs/Evidence-Based-Public-Health/Evidence-Based-Publi....
  2. Ontario Agency for Health Protection and Promotion. A framework for the ethical conduct of public health initiatives. 2012. Available from www.publichealthontario.ca/en/eRepository/PHO%20%20Framework%20for%20Eth...
  3. Willison DJ, Ondrusek N, Dawson A, Emerson C, Ferris LE, Saginur R, et al. What makes public health studies ethical? Dissolving the boundary between research and practice. BMC Medical Ethics. 2014;15(1) . Available from: http://bmcmedethics.biomedcentral.com/articles/10.1186/1472-6939-15-61.
  4. Canadian Evaluation Society. CES competencies for Canadian evaluation practice [document on internet]. 2010 [cited 2016 Sept 13]. Available from: http://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice.pdf.
  5. Canadian Institutes of Health Research, Natural Sciences and Engineering Council of Canada, and Social Sciences and Humanities research Council of Canada. Tri-Council policy statement: ethical conduct for research involving humans; 2014. Available from: http://www.pre.ethics.gc.ca/pdf/eng/tcps2-2014/TCPS_2_FINAL_Web.pdf.
  6. Government of Canada. TCPS 2 tutorial course on research ethics. 2016. Available from: http://pre.ethics.gc.ca/eng/education/tutorial-didacticiel/.
  7. Ontario Agency for Health Protection and Promotion (Public Health Ontario). At a glance: The ten steps for conducting an evaluation. Toronto, ON: Queen’s Printer for Ontario; 2015. Available from: http://www.publichealthontario.ca/en/eRepository/At_A_Glance_Evaluation_....
  8. Ontario Agency for Health Protection and Promotion (Public Health Ontario). Evaluating health promotion programs: introductory workbook. Toronto, ON: Queen's Printer for Ontario; 2016. Available from: https://www.publichealthontario.ca/en/erepository/Evaluating_health_prom...