Equitable Practices in Peer Review

The University of Toronto (U of T) is committed to fostering a diverse, equitable, and inclusive intellectual community that allows all researchers to reach their potential for excellence. Drawing on promising practices cited in the literature, the Division of the Vice-President, Research & Innovation (VPRI) has developed this resource to support internal peer review committees in carrying out equitable and transparent adjudication processes. Committee chairs, equity officers, and committee members are encouraged to complete this checklist as they move through the peer review and selection process.

• The committee is diverse and engages members of under-represented groups, including women, Black and racialized persons, persons with disabilities, Indigenous Peoples, and members of 2SLGBTQ+ communities.
• The recruitment of diverse committee members respected faculty members’ privacy, avoided suggestions of tokenism, and was mindful of the disproportionate service burdens often faced by faculty members from under-represented groups.
• Committee invitations emphasized that all committee members are responsible for commitment to the principles of equity, diversity, and inclusion (EDI).
• The committee includes an equity officer or designated equity champion who attends deliberation meetings and supports the committee in implementing equitable and consistent practices. The equity officer can be a faculty or staff member with demonstrated knowledge of EDI, particularly equitable practices related to peer review.

• All committee members have, at minimum, completed the online Unconscious Bias Education Modules (Modules 1–3) developed by the Toronto Initiative for Diversity & Excellence (TIDE). Committee members can also make use of additional resources available from TIDE and external sources, such as the Tri-Agency/Canada Research Chairs unconscious bias training module.
• The chair and equity officer have communicated that the purpose of such training is not merely to create awareness of unconscious bias, but rather to aid reviewers in taking steps to mitigate the impact of bias on the peer review process. Committee members are encouraged to reflect on and apply these learnings throughout the review process.

Peer review committee members took the following steps when considering and applying the evaluation criteria.

• Determined and prioritized evaluation and selection criteria before opening any applications. (To promote transparency, the initial call for applications included a list of evaluation criteria, as well as any secondary criteria used to prioritize the distribution of funding or awards.)
• Developed clear and inclusive definitions of excellence/quality for each criterion.
• As applicable, determined the short-listing process in advance of reviewing any applications.
• Guarded against reliance on holistic or intangible qualities like “fit” or perceived personal attributes, and evaluated applicants only based on qualifications explicitly listed in the application guidelines and selection criteria.

When reviewing and assessing individual applications, peer review committee members took the following steps.

• Reviewed all of the required materials submitted by each qualified applicant but did not consider any extraneous information (such as personal knowledge of the applicant).
• Spent sufficient time reviewing each qualified applicant—implicit biases tend to be most pronounced when we are rushed or distracted. Committee members devoted approximately the same amount of time to each eligible application.
• Recorded assessments of applicants using an evaluation criteria grid or rubric, consistently applying the above-mentioned criteria of excellence to all applicants.
• Avoided over-reliance on one piece of information or on first impressions.
• In cases where committee members could not review all applications, each member reviewed the same agreed-upon number of applications, and all applications were reviewed by at least two committee members.
• Considered career interruptions for family leave, medical needs, or other reasons, as well as special circumstances that involve slow-downs in research productivity, taking into account the effects of these interruptions on the applicant’s research record (e.g., the applicant with the highest number of publications is not automatically the “best” candidate).
• Fairly considered the challenges that may come with interdisciplinary, partnered, or collaborative research, including potential effects on research timelines. Committee members were aware that some emerging sub-disciplines or fields of study may present fewer opportunities to publish in traditional top-tier venues or to attract significant research funding
• As applicable to the applications received, committee members demonstrated familiarity with and knowledge of Indigenous ways of knowing, Indigenous research methodologies, and/or community engaged research. (Consult SSHRC’s Guidelines for the Merit Review of Indigenous Research for more information)

• The committee chair included in their opening remarks a statement emphasizing the University’s/department’s commitment to the principles of equity, diversity, and excellence.
• The committee chair or the equity officer confirmed that all members have completed unconscious bias training and that members do not have any conflicts of interest that would prevent them from assessing candidates in an objective manner.
• The equity officer explained their role in the review process and provided a brief overview of the ways in which unconscious bias can affect peer review.
• The committee chair ensured that all members have the opportunity to contribute to conversations and have their opinions considered.
• With the support of the equity officer, the committee conducted periodic check-ins during the review process to ensure that criteria are applied consistently to all applicants and the committee is taking the appropriate steps to minimize the impact of unconscious bias.
• The chair or equity officer encouraged all committee members to raise any concerns related to equitable processes during the meeting or in conversation with the chair or the equity officer.

• Develop resources and tools to guide equitable and transparent peer review practices.
• Serve on review committees when needed.
• Advise committee chairs on implementing equitable practices throughout the review process, e.g., the recruitment of a diverse committee, consideration of career interruptions.
• Recommend inclusive definitions of excellence and evaluation criteria for funding and awards.