ReAct aims to develop a new impact assessment framework for the humanities and social sciences based on a reflexive, participatory and inclusive notion of societal interactions and benefits arising form the research process. In several instances, impact assessment exercises are controversial. There is criticism of well-known assessment methods for being unable to capture the richness and dynamic nature of research engagement, interactions and networks. Unintended impact is rarely measured or acknowledged yet important for the understanding of the impact ecosystem.
In many instances, no attention is paid to the relation between the research practice and the tangible impact emanating from it. Instead, there is an overwhelming focus in the academic literature and policy on measurable artifacts, such as number of patents, licenses, royalties or social media uptake. The use of the term impact echoes a long tradition of evaluating the (‘positive’) outcomes of science and technology solely in quantitative and mechanical terms.
ReAct takes an alternative approach, which is characterized by re-assessing the research practice as embedded in a social context and directed towards a number of societal ends and towards solving societal challenges. This leads to a call for a responsible impact assessment framework, which cannot be justified on purely quantitative terms but combines qualitative judgment and participatory contributions with annotated data.
Manifestos that guide the responsible use of impact assessment tools are not new. In 2015, a group of authors published The Leiden Manifesto, in which ten principles for the proper use of scientometric indicators in research were presented. The Leiden Manifesto considers scientometric indicators in general, while the three principles presented below focus specifically on broader impact assessment. The principles are not exhaustive. They are based on the intuition that impact assessment in research needs to be driven by “trust in people not in numbers”. First and foremost, impact of research is a living story with various aspects, perspectives, narratives and limits – all of which deserve to be discussed.
1. Need for a reflexive approach to impact (Principle of Reflexivity)
Impact is an essentially contested concept. Any definition of impact depends on cognitive and societal interests, temporality, locality – and normativity. There exists no one-size-fits-all method or field-independent model that can be used to identify which type of research that makes a large contribution to society. Instead, impact assessment should support good decision-making, clear planning and realistic methodologies. It cannot be pursued without explicit consideration of the motives and objectives of the research in questions. In the absence of such consideration, impact assessment is likely to lead to poor data, missed opportunities and misaligned or underachieved goals. Criticisms of impact assessment methods are well-known. There is an inherent risk that “when a measure becomes a target, it ceases to be a good measure”. Within research institutions there is a sense in which activities that are measured will be the only activities that are optimized and promoted. Hence, without reflexivity even the simplest measure might introduce shortsightedness and narrow-mindedness. In contrast, a framework for Responsible Impact introduces an explicit value-dimension. It encourages an impact culture in which researchers can learn from their own engagement and dissemination activities, and use them for research purposes, rather than forming the basis of evaluation and ranking. Impact is a multifaceted concept, and the best way to avoid misuse of impact indicators is by allowing researchers to have significant influence on how their impact is represented and communicated.
2. Need for a participatory model of impact (Principle of Participation)
Academic research translates into societal impact in numerous non-linear and interconnected ways, which are rarely amenable to precise, quantitative metrics. Any reflexive attempt to create new insights into the impact ecosystem needs to start by developing proxy categories of connectivity between researchers and research-users. Such categories may form important steps towards impacts, i.e. the societal change or benefit arising from the activities. Understanding these connections can lead to a deeper appreciation of the factors that shape the processes leading to research uptake in society (culture, economy, policy, media, etc.). Hence, any attempt to build metrics or indicators, should be preceded by a detailed approach to identify the flows of knowledge, expertise and influence that take place during the process of knowledge creation. Developing impact taxonomies and designing appropriate field-level indicators must take into account the nature and changing structure of scholarship, analyzing possible pathways and impact classifications by engaging researchers in the validation of categories. A framework for Responsible Impact encourages active participation at all levels of impact assessment – from developing comprehensive taxonomies to formulating evidence-based impact case studies. Feedback from researchers may include (but is not limited to) insight and deliberations about the proper categories, roles and types of impact they are practicing. From outside academia, feedback should be invited from users and partners to ensure that impact is described in a manner consistent with concerns and interests of stakeholders.
3. Need for an inclusive concept of societal impact (Principle of Diversity)
Impact can occur at any state in the research lifecycle. Opportunities for knowledge exchange can arise at different stages of the research process. Even preliminary achievements, work-in-progress, or seemingly insignificant relations and interactions can translate into direct or indirect societal benefits e.g. by contributing to capacity building, knowledge uptake, learning or problem-solving among stakeholders and audiences. For example, impacts can be generated already at the outset of a research project by involving stakeholders to inform relevant research questions and research design; they can occur during the research process, for example linked to the diffusion of research methodologies and tools, or they can be associated with the communication and dissemination of research findings. Impact is constantly evolving in line with the absorptive capacity among stakeholders and can derive from a wide range of research outputs. For these reasons, a framework for Responsible Impact builds on an inclusive notion of impact, including policy impact, cultural impact, economic impact, technological impact, health impact, impact on teaching and training, social impact, public impact (including impact on public services and public discourse) leading to a multi-dimensional and multi-scalable notion of the broader value of research in society.
Copenhagen May 31, 2017