The quality and reliability of impact indicators will vary according to the impact we are trying to describe and link to research. A key concern here is that we could find that universities which can afford to employ either consultants or impact administrators will generate the best case studies. The current definition of health, formulated by the WHO, is no longer adequate for dealing with the new challenges in health care systems. % Assessment for Learning is the process of seeking and interpreting evidence for use by learners and their teachers to decide where the learners are in their learning, where they need to go and. The University and College Union (University and College Union 2011) organized a petition calling on the UK funding councils to withdraw the inclusion of impact assessment from the REF proposals once plans for the new assessment of university research were released. Overview of the types of information that systems need to capture and link. What are the reasons behind trying to understand and evaluate research impact? Key features of the adapted criteria . Inform funding. n.d.). The term comes from the French word 'valuer', meaning "to find the value of". This database of evidence needs to establish both where impact can be directly attributed to a piece of research as well as various contributions to impact made during the pathway. Capturing data, interactions, and indicators as they emerge increases the chance of capturing all relevant information and tools to enable researchers to capture much of this would be valuable. Test, measurement, and evaluation are concepts used in education to explain how the progress of learning and the final learning outcomes of students are assessed. The Author 2013. 2007). Systems need to be able to capture links between and evidence of the full pathway from research to impact, including knowledge exchange, outputs, outcomes, and interim impacts, to allow the route to impact to be traced. The first attempt globally to comprehensively capture the socio-economic impact of research across all disciplines was undertaken for the Australian Research Quality Framework (RQF), using a case study approach. It is perhaps worth noting that the expert panels, who assessed the pilot exercise for the REF, commented that the evidence provided by research institutes to demonstrate impact were a unique collection. While aspects of impact can be adequately interpreted using metrics, narratives, and other evidence, the mixed-method case study approach is an excellent means of pulling all available information, data, and evidence together, allowing a comprehensive summary of the impact within context. 2009). This framework is intended to be used as a learning tool to develop a better understanding of how research interactions lead to social impact rather than as an assessment tool for judging, showcasing, or even linking impact to a specific piece of research. Recommendations from the REF pilot were that the panel should be able to extend the time frame where appropriate; this, however, poses difficult decisions when submitting a case study to the REF as to what the view of the panel will be and whether if deemed inappropriate this will render the case study unclassified. Assessment for learning is ongoing, and requires deep involvement on the part of the learner in clarifying outcomes, monitoring on-going learning, collecting evidence and presenting evidence of learning to others.. Frameworks for assessing impact have been designed and are employed at an organizational level addressing the specific requirements of the organization and stakeholders. (2008), and Hanney and Gonzlez-Block (2011). While valuing and supporting knowledge exchange is important, SIAMPI perhaps takes this a step further in enabling these exchange events to be captured and analysed. Assessment is the process of gathering and discussing information from multiple and diverse sources in order to develop a deep understanding of what students know, understand, and can do with their knowledge as a result of their educational experiences; the process culminates when assessment results are used to improve subsequent learning. Media coverage is a useful means of disseminating our research and ideas and may be considered alongside other evidence as contributing to or an indicator of impact. While defining the terminology used to understand impact and indicators will enable comparable data to be stored and shared between organizations, we would recommend that any categorization of impacts be flexible such that impacts arising from non-standard routes can be placed. Evidence of academic impact may be derived through various bibliometric methods, one example of which is the H index, which has incorporated factors such as the number of publications and citations. This atmosphere of excitement, arising from imaginative consideration transforms knowledge.. Any person who has made a significant . The growing trend for accountability within the university system is not limited to research and is mirrored in assessments of teaching quality, which now feed into evaluation of universities to ensure fee-paying students satisfaction. Any tool for impact evaluation needs to be flexible, such that it enables access to impact data for a variety of purposes (Scoble et al. Productive interactions, which can perhaps be viewed as instances of knowledge exchange, are widely valued and supported internationally as mechanisms for enabling impact and are often supported financially for example by Canadas Social Sciences and Humanities Research Council, which aims to support knowledge exchange (financially) with a view to enabling long-term impact. For more extensive reviews of the Payback Framework, see Davies et al. The Payback Framework has been adopted internationally, largely within the health sector, by organizations such as the Canadian Institute of Health Research, the Dutch Public Health Authority, the Australian National Health and Medical Research Council, and the Welfare Bureau in Hong Kong (Bernstein et al. Classroom Assessment -- (sometime referred to as Course-based Assessment) - is a process of gathering data on student learning during the educational experience, designed to help the instructor determine which concepts or skills the students are not learning well, so that steps may be taken to improve the students' learning while the course is 2007). What are the methodologies and frameworks that have been employed globally to assess research impact and how do these compare? 0000346296 00000 n << /Length 5 0 R /Filter /FlateDecode >> In the UK, evaluation of academic and broader socio-economic impact takes place separately. RAND selected four frameworks to represent the international arena (Grant et al. Indicators, evidence, and impact within systems, Department for Business, Innovation and Skills 2012, http://www.arc.gov.au/pdf/ERA_Indicator_Principles.pdf, http://www.charitystar.org/wp-content/uploads/2011/05/Return_on_donations_a_white_paper_on_charity_impact_measurement.pdf, http://www.oecd.org/science/innovationinsciencetechnologyandindustry/37450246.pdf, http://www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf, http://mice.cerch.kcl.ac.uk/wp-uploads/2011/07/MICE_report_Goldsmiths_final.pdf, http://www.timeshighereducation.co.uk/story.asp?storyCode=409614§ioncode=26, http://www.odi.org.uk/rapid/Events/ESRC/docs/background_paper.pdf, http://www.iscintelligence.com/archivos_subidos/usfacultyburden_5.pdf, http://blogs.lse.ac.uk/impactofsocialsciences/tag/claire-donovan/, http://www.atn.edu.au/docs/Research%20Global%20-%20Measuring%20the%20impact%20of%20research.pdf, http://www.hbs.edu/research/pdf/10-099.pdf, http://www.esf.org/index.php?eID=tx_ccdamdl_file&p[file]=25668&p[dl]=1&p[pid]=6767&p[site]=European%20Science%20Foundation&p[t]=1351858982&hash=93e987c5832f10aeee3911bac23b4e0f&l=en, http://www.rand.org/pubs/research_briefs/2007/RAND_RB9202.pdf, http://www.rand.org/pubs/documented_briefings/2010/RAND_DB578.pdf, http://ukirc.ac.uk/object/report/8025/doc/CIHE_0612ImpactReport_summary.pdf, http://www.timeshighereducation.co.uk/story.asp?storyCode=415340§ioncode=26, http://www.publicengagement.ac.uk/sites/default/files/80096%20NCCPE%20Social%20Value%20Report.pdf, http://www2.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.pdf, http://www.artscouncil.org.uk/media/uploads/documents/publications/340.pdf, http://www.ref.ac.uk/media/ref/content/pub/researchexcellenceframeworkimpactpilotexercisefindingsoftheexpertpanels/re01_10.pdf, http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/02_11.pdf, http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf, http://www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01_12.pdf, http://www.russellgroup.ac.uk/uploads/REF-consultation-response-FINAL-Dec09.pdf, http://www.siampi.eu/Pages/SIA/12/625.bGFuZz1FTkc.html, http://www.siampi.eu/Content/SIAMPI/SIAMPI_Final%20report.pdf, http://www.thesroinetwork.org/publications/doc_details/241-a-guide-to-social-return-on-investment-2012, http://www.ucu.org.uk/media/pdf/n/q/ucu_REFstatement_finalsignatures.pdf, http://www.esrc.ac.uk/_images/Case_Study_of_the_Future_of_Work_Programme_Volume_2_tcm8-4563.pdf, Receive exclusive offers and updates from Oxford Academic, Automated collation of evidence is difficult, Allows evidence to be contextualized and a story told, Incorporating perspective can make it difficult to assess critically, Enables assessment in the absence of quantitative data, Preserves distinctive account or disciplinary perspective, Rewards those who can write well, and/or afford to pay for external input. One way in which change of opinion and user perceptions can be evidenced is by gathering of stakeholder and user testimonies or undertaking surveys. This work was supported by Jisc [DIINN10]. , . This petition was signed by 17,570 academics (52,409 academics were returned to the 2008 Research Assessment Exercise), including Nobel laureates and Fellows of the Royal Society (University and College Union 2011). According to Hanna- " The process of gathering and interpreted evidence changes in the behavior of all students as they progress through school is called evaluation". (2006) on the impact arising from health research. working paper). The first category includes approaches that promote invalid or incomplete findings (referred to as pseudoevaluations), while the other three include approaches that agree, more or less, with the definition (i.e., Questions and/or Methods- This presents particular difficulties in research disciplines conducting basic research, such as pure mathematics, where the impact of research is unlikely to be foreseen. What emerged on testing the MICE taxonomy (Cooke and Nadim 2011), by mapping impacts from case studies, was that detailed categorization of impact was found to be too prescriptive. 2007; Nason et al. Impact has become the term of choice in the UK for research influence beyond academia. For example, following the discovery of a new potential drug, preclinical work is required, followed by Phase 1, 2, and 3 trials, and then regulatory approval is granted before the drug is used to deliver potential health benefits. Donovan (2011) asserts that there should be no disincentive for conducting basic research. There are standardized tests involved in the process of measurement assessment and evaluation to enable the students to make better use of the data available in the daily classroom.