future research assessment programme

Evaluating research assessment

In working towards designing a new assessment framework for research, the funding bodies commissioned a programme of evaluation of REF 2021.

These evaluation activities were initiated in 2022, with the majority reporting in late 2022, and the remainder reporting in 2023. Further evaluations of and reflections on REF 2021 can be found on the REF 2021 website , including the reports of the expert panels.

Impact case study analysis

The funding bodies and UKRI have commissioned a project (delivered by RAND Europe) to carry out qualitative and quantitative analyses of the impact case studies (almost 7,000) submitted to REF 2021. The aim is to extract common themes and messages that will form evidence of the broad impact of higher education research on wider society.

• Download the report: Analysis of the REF 2021 impact case studies (pdf)
• Find out more about the impact of higher education research in Scotland (new reports will be published on impact case studies and funding by RAND and the Fraser of Allander Institute)
• Download the report: The impacts of research from Welsh universities (pdf)

Review of REF costs and benefits

The funding bodies have commissioned a project to understand how REF 2021 costs have been contained or grown compared to previous exercises, including understanding how the balance of costs against benefits played out in REF 2021 and the effect of changes introduced in the current exercise.

The project was delivered by Technopolis Group:

• REF 2021 Cost evaluation report (pdf)

Analysis of REF outputs 

A project exploring ways in which data generated by the REF may be utilised to gain insights into the health of disciplines or research areas at sector level. The project included four areas of experimental analyses of REF 2021 data: analysis of medical sub-disciplines; analysis of ageing and gerontology research; exploring research excellence within the arts and humanities; and understanding interdisciplinary research. Through this project, the funding bodies have also gained insights into the feasibility of carrying out a more granular analysis of research assessment data, or analysis of cross-cutting themes in future exercises.

This project was delivered by Technopolis Group:

• Download the Analysis of REF Outputs report (pdf) .
• Download the Analysis of REF Outputs appendices (pdf) .

Responsible use of technology in research assessment

A project  exploring the feasibility of using artificial intelligence and machine learning approaches to streamline research assessment, from the allocation of outputs and analysis of outcomes to assessment itself..

The project was delivered by the Statistical Cybermetrics and Research Evaluation Group at the University of Wolverhampton.

Download the Responsible Use of Technology in Research Assessment report (pdf) .

Reviewing the role of metrics in research assessment

The funding bodies invited an expert panel to lead a review of the role of metrics in research management and assessment. This review took a short, sharp, evidence-informed look at current and potential uses of metrics against a set of tightly-defined objectives:

• To revisit the conclusions and recommendations of the last detailed review of these questions—The Metric Tide (2015)—and assess progress against these
• To consider whether developments over recent years in the infrastructures, methodologies and uses of research metrics negate or change any of those 2015 conclusions, or suggest additional priorities
• To look afresh at the role of metrics in any future Research Excellence Framework (REF), and consider whether design changes now under consideration as part of the FRAP suggest similar or different conclusions to those reached in 2015
• To offer updated advice to research funders on the most effective ways of supporting and incentivising responsible research assessment and responsible uses of metrics

Find out more about the review and expert panel .

Download the ‘ Harnessing the Metric Tide: indicators, infrastructures and priorities for responsible research assessment in UK ’ report.

Understanding perceptions of the REF

A large-scale review, commissioned by Research England on behalf of the four higher education funding bodies and published by RAND Europe, collected attitudes to the REF in real time as UK institutions prepared their submissions. It gathered views via a survey (with 3,000+ researcher responses), as well as focus groups and one-to-one interviews with researchers, research managers, and institutional leads.

The review also considered the impact of changes made to the REF since the previous exercise in 2014. 

Download the real-time REF review report (pdf) .

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• CAREER COLUMN
• 31 July 2023

UK research assessment is being reformed — but the changes miss the mark

• Richard Watermeyer 0 ,
• Gemma Derrick 1 &
• Kate Sang 2

Richard Watermeyer is professor of higher education and co-director of the Centre for Higher Education Transformations at the University of Bristol, UK.

You can also search for this author in PubMed   Google Scholar

Gemma Derrick is associate professor of higher education at the University of Bristol, UK.

Kate Sang is a gender- and employment-studies researcher and director of the Centre for Research on Employment, Work and the Professions (CREWS) at Heriot-Watt University, Edinburgh, UK.

On 15 June, updated rules were proposed for the next round of the Research Excellence Framework (REF), the assessment system used to distribute around £2 billion (US$2.5 billion) of annual funding across UK universities. These were unveiled by the United Kingdom’s four higher-education funding agencies.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-023-02469-w

This is an article from the Nature Careers Community, a place for Nature readers to share their professional experiences and advice . Guest posts are encouraged.

Competing Interests

The authors declare no competing interests.

Related Articles

• Institutions
• Research management

Massive Attack’s science-led drive to lower music’s carbon footprint

Career Feature 04 SEP 24

Tales of a migratory marine biologist

Career Feature 28 AUG 24

Nail your tech-industry interviews with these six techniques

Career Column 28 AUG 24

Can South Korea regain its edge in innovation?

Nature Index 21 AUG 24

What will it take to open South Korean research to the world?

How South Korea can support female research leaders

Binning out-of-date chemicals? Somebody think about the carbon!

Correspondence 27 AUG 24

No more hunting for replication studies: crowdsourced database makes them easy to find

Nature Index 27 AUG 24

Partners in drug discovery: how to collaborate with non-governmental organizations

Career Feature 23 AUG 24

Postdoc/PhD opportunity – Pharmacology of Opioids

Join us at MedUni Vienna to explore the pharmacology of circular and stapled peptide therapeutics targetting the κ-opioid receptor in the periphery.

Vienna (AT)

Medical University of Vienna

Division Director - Experimental Hematology and Cancer Biology

Cincinnati Children’s Hospital seeks the next Director for the Division of Experimental Hematology and Cancer Biology.

Cincinnati, Ohio

Cincinnati Children's Hospital & Medical Center

Faculty and Research Positions, Postdoctoral Recruitment

Jointly sponsored by the Hangzhou Municipal People's Government and the University of Chinese Academy of Sciences.

Hangzhou, Zhejiang, China

Hangzhou Institute of Advanced Study, UCAS

Associate or Senior Editor, Nature Energy

Job Title: Associate or Senior Editor, Nature Energy Location: New York, Jersey City, Philadelphia or London — Hybrid Working Application Deadline:...

New York City, New York (US)

Springer Nature Ltd

Faculty Positions & Postdocs at Institute of Physics (IOP), Chinese Academy of Sciences

IOP is the leading research institute in China in condensed matter physics and related fields. Through the steadfast efforts of generations of scie...

Beijing, China

Institute of Physics (IOP), Chinese Academy of Sciences (CAS)

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

People, culture and environment in REF 2029

The funding bodies are inviting written comments on the assessment of people, culture and environment in REF 2029. This coincides with the launch of a tender for work to develop the indicators to be used in this assessment.

As part of the initial decisions on REF 2029, the funding bodies announced their intention to increase focus on the assessment of the conditions that are an essential feature of research excellence.

To ensure the robustness of this assessment the funding bodies seek to develop a tightly structured statement which draws on outcomes-focused indicators. While the assessment of people, culture and environment indicators is not covered in the formal consultation, informal discussions with the sector have identified the need for clearly defined indicators that can be consistently and fairly applied, and contextualised within institutional strategies and priorities.

The funding bodies are seeking, via the Crown Commercial Framework , to engage a commercial partner to develop and test methodological approaches for assessing research culture and environment in REF 2029. With co-production at its heart, this substantial piece of commissioned work will develop the people, culture and environment questionnaire and indicators in close collaboration with the sector.

The project will develop, test and iterate a set of quantitative and qualitative indicators of research culture, engaging widely with the sector, and providing advice to the panels during the criteria-setting phase of REF 2029. A copy of the procurement specification (pdf) is available for information.

Ahead of the project launch, and to inform its development, the funding bodies will be convening roundtables and open web events through the autumn. In parallel, they are inviting written comments on the challenges and opportunities of their proposed approach.

Respondents are also invited to outline any concerns they may have about this element of REF 2029 and suggest solutions to challenges and concerns identified. Insights from these responses will be used to inform the development of this element of REF 2029.

Responses are invited from any organisation, group or individual with an interest in the conduct, quality, funding or use of research in the UK. Respondents are requested to complete the feedback form (.docx) providing comments in no more than two pages.

Completed documents should be sent to [email protected] by 5pm on 1 December 2023.

• icon-search
• icon-comments 4
• About Wonkhe
• Our subscriptions
• Our partners
• Pitch an article
• Explore the whole archive
• Wonkhe research
• Wonk Corner
• Jobs posting & prices
• Wonkhe Pass
• Monday Briefing
• Daily Briefing
• Policy Update
• ADD NEW SUs USER
• COST OF LIVING HUB
• BRIEFINGS INDEX
• WEBINAR INDEX
• LATEST SUs BLOGS
• FREE SPEECH
• SU STRATEGIES

REF 2028 is coming

• Click to print (Opens in new window)
• Click to share on Twitter (Opens in new window)
• Click to share on Facebook (Opens in new window)
• Click to share on Pocket (Opens in new window)
• Click to share on LinkedIn (Opens in new window)
• Click to email a link to a friend (Opens in new window)
• Click to share on WhatsApp (Opens in new window)
• Click to share on Reddit (Opens in new window)

Author 1 ? 's' : '')?>

David kernohan, deputy editor.

David Kernohan is Deputy Editor of Wonkhe

If you’re a researcher your big question is whether or not you will be submitted to the next research excellence framework (REF).

And the answer, as of today, is no.

The awful idea of being “REFable” or not will have no salience next time round. There’ll be no requirement that any of your work is submitted – but the fact you exist and have research responsibilities counts towards the volume measure that determines how many outputs and case studies that can be submitted.

And of course, you’ll suspect that the quality of those outputs – as judged by your peers – are the single most important thing that determines your departmental REF score. And you’d be wrong – outputs will count for just 50 per cent of results, with measures of impact and the research environment taking up the remainder.

The story begins

Today sees the publication of the Future Research Assessment Programme (FRAP) initial decision report coded REF2028/01, a provisional equality impact assessment, and the international advisory report that has informed the development of both.

Assessment panels will be appointed at the back end of this calendar year, with the final guidance and criteria following in the winter of 2024 (following a consultation over the summer). You’ll be submitting to REF 2028 during 2027, with the all-important outcomes published in December 2028.

As a means of reducing burden many of the arrangements for REF 2021 will remain – feedback has suggested that adapting to new requirements is the principal driver of burden. So we’ll see the same panels, and some of the same definitions – and the REF rules for open access will duplicate the UKRI rules (though there is a consultation to follow on this).

But don’t let that make you think that REF 2028 is business as usual. A lot has changed.

Elemental forces

REF2028 will have, as REF2021 did, three elements, or profiles – though names and (notably) weighting are substantially different.

People, culture, and environment is worth 25 per cent of the overall REF score (up from 15 per cent for environment last time). This will be assessed on questionnaire responses collected at both institutional and disciplinary levels, supported by a “basket of indicators” (work on these will be commissioned later this year) drawing as far as possible on existing data. The new questionnaire template will be a lot more structured than previous iterations – there were concerns in REF 2021 that submissions here were not consistent, so this is an attempt to address this issue.

Engagement and impact , building on the REF 2021 impact profile, remains at 25 per cent of the overall score. The impact case studies – with the number required varying from one upwards based on the size of the unit of assessment in each provider – will be supported by an explanatory statement worth a minimum of 20 per cent of this profile, or at least one case study. The requirement that case studies need to be linked to research of at least 2* quality will be removed.

Contribution to knowledge and understanding will account for the final 50 per cent of the overall score. And there are big changes.

As indicated above, staff will no longer be submitted to the REF. Instead, the average full-time equivalent staff count, based on 2025-26 and 2026-27 HESA Staff returns will be used to determine the number of research outputs that can be submitted to each unit of assessment by each provider – 2.5 outputs per staff member contributing to a UoA.

This doesn’t mean that every staff member needs to have an output returned – indeed an institution can choose to return outputs from other staff members, including staff who do not have a specific responsibility for research and indeed anyone that can demonstrate a substantive link to the institution (defined as 0.2 FTE for at least 6 months).

There are no maximum (or minimum) number of outputs returnable for each member of staff. But an explanatory narrative statement (worth at least 10 per cent of the total marks for this element) will be needed to explain why the selected outputs are representative of that subject at that institution, and will include an evidence based statement about the institution’s wider contribution to knowledge in this area.

Research England has been very careful to emphasise that decisions on funding outcomes (currently based, in part, on volume) are separate to this process – and we should note a review of strategic institutional research funding is already underway.

Yes, but does it correlate?

As an illustration we’ve run the REF 2021 data using these weightings to calculate revised “overall” proportion (note that this is indicative only, doesn’t include the new approaches to assessment under each profile, and is limited by the resolution of the data available).

[ Full screen ]

The story here is not one of vast changes and recalibrating the REF entirely. It makes sense that excellent outputs have excellent impact and come from an excellent environment – so in the main this does not radically alter what we saw using the other weightings. There is an apparent correlation between excellent outputs, and excellent research environments, and excellent evidence of impact. And – notably – a very significant correlation between overall results calculated using the old and new weightings.

I built a tool to let you explore these relationships – you can set each axis on the chart to show the proportion of 4* activity in each UoA for outputs, impact, environment, the 2021 weighted overall assessment, and the 2028 weighted overall assessment (using the 2021 data, so not including the revised assessment plans).

We appear to have hit a sweet spot between shifting weightings enough to drive institutional behavioural changes without fundamentally challenging the integrity of results. If the marginal gains energy of institutions becomes, as a result, directed less on the wilder shores of performance management and more on a supportive research environment we will all benefit.

Direction of travel

The International Advisory Group report, also published today, has clearly been (as it was commissioned to be) a huge influence on initial plans for REF 2028. This is a group charged with keeping an eye on international trends in the assessment and funding of research – and one of those key shifts has been away from the assessment of outputs to a systemic consideration of the “health” of research.

It’s from here we get the shift towards the institution and away from the individual, the recognition of the research work conducted by staff outside of the pool of traditional research active academics, and – notably – the reduction of weight placed on the assessment of outputs. Indeed, the IAG recommendation was that outputs should form only a third of the overall judgement – the reduction to a half (including the outputs statement) feels like a staging post towards this aim.

That the research sector is not sustainable, and is largely not livable, will not be news to anyone who works in it – and numerous UK government reports (notably Paul Nurse on the research landscape , Adam Tickell on research bureaucracy , and indeed the government itself in the people and culture strategy ) have made this point. But this is an international issue – not one specific to the UK – and recent international initiatives (DORA, Leiden, the work of RoRI and Science Europe) exemplify what we may characterise as a de-escalation in research assessment.

The IAG report is unequivocal about the move away from the focus on the individual as a unit of research excellence, and towards REF being a tool to facilitate an integrated understanding of everything a research performing organisation does – research, teaching, and engagement – and the impact this has on staff. We can see the fruits of this in the incorporation of equality, diversity, and inclusion in the metrics and statements that will underpin the assessment of an institution’s (and UOA’s) strength in “people, culture, and the environment”.

Whither “excellence”?

There is very likely to be a push-back against the re-weighting of the REF by more traditionally-minded parts of the sector. The Research Excellence Framework, after all, is just that – so perhaps it should be primarily about the excellence of research that is produced?

What the FRAP project has done in rebalancing the exercises away from outputs is both brave and timely. Despite previous REF guidance actively encouraging the submission of novel forms of output, and the continued assertion that journal impact factors (and the idea of “prestige” in publication more generally) are both completely meaningless and irrelevant to REF assessment, for many the presumption remains that “excellent” research is a well-cited article in a “good” journal.

Research is a process not an end point, and the work of a huge group of staff in many roles rather than a single researcher. An output that can be reviewed by peers is one aspect of this (and it is encouraging that peer review remains central to the REF in a metricised world), the impact of research on the lives of us all is another, and the sustainability of every role that supports the ongoing capacity to conduct research is a third.

We may now have a research assessment system that fully recognises the process rather than just the outcomes. And this is something that anyone involved in research will surely welcome.

festival side Festival side -->

Post list latest articles latest articles -->, ofs’ insight on the risks of franchising fall short at addressing the incentives, international quality standards are evolving, but england is watching from the sidelines, reasons to write for wonkhe this year, labour tweaks the job ad for the next ukri chief executive, why research integrity matters to all of us, students leaders want a framework to help themselves, hesa student data 2022-23, open data (part one), experiential learning is strengthened by moving outside disciplinary silos, how to resist the enshittification of higher education, is angst over the quality of international pathway programmes justified, 4 responses to “ ref 2028 is coming ”.

“the REF rules for open access will duplicate the UKRI rules” – are you sure about that? I’ve just been told this isn’t necessarily the case hence the consultation this autumn. Or is the consultation more about implementation/fine detail?

Hi Phil – the intention is to duplicate the rules. Of course, nothing is ever as simple as that, so the consultation will finesse the detail.

Somewhat idealised if there is a belief that the present management of HE is able to accommodate this implied change and remove the toxicity that exists around managing research. It may well need a strata of overpaid heads of department/schools, pro-VCs etc to be removed for this to happen. Given their tendency towards self-protection, I don’t see this happening and would expect the change to reinforce poor management processes and the poor behaviour of individual managers in HE. Stranger things have happened though.

The new REF rules are a step in the right direction. However the rules around who qualifies to be returned still allows a lot of game playing. Is anyone going to take a closer look at all the academics predominantly based outside the UK (mainly the US) on 20% contracts at some universities? Preparations are already underway for the next REF I believe!! Some of these appointments might be only for their publications. Are these appointments a good way to spend tax payers money that funds UK universities? The 20% contract REF rule has to be strictly monitored to ensure a level playing field. Any such contracts for academics based outside the UK must reflect a deep, meaningful and sustained contractual relationship.

Leave a Reply Cancel reply

Copyright © 2024 Wonkhe Ltd.

Company Number: 08784934

Wonkhe Ltd, 31-35 Kirby Street, London EC1N 8TE

• Moderation policy
• Terms and Conditions
• Privacy Policy

Wonkhe Mondays

Organisation

Job Area professional area Academic Learning and teaching Academic registry and Quality Finance Business Development Policy and public affairs Planning and strategy Marketing, PR and communications International Student Services, recruitment and WP HR and resources Research and doctoral Research manager Library services Estates management Legal and governance Data, IT and technology Students' Unions Unsure / other

Job Level level Entry Level (assistant, administrative) Intermediate Middle Management (e.g. Head of, manager, director) Senior Leadership (e.g. DVC/PVC/Dean) Head of organisation (VC/principal/CEO) Unsure / other

By submitting you agree to our terms and conditions

Some personal insights on the Future Research assessment programme

Sir Peter Gluckman ONZ FRS Chair, FRAP International Advisory Panel IG Government conference | Evaluating REF 2021: Looking Towards The Future Research Assessment Programme May 24 2022

I am afraid I am going to disappoint you, because I am not going to say very much or anything that is very new; and those of you who read David Sweeney’s blog in Research Professional will see a high level of alignment.

Firstly, when the International Advisory panel was established last year, it was important that we did not in any way be seen to influence the just completed round so our discussions have focused on principles rather than the detail. Yet, inevitably academics and managers want to know the detail.

Secondly, we have not met since the REF 2021 results were released, nor have we had time to digest and reflect on them and the associated reports and the consultations that have been underway.

Thus it would be premature for me to say much more than identify some of the challenges which you all know so well.

As a final disclaimer you are hearing my personal comments, not those of the panel which has not met since the results have been published.

It is always important that form follows function and thus we need to start by thinking about the overall concept before the specific processes, which is where so much of the discussion tends to focus.

The original concept of the REF, or at least its predecessors, was fairly clear to provide an accountable and transparent system of providing QR funds to Universities. That remains its core function, and I would suggest individual academics who criticize the REF reflect on how important QR funding is in the context of sustaining the UK’s university research system. It is critical glue money that provides strategic capacity in the system in multiple ways, and it is perhaps more important than ever given the consequences of Brexit to research funding.

From the early policy work, three purposes were clearly assigned to the REF or its predecessors:

• Accountability to the taxpayer
• A means to allocate strategic funds to support research to each higher education institute (HEI)
• Reputational benchmarking – although that is also achieved through other ranking mechanisms

Nick Stern’s committee effectively identified another three purposes:

• To provide evidence base on the research profile for prioritization by UKRI, etc
• To provide performance incentives for universities and researchers, although one might hope institutions and academics have enough other incentives to be self-reflexive and enhance their performance. But the magnitude of the internal effort by institutions suggests otherwise, or at least reflects the growing managerialism that has infected universities worldwide.
• And to assist internal choices made by a University – this is inherent in how QR funds can be used.

The Stern recommendations added greater emphasis that certainly overlaps with the above concept of accountability – that is, to have the institutions demonstrate the value of their work to the taxpayer via demonstrating impact – a matter I shall return to.

But a further inference identified in subsequent commentary has led to a potential additional and explicit purpose, namely culture change within institutions. A start in that direction was met by the focus on interdisciplinarity in the last round and indeed in the growing emphasis on impact. But critically such an objective extends to the environmental assessment and to matters such as EDI and the nature of the research community, diversity, the role of early career researchers, etc.

And there is one other implied but critical aspect: on one hand the outcome must be judged by the Crown to meet the first 3 objectives and the system as a whole must see the entire exercise to be judged as ‘fair’, yet on the other hand the latter will always be subject to debate across stakeholder groups.

All funding systems have an implied or explicit incentive dimension. In some countries, research excellence and funding have been decoupled, but to completely decouple them takes away the advantage that the QR funding brings. The UK has been a research superpower and in a post-Brexit world, tools such as REF that help sustain that position are of overwhelming economic, social and geopolitical importance.

There is a challenge in that the 7 or 8 implied or explicit purposes of the REF are not necessarily aligned and that ambiguity may not be helpful. How those purposes are explicitly ranked must inform any reshaping of the incentive scheme. Incentivising culture change is not the same as rewarding a highly impactful research unit for its outputs.

And that leads to a big question. The REF as it stands now is largely summative – reflecting past performance. Is that the most desirable or should it be more proactive in incentivising future performance, that is, focused more on inputs than outputs? I shall return to this point later.

I imagine that, in any reconsideration of the REF, this matter will for relative purpose be core to its top-level design.

Caught up in this is the contestation between what the Crown wants from the REF and what the institutions do with it for their own purposes. And here there is, in some commentators’ minds, a quagmire. How much of the cost is due to institutional uses and needs, and how much process is required to have a trusted system to distribute funds? Large within-institution factories have been created around the REF, but for purposes such as internal management that are well outside the Crown’s direct needs. Yet confusion on this matter underlies some critiques of the REF.

Form must follow function, and so discussion of the processes and mode of evaluation require some relative clarity on these matters.

One of the problems is that of language. ‘Excellence’ in research cannot be easily defined and will vary depending on the person defining it. Excellence has many dimensions: the nature of the question, how the work was done, how the team that did the work was composed, how the results were communicated, where they were communicated (which has undue influence), and what the work led to. Yet, metrics and the academic preference to trying to make a subjective  judgment objective leads to a narrow focus. The lower the unit of assessment, the more distorting that narrow focus becomes. In that sense, the move away from selected staff to universal portfolio submission following Stern’s report was critically valuable. What might seem like excellent research to one person may not be at a different time or under a different circumstance, and vice versa. Hence, averaging by integration becomes logical. As David Sweeney points out, even the word ‘excellence may be misleading and biasing.

Impact is an equally problematic but critically important concept. It is too early to know what the shape of the impact portfolio was in the 2021 round, but the past tendency to submit what some called exceptional impact cases at the expense of normal impact cases is a concern. Exceptional impact is what happens when one research finding leads to something happening, like the discovery of a new drug. But much impact occurs over many years in a less dramatic way. Some examples of critical forms of impact might be a HEI assisting local communities in environmental restoration, or providing government confidentially with a progressive understanding of the origins of extremism, or a novelist providing a narrative that changes public understandings. Often these more normal types of impact are the most critical and valuable, yet appear harder for institutions to focus on. This in turn relates to the ongoing need to better define and know how to assess transdisciplinary and stakeholder engaged research. To what extent are these the types of impact that the University as a whole rather than any lower unit of assessment should be accountable for? Is this a form of cultural change the REF should promote?

The unit of assessment is an important matter to consider. The REF is currently very output focused; is that the right balance between input and output, decision maker and performer? To quote from a recent email from a committee member to to me:

The institutions benefit from the REF funding and have the power to make change but it is the performance (excellence & impact) of the researchers that is assessed.

Does this mean that we should be more focused on assessing institutional efforts that will create incentives for institutional change and enhanced research quality and impact by internal decisions that benefit the researchers and their outputs in a desired manner? If so, how?

I cannot answer any of these questions, and you should not read any conclusion into my comments, but these questions are appropriate at this stage of my panel’s contributions in a review that can only now explore the results of the last round and the various consultations underway.

I am pleased that James Wilsdon and colleagues are undertaking a mini-review of whether we can use metrics in any way to further assist. The issue remains that different domains of scholarship have very different kinds of output, and that is a challenge of the REF process.

So, finally, to the REF process itself. No one wants excessive bureaucracy, and yet everyone wants fairness and objectivity. Given what we are dealing with, these aren’t easy to reconcile. As i have said earlier, institutions and academics can be complicit in generating this complexity. As we all know from the (likely apocryphal) quote from Einstein, Not everything that is important can be measured, and not everything that can be measured is important .

In closing, let me acknowledge the other members of the FRAP independent advisory panel; our global membership creates some challenging time zones for our Zoom calls, and I particularly acknowledge the officials at Research England and in the other three jurisdictions. I have worked with many officials in many countries over the years, and their grasp of the issues is absolutely exemplary.

So, the question remains: is it valuable to the UK to have an incentive system to reward impactful and quality research? Does it give confidence to the Crown to sustain funding in a way that can censure the continued importance of UK research of every type? In my view, the short answer is yes, and the long answer is a lot more reflection is needed to know what is the best and most acceptable way to do it.

Related items

Opening address to the International Ministerial Conference

Keynote address at the Institute for Advanced Study, University of Amsterdam

Science and science systems: beyond semantics

Future society, human potential, navigating change, informing policy.

The University of Chicago The Law School

Abrams environmental law clinic—significant achievements for 2023-24, protecting our great lakes, rivers, and shorelines.

The Abrams Clinic represents Friends of the Chicago River and the Sierra Club in their efforts to hold Trump Tower in downtown Chicago accountable for withdrawing water illegally from the Chicago River. To cool the building, Trump Tower draws water at high volumes, similar to industrial factories or power plants, but Trump Tower operated for more than a decade without ever conducting the legally required studies to determine the impact of those operations on aquatic life or without installing sufficient equipment to protect aquatic life consistent with federal regulations. After the Clinic sent a notice of intent to sue Trump Tower, the State of Illinois filed its own case in the summer of 2018, and the Clinic moved successfully to intervene in that case. In 2023-24, motions practice and discovery continued. Working with co-counsel at Northwestern University’s Pritzker Law School’s Environmental Advocacy Center, the Clinic moved to amend its complaint to include Trump Tower’s systematic underreporting each month of the volume of water that it intakes from and discharges to the Chicago River. The Clinic and co-counsel addressed Trump Tower’s motion to dismiss some of our clients’ claims, and we filed a motion for summary judgment on our claim that Trump Tower has committed a public nuisance. We also worked closely with our expert, Dr. Peter Henderson, on a supplemental disclosure and on defending an additional deposition of him. In summer 2024, the Clinic is defending its motion for summary judgment and challenging Trump Tower’s own motion for summary judgment. The Clinic is also preparing for trial, which could take place as early as fall 2024.

Since 2016, the Abrams Clinic has worked with the Chicago chapter of the Surfrider Foundation to protect water quality along the Lake Michigan shoreline in northwest Indiana, where its members surf. In April 2017, the U. S. Steel plant in Portage, Indiana, spilled approximately 300 pounds of hexavalent chromium into Lake Michigan. In January 2018, the Abrams Clinic filed a suit on behalf of Surfrider against U. S. Steel, alleging multiple violations of U. S. Steel’s discharge permits; the City of Chicago filed suit shortly after. When the US government and the State of Indiana filed their own, separate case, the Clinic filed extensive comments on the proposed consent decree. In August 2021, the court entered a revised consent decree which included provisions advocated for by Surfrider and the City of Chicago, namely a water sampling project that alerts beachgoers as to Lake Michigan’s water quality conditions, better notifications in case of future spills, and improvements to U. S. Steel’s operations and maintenance plans. In the 2023-24 academic year, the Clinic successfully litigated its claims for attorneys’ fees as a substantially prevailing party. Significantly, the court’s order adopted the “Fitzpatrick matrix,” used by the US Attorney’s Office for the District of Columbia to determine appropriate hourly rates for civil litigants, endorsed Chicago legal market rates as the appropriate rates for complex environmental litigation in Northwest Indiana, and allowed for partially reconstructed time records. The Clinic’s work, which has received significant media attention, helped to spawn other litigation to address pollution by other industrial facilities in Northwest Indiana and other enforcement against U. S. Steel by the State of Indiana.

In Winter Quarter 2024, Clinic students worked closely with Dr. John Ikerd, an agricultural economist and emeritus professor at the University of Missouri, to file an amicus brief in Food & Water Watch v. U.S. Environmental Protection Agency . In that case pending before the Ninth Circuit, Food & Water Watch argues that US EPA is illegally allowing Concentrated Animal Feeding Operations, more commonly known as factory farms, to pollute waterways significantly more than is allowable under the Clean Water Act. In the brief for Dr. Ikerd and co-amici Austin Frerick, Crawford Stewardship Project, Family Farm Defenders, Farm Aid, Missouri Rural Crisis Center, National Family Farm Coalition, National Sustainable Agriculture Coalition, and Western Organization of Resource Councils, we argued that EPA’s refusal to regulate CAFOs effectively is an unwarranted application of “agricultural exceptionalism” to industrial agriculture and that EPA effectively distorts the animal production market by allowing CAFOs to externalize their pollution costs and diminishing the ability of family farms to compete. Attorneys for the litigants will argue the case in September 2024.

Energy and Climate

Energy justice.

The Abrams Clinic supported grassroots organizations advocating for energy justice in low-income communities and Black, Indigenous, and People of Color (BIPOC) communities in Michigan. With the Clinic’s representation, these organizations intervened in cases before the Michigan Public Service Commission (MPSC), which regulates investor-owned utilities. Students conducted discovery, drafted written testimony, cross-examined utility executives, participated in settlement discussions, and filed briefs for these projects. The Clinic’s representation has elevated the concerns of these community organizations and forced both the utilities and regulators to consider issues of equity to an unprecedented degree. This year, on behalf of Soulardarity (Highland Park, MI), We Want Green, Too (Detroit, MI), and Urban Core Collective (Grand Rapids, MI), Clinic students engaged in eight contested cases before the MPSC against DTE Electric, DTE Gas, and Consumers Energy, as well as provided support for our clients’ advocacy in other non-contested MPSC proceedings.

The Clinic started this past fall with wins in three cases. First, the Clinic’s clients settled with DTE Electric in its Integrated Resource Plan case. The settlement included an agreement to close the second dirtiest coal power plant in Michigan three years early, $30 million from DTE’s shareholders to assist low-income customers in paying their bills, and $8 million from DTE’s shareholders toward a community fund that assists low-income customers with installing energy efficiency improvements, renewable energy, and battery technology. Second, in DTE Electric’s 2023 request for a rate hike (a “rate case”), the Commission required DTE Electric to develop a more robust environmental justice analysis and rejected the Company’s second attempt to waive consumer protections through a proposed electric utility prepayment program with a questionable history of success during its pilot run. The final Commission order and the administrative law judge’s proposal for final decision cited the Clinic’s testimony and briefs. Third, in Consumers Electric’s 2023 rate case, the Commission rejected the Company’s request for a higher ratepayer-funded return on its investments and required the Company to create a process that will enable intervenors to obtain accurate GIS data. The Clinic intends to use this data to map the disparate impact of infrastructure investment in low-income and BIPOC communities.

In the winter, the Clinic filed public comments regarding DTE Electric and Consumers Energy’s “distribution grid plans” (DGP) as well as supported interventions in two additional cases: Consumers Energy’s voluntary green pricing (VGP) case and the Clinic’s first case against the gas utility DTE Gas. Beginning with the DGP comments, the Clinic first addressed Consumers’s 2023 Electric Distribution Infrastructure Investment Plan (EDIIP), which detailed current distribution system health and the utility’s approximately $7 billion capital project planning ($2 billion of which went unaccounted for in the EDIIP) over 2023–2028. The Clinic then commented on DTE Electric’s 2023 DGP, which outlined the utility’s opaque project prioritization and planned more than $9 billion in capital investments and associated maintenance over 2024–2028. The comments targeted four areas of deficiencies in both the EDIIP and DGP: (1) inadequate consideration of distributed energy resources (DERs) as providing grid reliability, resiliency, and energy transition benefits; (2) flawed environmental justice analysis, particularly with respect to the collection of performance metrics and the narrow implementation of the Michigan Environmental Justice Screen Tool; (3) inequitable investment patterns across census tracts, with emphasis on DTE Electric’s skewed prioritization for retaining its old circuits rather than upgrading those circuits; and (4) failing to engage with community feedback.

For the VGP case against Consumers, the Clinic supported the filing of both an initial brief and reply brief requesting that the Commission reject the Company’s flawed proposal for a “community solar” program. In a prior case, the Clinic advocated for the development of a community solar program that would provide low-income, BIPOC communities with access to clean energy. As a result of our efforts, the Commission approved a settlement agreement requiring the Company “to evaluate and provide a strawman recommendation on community solar in its Voluntary Green Pricing Program.” However, the Company’s subsequent proposal in its VGP case violated the Commission’s order because it (1) was not consistent with the applicable law, MCL 460.1061; (2) was not a true community solar program; (3) lacked essential details; (4) failed to compensate subscribers sufficiently; (5) included overpriced and inflexible subscriptions; (6) excessively limited capacity; and (7) failed to provide a clear pathway for certain participants to transition into other VGP programs. For these reasons, the Clinic argued that the Commission should reject the Company’s proposal.

In DTE Gas’s current rate case, the Clinic worked with four witnesses to develop testimony that would rebut DTE Gas’s request for a rate hike on its customers. The testimony advocated for a pathway to a just energy transition that avoids dumping the costs of stranded gas assets on the low-income and BIPOC communities that are likely to be the last to electrify. Instead, the testimony proposed that the gas and electric utilities undertake integrated planning that would prioritize electric infrastructure over gas infrastructure investment to ensure that DTE Gas does not over-invest in gas infrastructure that will be rendered obsolete in the coming decades. The Clinic also worked with one expert witness to develop an analysis of DTE Gas’s unaffordable bills and inequitable shutoff, deposit, and collections practices. Lastly, the Clinic offered testimony on behalf of and from community members who would be directly impacted by the Company’s rate hike and lack of affordable and quality service. Clinic students have spent the summer drafting an approximately one-hundred-page brief making these arguments formally. We expect the Commission’s decision this fall.

Finally, both DTE Electric and Consumers Energy have filed additional requests for rate increases after the conclusion of their respective rate cases filed in 2023. On behalf of our Clients, the Clinic has intervened in these cases, and clinic students have already reviewed thousands of pages of documents and started to develop arguments and strategies to protect low-income and BIPOC communities from the utility’s ceaseless efforts to increase the cost of energy.

Corporate Climate Greenwashing

The Abrams Environmental Law Clinic worked with a leading international nonprofit dedicated to using the law to protect the environment to research corporate climate greenwashing, focusing on consumer protection, green financing, and securities liability. Clinic students spent the year examining an innovative state law, drafted a fifty-page guide to the statute and relevant cases, and examined how the law would apply to a variety of potential cases. Students then presented their findings in a case study and oral presentation to members of ClientEarth, including the organization’s North American head and members of its European team. The project helped identify the strengths and weaknesses of potential new strategies for increasing corporate accountability in the fight against climate change.

Land Contamination, Lead, and Hazardous Waste

The Abrams Clinic continues to represent East Chicago, Indiana, residents who live or lived on or adjacent to the USS Lead Superfund site. This year, the Clinic worked closely with the East Chicago/Calumet Coalition Community Advisory Group (CAG) to advance the CAG’s advocacy beyond the Superfund site and the adjacent Dupont RCRA site. Through multiple forms of advocacy, the clinics challenged the poor performance and permit modification and renewal attempts of Tradebe Treatment and Recycling, LLC (Tradebe), a hazardous waste storage and recycling facility in the community. Clinic students sent letters to US EPA and Indiana Department of Environmental Management officials about how IDEM has failed to assess meaningful penalties against Tradebe for repeated violations of the law and how IDEM has allowed Tradebe to continue to threaten public and worker health and safety by not improving its operations. Students also drafted substantial comments for the CAG on the US EPA’s Lead and Copper Rule improvements, the Suppliers’ Park proposed cleanup, and Sims Metal’s proposed air permit revisions. The Clinic has also continued working with the CAG, environmental experts, and regulators since US EPA awarded $200,000 to the CAG for community air monitoring. The Clinic and its clients also joined comments drafted by other environmental organizations about poor operations and loose regulatory oversight of several industrial facilities in the area.

Endangered Species

The Abrams Clinic represented the Center for Biological Diversity (CBD) and the Hoosier Environmental Council (HEC) in litigation regarding the US Fish and Wildlife Service’s (Service) failure to list the Kirtland’s snake as threatened or endangered under the Endangered Species Act. The Kirtland’s snake is a small, secretive, non-venomous snake historically located across the Midwest and the Ohio River Valley. Development and climate change have undermined large portions of the snake’s habitat, and populations are declining. Accordingly, the Clinic sued the Service in the US District Court for the District of Columbia last summer over the Service’s denial of CBD’s request to have the Kirtland’s snake protected. This spring, the Clinic was able to reach a settlement with the Service that requires the Service to reconsider its listing decision for the Kirtland’s snake and to pay attorney fees.

The Clinic also represented CBD in preparation for litigation regarding the Service’s failure to list another species as threatened or endangered. Threats from land development and climate change have devastated this species as well, and the species has already been extirpated from two of the sixteen US states in its range. As such, the Clinic worked this winter and spring to prepare a notice of intent (NOI) to sue the Service. The Team poured over hundreds of FOIA documents and dug into the Service’s supporting documentation to create strong arguments against the Service in the imminent litigation. The Clinic will send the NOI and file a complaint in the next few months.

Students and Faculty

Twenty-four law school students from the classes of 2024 and 2025 participated in the Clinic, performing complex legal research, reviewing documents obtained through discovery, drafting legal research memos and briefs, conferring with clients, conducting cross-examination, participating in settlement conferences, and arguing motions. Students secured nine clerkships, five were heading to private practice after graduation, and two are pursuing public interest work. Sam Heppell joined the Clinic from civil rights private practice, bringing the Clinic to its full complement of three attorneys.