Science Agencies Need Metascience Units

Summary

The US government is the largest funder of basic research in the US and the second largest funder of applied research after industry. Despite this predominant role, federal science agencies have limited information on how well that funding is being spent and how to spend it better.

Ideas about how to improve science funding abound, from shortening application timelines with fast grants to uplifting reviewers with AI tools. But most ideas lack strong evidence of efficacy, especially at scale or within the constraints of government. To find out how these ideas perform in practice, agencies need to be able to run experiments within their own walls, applying the scientific method to science itself. 

Science agencies can accomplish this by creating metascience units: nimble teams of government staff and academic experts housed within agency leadership offices. Using the tools of metascience, these units would run institutional experiments and analyze the treasure troves of internal agency data to provide evidence about how to most effectively advance breakthroughs.

These experiments and analyses would allow agency leaders to identify and quickly scale up improvements to program design. Metascience units could lead to agencies funding higher quality projects, cutting review timelines, and closing gaps in the scientific ecosystem. 

Motivation

The federal government spends nearly $200 billion a year on research and development. If we could make the process of identifying, distributing, and managing those funds even 5% more effective, we would effectively unlock another $10 billion a year for innovations that benefit Americans. For context, that is slightly more than what we spend annually on NASA’s entire science budget or on the National Cancer Institute.

The federal scientific enterprise is vast and decentralized. This allows agencies to develop a diversity of approaches to funding and conducting science, from small grants and innovation prizes to ARPAs and National Labs. But knowing which approach will work best to further an agency’s mission is not a solved problem. There is much we do not know about where scientific progress comes from.

When agencies make decisions about how they structure, fund and conduct science, they typically lack high-quality evidence. What federal agencies do know about the efficacy and efficiency of their science investments comes from ad hoc external program evaluations, idiosyncratic internal analyses, or academic studies with limited, largely historical data. Often, these products are neither timely nor focused enough to influence program decisions. At the same time, they don’t make sufficient use of the agencies’ internal data, even though that data could provide some of the clearest signals about program strengths and weaknesses.

Inefficiencies in federally funded science are well-documented. These include slowness in funding grantees, an overwhelming volume of paperwork for academics, limited funding for early career scientists who might have the most disruptive ideas, risk aversion in peer review, and difficulty in coordinating within and across agencies.  

We know the problems, but we need evidence on how to solve them. The best way to improve science investments would be for agencies themselves to conduct experiments, analyze their treasure troves of data, and scale approaches backed by their results. This will not happen organically. Existing agency staff do not have the mandate or bandwidth to do such research — especially with considerable staffing shortages in science agencies — and external researchers and consultants lack internal data and institutional knowledge. We need dedicated, empowered spaces within agencies to do this kind of work. 

Solution

Establish metascience units within federal science agencies

To generate the decision-relevant evidence and analysis they lack, science agencies should establish metascience units: proactive, nimble teams akin to internal think tanks. Units would analyze existing programs, conduct internal research, and develop better ways to fund and conduct science. Above all, the units should be anchored around answering the question: how should the agency invest its resources — whether budget, staff time, or the time of the reviewer community — to most effectively advance breakthroughs and further its mission? 

While new to the US, the idea has found footing in the UK, which launched its own Metascience Unit in 2024 to conduct experiments and share insights across the country’s science ecosystem. Since then, the UK Unit has seen early successes, including a pilot that sped up grant application decisions by three months. 

Since we advocated for the establishment of metascience units last year, the idea has also picked up steam across the federal government. The National Science Foundation (NSF)’s Fiscal Year 2027 Budget Request called for a metascience unit, and the White House Office of Science and Technology Policy (OSTP) Report Science: A New Golden Age made the same recommendation across science agencies more broadly. The National Institutes of Health (NIH) is in the process of launching the Office of Research Economics, Planning, and Analysis (OREPA), which will conduct economic research and facilitate replications. 

Metascience units should generate evidence about improvements to program design

There are numerous possible research questions about how to structure, fund, and conduct science. But metascience units should start with questions they are best suited to answer with the levers they uniquely have: administrative data and prospective experimentation. This suggests the units should focus on post-hoc analyses, pilots, and controlled experimentation to improve funding mechanisms and award evaluation processes. 

Problems should also be relevant to program design. They should therefore be sourced from divisions and programs that would actually implement them, and agency leadership can prioritize metascience experiments that provide the most decision-relevant information.

Improvements that a metascience unit could research include:¹

• Fast grants, to move from application to funding in weeks rather than months (or years). The NSF effectively implemented fast grants during COVID-19, and other research funders have honed similar models. Metascience units should test how fast grants perform relative to conventional grants, using tools like cost-benefit analysis. They can gather evidence on when agencies should be using fast grants versus generally speeding up the conventional grant process. 
• Multi-stage review, to address the challenge of applicants and reviewers spending substantial amounts of time on applications that will not be funded. Multi-stage review requires applicants to submit a brief concept note to start, and then only invited applicants submit full applications. This approach is already used in some agencies, but a 2025 National Academies of Sciences, Engineering, and Medicine report proposed expanding it government-wide. While this is a promising direction, metascience units should conduct experiments that compare multi-stage review to conventional processes to see how it impacts both efficiency and the quality of research. 
• Golden Tickets, to address the problem of conventional review panels often rewarding consensus and punishing novel ideas. Golden tickets allow individual panelists to champion applications for funding even if other reviewers rated them poorly, potentially encouraging high-risk, high-reward projects. Metascience units should conduct experiments to compare golden tickets to more conventional panel review approaches. They can also use historical data to simulate what applications might have won if a panelist who was alone in giving a high score had gotten their way. A related approach would be testing the effectiveness of instructing reviewers to consider the benefits of higher-risk projects (as NIH has instructed panels to do) or encouraging reviewers to reward novelty. 
• Person-based grants, to learn whether betting on researchers’ longer-term agendas rather than specific proposals yields better results. Science agencies already fund some person-based grants at research institutions, such as NIH’s R35 Investigator-Focused Awards. Evidence suggests that person-based grants can free scientists to conduct high-impact research rather than focusing on grant writing and reporting. Person-based grants should be compared against standard project-based grants.
• Longer grant durations or higher award values, to test whether giving researchers more time or money leads to more ambitious and creative work. New program types, like NSF’s recently-launched X-Labs program, can combine flexibility with award sizes that encourage scientists to take on ambitious projects that require sophisticated infrastructure and multidisciplinary teams.² We think this will enable more high-impact research. Before expanding these pilots, metascience units should carefully analyze them to understand what’s working. 
• New AI-enabled tools, to identify the most appropriate ways to leverage the technology for funding decisions. Metascience units are well-suited to measure whether AI input could improve reviewer accuracy through controlled “uplift” studies. Uplift studies are used to compare how people perform tasks with and without access to an AI tool, and they can help metascience units assess whether the tool provides meaningful assistance to agency staff. Promising applications include AI-assisted proposal scoring, automated extraction of proposal characteristics for portfolio analysis, and reviewer matching based on AI network analysis. UK Research and Innovation, where the UK Metascience Unit is housed, has already begun studying AI as a supplemental tool in grant proposal review. This would be best approached as a test environment where metascience units iterate on AI applications faster than an agency’s usual processes allow, building on the agency’s past data and grounding the applications in meaningful correlates of scientific impact.

These pilots and controlled experiments test alternatives directly. Alongside those activities, post-hoc analysis of how existing programs have actually performed are also useful. These efforts can reinforce each other. Retrospective analysis of existing data may be one of the cheapest ways to identify which questions are worth the cost of an experiment.

Metascience units should make experimentation and internal data analysis easy

Metascience units can overcome the current difficulty of experimenting with new approaches and analyzing internal data. They can do this at a high level by making agency-level decisions about questions to tackle, cultivating talent in-house, shortening feedback loops between generating evidence and making decisions, and tapping into the institutional knowledge of agency staff in a way that external researchers or consultants could not.  

But more importantly, metascience units should build the infrastructure for experimentation and internal data analysis as a core part of their mission. 

To reduce the operational burden of running experiments, units can work with enterprise systems management and Chief Information Officers to build testing features into proposal management systems. Every improvement to this infrastructure lowers the cost of all following experiments. 

Without careful attention, experiments could become burdensome for programs to partake in. Metascience units should build light-touch experimentation into default systems and provide program offices with incentives to participate. In some randomized controlled trials of new approaches, metascience units could provide additional funding for control groups, so programs don’t have to sacrifice portions of their budgets to generate evidence.

For internal data, the metascience unit should coordinate with system owners to capture and structure the data that makes evidence building possible. Much of the unit’s early value will be in organizing administrative data on grant applications and selection processes, especially non-winning applications, that is largely inaccessible to external researchers. 

But the unit should also go beyond traditional output measures like citations and patents, tracking a wider range of outcomes: measures of scientific breakthroughs, grant turnaround times, staff time per award, and other process costs. Centralizing data on turnaround times and reviewer feedback quality in particular would give science agencies a clearer picture of how their processes perform over time.

Implementation

Metascience units will only pay off if enabled to answer a science agency’s most important questions. Without sufficient staffing, support for cross-program experimentation, or the determination to scale the approaches that work, the promise of a metascience unit may end up unrealized.

Rather than being buried in bureaucracy, metascience units should be housed in an agency leadership office and entrusted with the authority to mandate pilots or experiments across agency programs. This ensures alignment with agency-wide goals, protects against capture by any one program, and provides a clear home for experimentation, rather than having the unit serve an advisory role. 

To execute on the science agency’s metascience vision, leaders should prioritize recruiting talent to staff the offices and building strong relationships with stakeholders. These units should be staffed with a mix of civil servants and rotational experts like academic economists brought in via the Intergovernment Personnel Act (IPA). Career roles will hold institutional memory, while shorter rotational roles can bring in new ideas from the field.³ These staff must pay attention to the unit’s relationship with the metascience community and not assume it naturally maintains itself. 

Metascience units should not function as traditional grantmaking or program evaluation offices

Academic metascientists have contributed strong research on some of the questions outlined here, and agencies should continue to support these researchers with external grants. But metascience units should not become grants or contracting offices for metascience. This would distract from their distinct comparative advantage: conducting internal analysis and supporting decision making. 

For example, the programs on Science of Science & Innovation Policy (SciSIP) and Science of Science: Discovery, Communication and Impact (SoS:DCI) at NSF have funded foundational metascience research. But these sorts of programs do not focus on NSF’s own performance, nor should they. Tasking a metascience unit with external science-of-science grantmaking would distract from its goal, forcing the unit to develop extensive grant and contract management capabilities in addition to its core metascience capabilities.  The metascience unit should maintain awareness of the external literature to inform the experiments it runs, but it should not be primarily responsible for the oversight of those awards.

To develop actionable research, agencies need proactive and nimble analysis directed at decision support. The evaluation offices that responded to the Evidence Act pushed agencies to make better use of their data, but did not produce the actionable evidence needed to make decisions. Evaluation offices often included a mix of managing traditional program evaluation activities (often conducted by third party contractors), reporting to Congress and regulatory agencies, and requesting and organizing data from offices across their agency.  Unfortunately, in many cases they were overly burdened with compliance exercises and underdeveloped in key capability areas, which complicated their success. Metascience units must learn from those lessons. Agencies should resist the urge to combine metascience units with these and other program performance oversight functions. These units should be entrepreneurial and forward-looking, not compliance and backward-looking.

Which agencies should have metascience units?

Not all science agencies need a metascience unit. To begin with, metascience units should be placed in the agencies that are the largest science funders. The NSF’s Fiscal Year 2027 Budget Request includes funding for a metascience unit, and the NIH is in the process of standing up OREPA to perform some functions of a metascience unit. The NIH has substantial internal capacity already, including a science of science scholars program, but due to the complexity of NIH’s 27 Institute and Center structure, OREPA could still unlock more effective cross-agency experimentation if given sufficient authority. 

Additional agencies that should consider them include the Department of Energy (DOE), Department of War (DOW), and NASA. Given the prominence of Genesis Mission and massive ongoing infrastructure investments in the National Labs, DOE has unique metascience questions that won’t be answered by other agencies. DOW and NASA invest in diverse and complex R&D programs using a wide variety of innovative financing approaches, and both agencies could focus a metascience unit’s attention on industry-specific, translational questions.

Setting up metascience units for long-term success

Metascience units will likely produce decision-relevant evidence faster than external academics, and the experiments we suggested above could inform program design now. But the social return on scientific research is realized in long timelines (often over a decade or more), and some metascience unit projects will take time to pay off, likely crossing presidential administrations. 

Those long-horizon payoffs require not treating these units as disposable, short-term efforts. Instead, metascience units should receive stable, sustained investment even as individual experiments resolve quickly. The White House and Congress have important roles to play to that end.

The White House has been supportive of metascience units, as exemplified by the Science: A New Golden Age report. To ensure the units’ success, they should support agency leadership in staffing and funding requests for metascience units, as well as providing flexibility and recognizing that the units may need to shift focus. 

Metascience units provide benefits regardless of which party is in power by providing insights on science agency effectiveness and opportunities for improvement. Congress and the White House should therefore ensure that metascience units remain apolitical, truth-seeking offices and, to that end, support the sharing of findings publicly. A coordination body, perhaps convened by OSTP as a National Science and Technology Council subcommittee, could help in coordinating between metascience units, facilitating information sharing, and encouraging the external publication of results. 

It’s also important to remember that metascience units themselves are a new institution in science — and are an experiment. As such, they too should be evaluated and improved based on evidence. And if they succeed in delivering more efficient, higher impact science, metascience units could be their own kind of breakthrough.