empirical evidence and research

• History & Society
• Science & Tech
• Biographies
• Animals & Nature
• Geography & Travel
• Arts & Culture
• Games & Quizzes
• On This Day
• One Good Fact
• New Articles
• Lifestyles & Social Issues
• Philosophy & Religion
• Politics, Law & Government
• World History
• Health & Medicine
• Browse Biographies
• Birds, Reptiles & Other Vertebrates
• Bugs, Mollusks & Other Invertebrates
• Environment
• Fossils & Geologic Time
• Entertainment & Pop Culture
• Sports & Recreation
• Visual Arts
• Demystified
• Image Galleries
• Infographics
• Top Questions
• Britannica Kids
• Saving Earth
• Space Next 50
• Student Center

• Where was science invented?
• When did science begin?

empirical evidence

Our editors will review what you’ve submitted and determine whether to revise the article.

• National Center for Biotechnology Information - PubMed Central - Expert opinion vs. empirical evidence
• LiveScience - Empirical evidence: A definition
• Academia - Empirical Evidence

empirical evidence , information gathered directly or indirectly through observation or experimentation that may be used to confirm or disconfirm a scientific theory or to help justify, or establish as reasonable, a person’s belief in a given proposition. A belief may be said to be justified if there is sufficient evidence to make holding the belief reasonable.

The concept of evidence is the basis of philosophical evidentialism, an epistemological thesis according to which a person is justified in believing a given proposition p if and only if the person’s evidence for p is proper or sufficient. In this context , the Scottish Enlightenment philosopher David Hume (1711–76) famously asserted that the “wise man…proportions his belief to the evidence.” In a similar vein, the American astronomer Carl Sagan popularized the statement, “Extraordinary claims require extraordinary evidence.”

Foundationalists , however, defend the view that certain basic, or foundational, beliefs are either inherently justified or justified by something other than another belief (e.g., a sensation or perception) and that all other beliefs may be justified only if they are directly or indirectly supported by at least one foundational belief (that is, only if they are either supported by at least one foundational belief or supported by other beliefs that are themselves supported by at least one foundational belief). The most influential foundationalist of the modern period was the French philosopher and mathematician René Descartes (1596–1650), who attempted to establish a foundation for justified beliefs regarding an external world in his intuition that, for as long as he is thinking, he exists (“I think, therefore I am”; see cogito, ergo sum ). A traditional argument in favour of foundationalism asserts that no other account of inferential justification—the act of justifying a given belief by inferring it from another belief that itself is justified—is possible. Thus, assume that one belief, Belief 1, is justified by another belief, Belief 2. How is Belief 2 justified? It cannot be justified by Belief 1, because the inference from Belief 2 to Belief 1 would then be circular and invalid. It cannot be justified by a third nonfoundational Belief 3, because the same question would then apply to that belief, leading to an infinite regress. And one cannot simply assume that Belief 2 is not justified, for then Belief 1 would not be justified through the inference from Belief 2. Accordingly, there must be some beliefs whose justification does not depend on other beliefs, and those justified beliefs must function as a foundation for the inferential justification of other beliefs.

Empirical evidence can be quantitative or qualitative. Typically, numerical quantitative evidence can be represented visually by means of diagrams, graphs, or charts, reflecting the use of statistical or mathematical data and the researcher’s neutral noninteractive role. It can be obtained by methods such as experiments, surveys, correlational research (to study the relationship between variables), cross-sectional research (to compare different groups), causal-comparative research (to explore cause-effect relationships), and longitudinal studies (to test a subject during a given time period).

Qualitative evidence, on the other hand, can foster a deeper understanding of behaviour and related factors and is not typically expressed by using numbers. Often subjective and resulting from interaction between the researcher and participants, it can stem from the use of methods such as interviews (based on verbal interaction), observation (informing ethnographic research design), textual analysis (involving the description and interpretation of texts), focus groups (planned group discussions), and case studies (in-depth analyses of individuals or groups).

Empirical evidence is subject to assessments of its validity. Validity can be internal, involving the soundness of an experiment’s design and execution and the accuracy of subsequent data analysis , or external, involving generalizability to other research contexts ( see ecological validity ).

• Skip to main content
• Skip to primary sidebar
• Skip to footer
• QuestionPro

• Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case AskWhy Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
• Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Market Research

Empirical Research: Definition, Methods, Types and Examples

Content Index

Empirical research: Definition

Empirical research: origin, quantitative research methods, qualitative research methods, steps for conducting empirical research, empirical research methodology cycle, advantages of empirical research, disadvantages of empirical research, why is there a need for empirical research.

Empirical research is defined as any research where conclusions of the study is strictly drawn from concretely empirical evidence, and therefore “verifiable” evidence.

This empirical evidence can be gathered using quantitative market research and  qualitative market research  methods.

For example: A research is being conducted to find out if listening to happy music in the workplace while working may promote creativity? An experiment is conducted by using a music website survey on a set of audience who are exposed to happy music and another set who are not listening to music at all, and the subjects are then observed. The results derived from such a research will give empirical evidence if it does promote creativity or not.

LEARN ABOUT: Behavioral Research

You must have heard the quote” I will not believe it unless I see it”. This came from the ancient empiricists, a fundamental understanding that powered the emergence of medieval science during the renaissance period and laid the foundation of modern science, as we know it today. The word itself has its roots in greek. It is derived from the greek word empeirikos which means “experienced”.

In today’s world, the word empirical refers to collection of data using evidence that is collected through observation or experience or by using calibrated scientific instruments. All of the above origins have one thing in common which is dependence of observation and experiments to collect data and test them to come up with conclusions.

LEARN ABOUT: Causal Research

Types and methodologies of empirical research

Empirical research can be conducted and analysed using qualitative or quantitative methods.

• Quantitative research : Quantitative research methods are used to gather information through numerical data. It is used to quantify opinions, behaviors or other defined variables . These are predetermined and are in a more structured format. Some of the commonly used methods are survey, longitudinal studies, polls, etc
• Qualitative research:   Qualitative research methods are used to gather non numerical data.  It is used to find meanings, opinions, or the underlying reasons from its subjects. These methods are unstructured or semi structured. The sample size for such a research is usually small and it is a conversational type of method to provide more insight or in-depth information about the problem Some of the most popular forms of methods are focus groups, experiments, interviews, etc.

Data collected from these will need to be analysed. Empirical evidence can also be analysed either quantitatively and qualitatively. Using this, the researcher can answer empirical questions which have to be clearly defined and answerable with the findings he has got. The type of research design used will vary depending on the field in which it is going to be used. Many of them might choose to do a collective research involving quantitative and qualitative method to better answer questions which cannot be studied in a laboratory setting.

LEARN ABOUT: Qualitative Research Questions and Questionnaires

Quantitative research methods aid in analyzing the empirical evidence gathered. By using these a researcher can find out if his hypothesis is supported or not.

• Survey research: Survey research generally involves a large audience to collect a large amount of data. This is a quantitative method having a predetermined set of closed questions which are pretty easy to answer. Because of the simplicity of such a method, high responses are achieved. It is one of the most commonly used methods for all kinds of research in today’s world.

Previously, surveys were taken face to face only with maybe a recorder. However, with advancement in technology and for ease, new mediums such as emails , or social media have emerged.

For example: Depletion of energy resources is a growing concern and hence there is a need for awareness about renewable energy. According to recent studies, fossil fuels still account for around 80% of energy consumption in the United States. Even though there is a rise in the use of green energy every year, there are certain parameters because of which the general population is still not opting for green energy. In order to understand why, a survey can be conducted to gather opinions of the general population about green energy and the factors that influence their choice of switching to renewable energy. Such a survey can help institutions or governing bodies to promote appropriate awareness and incentive schemes to push the use of greener energy.

Learn more: Renewable Energy Survey Template Descriptive Research vs Correlational Research

• Experimental research: In experimental research , an experiment is set up and a hypothesis is tested by creating a situation in which one of the variable is manipulated. This is also used to check cause and effect. It is tested to see what happens to the independent variable if the other one is removed or altered. The process for such a method is usually proposing a hypothesis, experimenting on it, analyzing the findings and reporting the findings to understand if it supports the theory or not.

For example: A particular product company is trying to find what is the reason for them to not be able to capture the market. So the organisation makes changes in each one of the processes like manufacturing, marketing, sales and operations. Through the experiment they understand that sales training directly impacts the market coverage for their product. If the person is trained well, then the product will have better coverage.

• Correlational research: Correlational research is used to find relation between two set of variables . Regression analysis is generally used to predict outcomes of such a method. It can be positive, negative or neutral correlation.

LEARN ABOUT: Level of Analysis

For example: Higher educated individuals will get higher paying jobs. This means higher education enables the individual to high paying job and less education will lead to lower paying jobs.

• Longitudinal study: Longitudinal study is used to understand the traits or behavior of a subject under observation after repeatedly testing the subject over a period of time. Data collected from such a method can be qualitative or quantitative in nature.

For example: A research to find out benefits of exercise. The target is asked to exercise everyday for a particular period of time and the results show higher endurance, stamina, and muscle growth. This supports the fact that exercise benefits an individual body.

• Cross sectional: Cross sectional study is an observational type of method, in which a set of audience is observed at a given point in time. In this type, the set of people are chosen in a fashion which depicts similarity in all the variables except the one which is being researched. This type does not enable the researcher to establish a cause and effect relationship as it is not observed for a continuous time period. It is majorly used by healthcare sector or the retail industry.

For example: A medical study to find the prevalence of under-nutrition disorders in kids of a given population. This will involve looking at a wide range of parameters like age, ethnicity, location, incomes  and social backgrounds. If a significant number of kids coming from poor families show under-nutrition disorders, the researcher can further investigate into it. Usually a cross sectional study is followed by a longitudinal study to find out the exact reason.

• Causal-Comparative research : This method is based on comparison. It is mainly used to find out cause-effect relationship between two variables or even multiple variables.

For example: A researcher measured the productivity of employees in a company which gave breaks to the employees during work and compared that to the employees of the company which did not give breaks at all.

LEARN ABOUT: Action Research

Some research questions need to be analysed qualitatively, as quantitative methods are not applicable there. In many cases, in-depth information is needed or a researcher may need to observe a target audience behavior, hence the results needed are in a descriptive analysis form. Qualitative research results will be descriptive rather than predictive. It enables the researcher to build or support theories for future potential quantitative research. In such a situation qualitative research methods are used to derive a conclusion to support the theory or hypothesis being studied.

LEARN ABOUT: Qualitative Interview

• Case study: Case study method is used to find more information through carefully analyzing existing cases. It is very often used for business research or to gather empirical evidence for investigation purpose. It is a method to investigate a problem within its real life context through existing cases. The researcher has to carefully analyse making sure the parameter and variables in the existing case are the same as to the case that is being investigated. Using the findings from the case study, conclusions can be drawn regarding the topic that is being studied.

For example: A report mentioning the solution provided by a company to its client. The challenges they faced during initiation and deployment, the findings of the case and solutions they offered for the problems. Such case studies are used by most companies as it forms an empirical evidence for the company to promote in order to get more business.

• Observational method:   Observational method is a process to observe and gather data from its target. Since it is a qualitative method it is time consuming and very personal. It can be said that observational research method is a part of ethnographic research which is also used to gather empirical evidence. This is usually a qualitative form of research, however in some cases it can be quantitative as well depending on what is being studied.

For example: setting up a research to observe a particular animal in the rain-forests of amazon. Such a research usually take a lot of time as observation has to be done for a set amount of time to study patterns or behavior of the subject. Another example used widely nowadays is to observe people shopping in a mall to figure out buying behavior of consumers.

• One-on-one interview: Such a method is purely qualitative and one of the most widely used. The reason being it enables a researcher get precise meaningful data if the right questions are asked. It is a conversational method where in-depth data can be gathered depending on where the conversation leads.

For example: A one-on-one interview with the finance minister to gather data on financial policies of the country and its implications on the public.

• Focus groups: Focus groups are used when a researcher wants to find answers to why, what and how questions. A small group is generally chosen for such a method and it is not necessary to interact with the group in person. A moderator is generally needed in case the group is being addressed in person. This is widely used by product companies to collect data about their brands and the product.

For example: A mobile phone manufacturer wanting to have a feedback on the dimensions of one of their models which is yet to be launched. Such studies help the company meet the demand of the customer and position their model appropriately in the market.

• Text analysis: Text analysis method is a little new compared to the other types. Such a method is used to analyse social life by going through images or words used by the individual. In today’s world, with social media playing a major part of everyone’s life, such a method enables the research to follow the pattern that relates to his study.

For example: A lot of companies ask for feedback from the customer in detail mentioning how satisfied are they with their customer support team. Such data enables the researcher to take appropriate decisions to make their support team better.

Sometimes a combination of the methods is also needed for some questions that cannot be answered using only one type of method especially when a researcher needs to gain a complete understanding of complex subject matter.

We recently published a blog that talks about examples of qualitative data in education ; why don’t you check it out for more ideas?

Learn More: Data Collection Methods: Types & Examples

Since empirical research is based on observation and capturing experiences, it is important to plan the steps to conduct the experiment and how to analyse it. This will enable the researcher to resolve problems or obstacles which can occur during the experiment.

Step #1: Define the purpose of the research

This is the step where the researcher has to answer questions like what exactly do I want to find out? What is the problem statement? Are there any issues in terms of the availability of knowledge, data, time or resources. Will this research be more beneficial than what it will cost.

Before going ahead, a researcher has to clearly define his purpose for the research and set up a plan to carry out further tasks.

Step #2 : Supporting theories and relevant literature

The researcher needs to find out if there are theories which can be linked to his research problem . He has to figure out if any theory can help him support his findings. All kind of relevant literature will help the researcher to find if there are others who have researched this before, or what are the problems faced during this research. The researcher will also have to set up assumptions and also find out if there is any history regarding his research problem

Step #3: Creation of Hypothesis and measurement

Before beginning the actual research he needs to provide himself a working hypothesis or guess what will be the probable result. Researcher has to set up variables, decide the environment for the research and find out how can he relate between the variables.

Researcher will also need to define the units of measurements, tolerable degree for errors, and find out if the measurement chosen will be acceptable by others.

Step #4: Methodology, research design and data collection

In this step, the researcher has to define a strategy for conducting his research. He has to set up experiments to collect data which will enable him to propose the hypothesis. The researcher will decide whether he will need experimental or non experimental method for conducting the research. The type of research design will vary depending on the field in which the research is being conducted. Last but not the least, the researcher will have to find out parameters that will affect the validity of the research design. Data collection will need to be done by choosing appropriate samples depending on the research question. To carry out the research, he can use one of the many sampling techniques. Once data collection is complete, researcher will have empirical data which needs to be analysed.

LEARN ABOUT: Best Data Collection Tools

Step #5: Data Analysis and result

Data analysis can be done in two ways, qualitatively and quantitatively. Researcher will need to find out what qualitative method or quantitative method will be needed or will he need a combination of both. Depending on the unit of analysis of his data, he will know if his hypothesis is supported or rejected. Analyzing this data is the most important part to support his hypothesis.

Step #6: Conclusion

A report will need to be made with the findings of the research. The researcher can give the theories and literature that support his research. He can make suggestions or recommendations for further research on his topic.

A.D. de Groot, a famous dutch psychologist and a chess expert conducted some of the most notable experiments using chess in the 1940’s. During his study, he came up with a cycle which is consistent and now widely used to conduct empirical research. It consists of 5 phases with each phase being as important as the next one. The empirical cycle captures the process of coming up with hypothesis about how certain subjects work or behave and then testing these hypothesis against empirical data in a systematic and rigorous approach. It can be said that it characterizes the deductive approach to science. Following is the empirical cycle.

• Observation: At this phase an idea is sparked for proposing a hypothesis. During this phase empirical data is gathered using observation. For example: a particular species of flower bloom in a different color only during a specific season.
• Induction: Inductive reasoning is then carried out to form a general conclusion from the data gathered through observation. For example: As stated above it is observed that the species of flower blooms in a different color during a specific season. A researcher may ask a question “does the temperature in the season cause the color change in the flower?” He can assume that is the case, however it is a mere conjecture and hence an experiment needs to be set up to support this hypothesis. So he tags a few set of flowers kept at a different temperature and observes if they still change the color?
• Deduction: This phase helps the researcher to deduce a conclusion out of his experiment. This has to be based on logic and rationality to come up with specific unbiased results.For example: In the experiment, if the tagged flowers in a different temperature environment do not change the color then it can be concluded that temperature plays a role in changing the color of the bloom.
• Testing: This phase involves the researcher to return to empirical methods to put his hypothesis to the test. The researcher now needs to make sense of his data and hence needs to use statistical analysis plans to determine the temperature and bloom color relationship. If the researcher finds out that most flowers bloom a different color when exposed to the certain temperature and the others do not when the temperature is different, he has found support to his hypothesis. Please note this not proof but just a support to his hypothesis.
• Evaluation: This phase is generally forgotten by most but is an important one to keep gaining knowledge. During this phase the researcher puts forth the data he has collected, the support argument and his conclusion. The researcher also states the limitations for the experiment and his hypothesis and suggests tips for others to pick it up and continue a more in-depth research for others in the future. LEARN MORE: Population vs Sample

LEARN MORE: Population vs Sample

There is a reason why empirical research is one of the most widely used method. There are a few advantages associated with it. Following are a few of them.

• It is used to authenticate traditional research through various experiments and observations.
• This research methodology makes the research being conducted more competent and authentic.
• It enables a researcher understand the dynamic changes that can happen and change his strategy accordingly.
• The level of control in such a research is high so the researcher can control multiple variables.
• It plays a vital role in increasing internal validity .

Even though empirical research makes the research more competent and authentic, it does have a few disadvantages. Following are a few of them.

• Such a research needs patience as it can be very time consuming. The researcher has to collect data from multiple sources and the parameters involved are quite a few, which will lead to a time consuming research.
• Most of the time, a researcher will need to conduct research at different locations or in different environments, this can lead to an expensive affair.
• There are a few rules in which experiments can be performed and hence permissions are needed. Many a times, it is very difficult to get certain permissions to carry out different methods of this research.
• Collection of data can be a problem sometimes, as it has to be collected from a variety of sources through different methods.

LEARN ABOUT:  Social Communication Questionnaire

Empirical research is important in today’s world because most people believe in something only that they can see, hear or experience. It is used to validate multiple hypothesis and increase human knowledge and continue doing it to keep advancing in various fields.

For example: Pharmaceutical companies use empirical research to try out a specific drug on controlled groups or random groups to study the effect and cause. This way, they prove certain theories they had proposed for the specific drug. Such research is very important as sometimes it can lead to finding a cure for a disease that has existed for many years. It is useful in science and many other fields like history, social sciences, business, etc.

LEARN ABOUT: 12 Best Tools for Researchers

With the advancement in today’s world, empirical research has become critical and a norm in many fields to support their hypothesis and gain more knowledge. The methods mentioned above are very useful for carrying out such research. However, a number of new methods will keep coming up as the nature of new investigative questions keeps getting unique or changing.

Create a single source of real data with a built-for-insights platform. Store past data, add nuggets of insights, and import research data from various sources into a CRM for insights. Build on ever-growing research with a real-time dashboard in a unified research management platform to turn insights into knowledge.

LEARN MORE         FREE TRIAL

MORE LIKE THIS

Jotform vs Microsoft Forms: Which Should You Choose?

Aug 26, 2024

Stay Conversations: What Is It, How to Use, Questions to Ask

Age Gating: Effective Strategies for Online Content Control

Aug 23, 2024

Work-Life Balance: Why We Need it & How to Improve It

Aug 22, 2024

Other categories

• Academic Research
• Artificial Intelligence
• Assessments
• Brand Awareness
• Case Studies
• Communities
• Consumer Insights
• Customer effort score
• Customer Engagement
• Customer Experience
• Customer Loyalty
• Customer Research
• Customer Satisfaction
• Employee Benefits
• Employee Engagement
• Employee Retention
• Friday Five
• General Data Protection Regulation
• Insights Hub
• Life@QuestionPro
• Market Research
• Mobile diaries
• Mobile Surveys
• New Features
• Online Communities
• Question Types
• Questionnaire
• QuestionPro Products
• Release Notes
• Research Tools and Apps
• Revenue at Risk
• Survey Templates
• Training Tips
• Tuesday CX Thoughts (TCXT)
• Uncategorized
• What’s Coming Up
• Workforce Intelligence

• University of Memphis Libraries
• Research Guides

Empirical Research: Defining, Identifying, & Finding

Defining empirical research, what is empirical research, quantitative or qualitative.

• Introduction
• Database Tools
• Search Terms
• Image Descriptions

Calfee & Chambliss (2005)  (UofM login required) describe empirical research as a "systematic approach for answering certain types of questions."  Those questions are answered "[t]hrough the collection of evidence under carefully defined and replicable conditions" (p. 43). 

The evidence collected during empirical research is often referred to as "data." 

Characteristics of Empirical Research

Emerald Publishing's guide to conducting empirical research identifies a number of common elements to empirical research: 

• A  research question , which will determine research objectives.
• A particular and planned  design  for the research, which will depend on the question and which will find ways of answering it with appropriate use of resources.
• The gathering of  primary data , which is then analysed.
• A particular  methodology  for collecting and analysing the data, such as an experiment or survey.
• The limitation of the data to a particular group, area or time scale, known as a sample [emphasis added]: for example, a specific number of employees of a particular company type, or all users of a library over a given time scale. The sample should be somehow representative of a wider population.
• The ability to  recreate  the study and test the results. This is known as  reliability .
• The ability to  generalize  from the findings to a larger sample and to other situations.

If you see these elements in a research article, you can feel confident that you have found empirical research. Emerald's guide goes into more detail on each element. 

Empirical research methodologies can be described as quantitative, qualitative, or a mix of both (usually called mixed-methods).

Ruane (2016)  (UofM login required) gets at the basic differences in approach between quantitative and qualitative research:

• Quantitative research  -- an approach to documenting reality that relies heavily on numbers both for the measurement of variables and for data analysis (p. 33).
• Qualitative research  -- an approach to documenting reality that relies on words and images as the primary data source (p. 33).

Both quantitative and qualitative methods are empirical . If you can recognize that a research study is quantitative or qualitative study, then you have also recognized that it is empirical study. 

Below are information on the characteristics of quantitative and qualitative research. This video from Scribbr also offers a good overall introduction to the two approaches to research methodology: 

Characteristics of Quantitative Research 

Researchers test hypotheses, or theories, based in assumptions about causality, i.e. we expect variable X to cause variable Y. Variables have to be controlled as much as possible to ensure validity. The results explain the relationship between the variables. Measures are based in pre-defined instruments.

Examples: experimental or quasi-experimental design, pretest & post-test, survey or questionnaire with closed-ended questions. Studies that identify factors that influence an outcomes, the utility of an intervention, or understanding predictors of outcomes. 

Characteristics of Qualitative Research

Researchers explore “meaning individuals or groups ascribe to social or human problems (Creswell & Creswell, 2018, p3).” Questions and procedures emerge rather than being prescribed. Complexity, nuance, and individual meaning are valued. Research is both inductive and deductive. Data sources are multiple and varied, i.e. interviews, observations, documents, photographs, etc. The researcher is a key instrument and must be reflective of their background, culture, and experiences as influential of the research.

Examples: open question interviews and surveys, focus groups, case studies, grounded theory, ethnography, discourse analysis, narrative, phenomenology, participatory action research.

Calfee, R. C. & Chambliss, M. (2005). The design of empirical research. In J. Flood, D. Lapp, J. R. Squire, & J. Jensen (Eds.),  Methods of research on teaching the English language arts: The methodology chapters from the handbook of research on teaching the English language arts (pp. 43-78). Routledge.  http://ezproxy.memphis.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=nlebk&AN=125955&site=eds-live&scope=site .

Creswell, J. W., & Creswell, J. D. (2018).  Research design: Qualitative, quantitative, and mixed methods approaches  (5th ed.). Thousand Oaks: Sage.

How to... conduct empirical research . (n.d.). Emerald Publishing.  https://www.emeraldgrouppublishing.com/how-to/research-methods/conduct-empirical-research .

Scribbr. (2019). Quantitative vs. qualitative: The differences explained  [video]. YouTube.  https://www.youtube.com/watch?v=a-XtVF7Bofg .

Ruane, J. M. (2016).  Introducing social research methods : Essentials for getting the edge . Wiley-Blackwell.  http://ezproxy.memphis.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=nlebk&AN=1107215&site=eds-live&scope=site .  

• << Previous: Home
• Next: Identifying Empirical Research >>
• Last Updated: Apr 2, 2024 11:25 AM
• URL: https://libguides.memphis.edu/empirical-research

Empirical evidence: A definition

Empirical evidence is information that is acquired by observation or experimentation.

The scientific method

Types of empirical research, identifying empirical evidence, empirical law vs. scientific law, empirical, anecdotal and logical evidence, additional resources and reading, bibliography.

Empirical evidence is information acquired by observation or experimentation. Scientists record and analyze this data. The process is a central part of the scientific method , leading to the proving or disproving of a hypothesis and our better understanding of the world as a result.

Empirical evidence might be obtained through experiments that seek to provide a measurable or observable reaction, trials that repeat an experiment to test its efficacy (such as a drug trial, for instance) or other forms of data gathering against which a hypothesis can be tested and reliably measured. 

"If a statement is about something that is itself observable, then the empirical testing can be direct. We just have a look to see if it is true. For example, the statement, 'The litmus paper is pink', is subject to direct empirical testing," wrote Peter Kosso in " A Summary of Scientific Method " (Springer, 2011).

"Science is most interesting and most useful to us when it is describing the unobservable things like atoms , germs , black holes , gravity , the process of evolution as it happened in the past, and so on," wrote Kosso. Scientific theories , meaning theories about nature that are unobservable, cannot be proven by direct empirical testing, but they can be tested indirectly, according to Kosso. "The nature of this indirect evidence, and the logical relation between evidence and theory, are the crux of scientific method," wrote Kosso.

The scientific method begins with scientists forming questions, or hypotheses , and then acquiring the knowledge through observations and experiments to either support or disprove a specific theory. "Empirical" means "based on observation or experience," according to the Merriam-Webster Dictionary . Empirical research is the process of finding empirical evidence. Empirical data is the information that comes from the research.

Before any pieces of empirical data are collected, scientists carefully design their research methods to ensure the accuracy, quality and integrity of the data. If there are flaws in the way that empirical data is collected, the research will not be considered valid.

The scientific method often involves lab experiments that are repeated over and over, and these experiments result in quantitative data in the form of numbers and statistics. However, that is not the only process used for gathering information to support or refute a theory. 

This methodology mostly applies to the natural sciences. "The role of empirical experimentation and observation is negligible in mathematics compared to natural sciences such as psychology, biology or physics," wrote Mark Chang, an adjunct professor at Boston University, in " Principles of Scientific Methods " (Chapman and Hall, 2017).

"Empirical evidence includes measurements or data collected through direct observation or experimentation," said Jaime Tanner, a professor of biology at Marlboro College in Vermont. There are two research methods used to gather empirical measurements and data: qualitative and quantitative.

Qualitative research, often used in the social sciences, examines the reasons behind human behavior, according to the National Center for Biotechnology Information (NCBI) . It involves data that can be found using the human senses . This type of research is often done in the beginning of an experiment. "When combined with quantitative measures, qualitative study can give a better understanding of health related issues," wrote Dr. Sanjay Kalra for NCBI.

Quantitative research involves methods that are used to collect numerical data and analyze it using statistical methods, ."Quantitative research methods emphasize objective measurements and the statistical, mathematical, or numerical analysis of data collected through polls, questionnaires, and surveys, or by manipulating pre-existing statistical data using computational techniques," according to the LeTourneau University . This type of research is often used at the end of an experiment to refine and test the previous research.

Identifying empirical evidence in another researcher's experiments can sometimes be difficult. According to the Pennsylvania State University Libraries , there are some things one can look for when determining if evidence is empirical:

• Can the experiment be recreated and tested?
• Does the experiment have a statement about the methodology, tools and controls used?
• Is there a definition of the group or phenomena being studied?

The objective of science is that all empirical data that has been gathered through observation, experience and experimentation is without bias. The strength of any scientific research depends on the ability to gather and analyze empirical data in the most unbiased and controlled fashion possible. 

However, in the 1960s, scientific historian and philosopher Thomas Kuhn promoted the idea that scientists can be influenced by prior beliefs and experiences, according to the Center for the Study of Language and Information . 

— Amazing Black scientists

— Marie Curie: Facts and biography

— What is multiverse theory?

"Missing observations or incomplete data can also cause bias in data analysis, especially when the missing mechanism is not random," wrote Chang.

Because scientists are human and prone to error, empirical data is often gathered by multiple scientists who independently replicate experiments. This also guards against scientists who unconsciously, or in rare cases consciously, veer from the prescribed research parameters, which could skew the results.

The recording of empirical data is also crucial to the scientific method, as science can only be advanced if data is shared and analyzed. Peer review of empirical data is essential to protect against bad science, according to the University of California .

Empirical laws and scientific laws are often the same thing. "Laws are descriptions — often mathematical descriptions — of natural phenomenon," Peter Coppinger, associate professor of biology and biomedical engineering at the Rose-Hulman Institute of Technology, told Live Science. 

Empirical laws are scientific laws that can be proven or disproved using observations or experiments, according to the Merriam-Webster Dictionary . So, as long as a scientific law can be tested using experiments or observations, it is considered an empirical law.

Empirical, anecdotal and logical evidence should not be confused. They are separate types of evidence that can be used to try to prove or disprove and idea or claim.

Logical evidence is used proven or disprove an idea using logic. Deductive reasoning may be used to come to a conclusion to provide logical evidence. For example, "All men are mortal. Harold is a man. Therefore, Harold is mortal."

Anecdotal evidence consists of stories that have been experienced by a person that are told to prove or disprove a point. For example, many people have told stories about their alien abductions to prove that aliens exist. Often, a person's anecdotal evidence cannot be proven or disproven. 

There are some things in nature that science is still working to build evidence for, such as the hunt to explain consciousness .

Meanwhile, in other scientific fields, efforts are still being made to improve research methods, such as the plan by some psychologists to fix the science of psychology .

" A Summary of Scientific Method " by Peter Kosso (Springer, 2011)

"Empirical" Merriam-Webster Dictionary

" Principles of Scientific Methods " by Mark Chang (Chapman and Hall, 2017)

"Qualitative research" by Dr. Sanjay Kalra National Center for Biotechnology Information (NCBI)

"Quantitative Research and Analysis: Quantitative Methods Overview" LeTourneau University

"Empirical Research in the Social Sciences and Education" Pennsylvania State University Libraries

"Thomas Kuhn" Center for the Study of Language and Information

"Misconceptions about science" University of California

Sign up for the Live Science daily newsletter now

Get the world’s most fascinating discoveries delivered straight to your inbox.

Live Science x HowTheLightGetsIn — Get discounted tickets to the world’s largest ideas and music festival

Do opposites really attract in relationships?

Boeing Starliner will return from space without a crew, NASA announces in long-awaited decision

Most Popular

• 2 'Enhancing' future generations with CRISPR is a road to a 'new eugenics,' says ethicist Rosemarie Garland-Thomson
• 3 'Who are we to say they shouldn't exist?': Dr. Neal Baer on the threat of CRISPR-driven eugenics
• 4 'Unbreakable' quantum communication closer to reality thanks to new, exceptionally bright photons
• 5 Polar Ignite 3 Titanium review

• Ask a Librarian

Research: Overview & Approaches

• Getting Started with Undergraduate Research
• Planning & Getting Started
• Building Your Knowledge Base
• Locating Sources
• Reading Scholarly Articles
• Creating a Literature Review
• Productivity & Organizing Research
• Scholarly and Professional Relationships

Introduction to Empirical Research

Databases for finding empirical research, guided search, google scholar, examples of empirical research, sources and further reading.

• Interpretive Research
• Action-Based Research
• Creative & Experimental Approaches

Your Librarian

• Introductory Video This video covers what empirical research is, what kinds of questions and methods empirical researchers use, and some tips for finding empirical research articles in your discipline.

• Guided Search: Finding Empirical Research Articles This is a hands-on tutorial that will allow you to use your own search terms to find resources.

• Study on radiation transfer in human skin for cosmetics
• Long-Term Mobile Phone Use and the Risk of Vestibular Schwannoma: A Danish Nationwide Cohort Study
• Emissions Impacts and Benefits of Plug-In Hybrid Electric Vehicles and Vehicle-to-Grid Services
• Review of design considerations and technological challenges for successful development and deployment of plug-in hybrid electric vehicles
• Endocrine disrupters and human health: could oestrogenic chemicals in body care cosmetics adversely affect breast cancer incidence in women?

• << Previous: Scholarly and Professional Relationships
• Next: Interpretive Research >>
• Last Updated: Aug 13, 2024 12:18 PM
• URL: https://guides.lib.purdue.edu/research_approaches

Empirical Research

• Living reference work entry
• First Online: 22 May 2017
• Cite this living reference work entry

• Emeka Thaddues Njoku 2  

544 Accesses

1 Citations

The term “empirical” entails gathered data based on experience, observations, or experimentation. In empirical research, knowledge is developed from factual experience as opposed to theoretical assumption and usually involved the use of data sources like datasets or fieldwork, but can also be based on observations within a laboratory setting. Testing hypothesis or answering definite questions is a primary feature of empirical research. Empirical research, in other words, involves the process of employing working hypothesis that are tested through experimentation or observation. Hence, empirical research is a method of uncovering empirical evidence.

Through the process of gathering valid empirical data, scientists from a variety of fields, ranging from the social to the natural sciences, have to carefully design their methods. This helps to ensure quality and accuracy of data collection and treatment. However, any error in empirical data collection process could inevitably render such...

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Bibliography

Bhattacherjee, A. (2012). Social science research: Principles, methods, and practices. Textbooks Collection . Book 3.

Google Scholar  

Comte, A., & Bridges, J. H. (Tr.) (1865). A general view of positivism . Trubner and Co. (reissued by Cambridge University Press , 2009).

Dilworth, C. B. (1982). Empirical research in the literature class. English Journal, 71 (3), 95–97.

Article   Google Scholar  

Heisenberg, W. (1971). Positivism, metaphysics and religion. In R. N. Nanshen (Ed.), Werner Heisenberg – Physics and beyond – Encounters and conversations , World Perspectives. 42. Translator: Arnold J. Pomerans. New York: Harper and Row.

Hossain, F. M. A. (2014). A critical analysis of empiricism. Open Journal of Philosophy, 2014 (4), 225–230.

Kant, I. (1783). Prolegomena to any future metaphysic (trans: Bennett, J.). Early Modern Texts. www.earlymoderntexts.com

Koch, S. (1992). Psychology’s Bridgman vs. Bridgman’s Bridgman: An essay in reconstruction. Theory and Psychology, 2 (3), 261–290.

Matin, A. (1968). An outline of philosophy . Dhaka: Mullick Brothers.

Mcleod, S. (2008). Psychology as science. http://www.simplypsychology.org/science-psychology.html

Popper, K. (1963). Conjectures and refutations: The growth of scientific knowledge . London: Routledge.

Simmel, G. (1908). The problem areas of sociology in Kurt H. Wolf: The sociology of Georg Simmel . London: The Free Press.

Weber, M. (1991). The nature of social action. In W. G. Runciman (Ed.), Weber: Selections in translation . Cambridge: Cambridge University Press.

Download references

Author information

Authors and affiliations.

Department of Political Science, University of Ibadan, Ibadan, Oyo, 200284, Nigeria

Emeka Thaddues Njoku

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Emeka Thaddues Njoku .

Editor information

Editors and affiliations.

Rhinebeck, New York, USA

David A. Leeming

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer-Verlag GmbH Germany

About this entry

Cite this entry.

Njoku, E.T. (2017). Empirical Research. In: Leeming, D. (eds) Encyclopedia of Psychology and Religion. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-27771-9_200051-1

Download citation

DOI : https://doi.org/10.1007/978-3-642-27771-9_200051-1

Received : 01 April 2017

Accepted : 08 May 2017

Published : 22 May 2017

Publisher Name : Springer, Berlin, Heidelberg

Print ISBN : 978-3-642-27771-9

Online ISBN : 978-3-642-27771-9

eBook Packages : Springer Reference Behavioral Science and Psychology Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Penn State University Libraries

Empirical research in the social sciences and education.

• What is Empirical Research and How to Read It
• Finding Empirical Research in Library Databases
• Designing Empirical Research
• Ethics, Cultural Responsiveness, and Anti-Racism in Research
• Citing, Writing, and Presenting Your Work

Contact the Librarian at your campus for more help!

Introduction: What is Empirical Research?

Empirical research is based on observed and measured phenomena and derives knowledge from actual experience rather than from theory or belief. 

How do you know if a study is empirical? Read the subheadings within the article, book, or report and look for a description of the research "methodology."  Ask yourself: Could I recreate this study and test these results?

Key characteristics to look for:

• Specific research questions to be answered
• Definition of the population, behavior, or phenomena being studied
• Description of the process used to study this population or phenomena, including selection criteria, controls, and testing instruments (such as surveys)

Another hint: some scholarly journals use a specific layout, called the "IMRaD" format, to communicate empirical research findings. Such articles typically have 4 components:

• Introduction: sometimes called "literature review" -- what is currently known about the topic -- usually includes a theoretical framework and/or discussion of previous studies
• Methodology: sometimes called "research design" -- how to recreate the study -- usually describes the population, research process, and analytical tools used in the present study
• Results: sometimes called "findings" -- what was learned through the study -- usually appears as statistical data or as substantial quotations from research participants
• Discussion: sometimes called "conclusion" or "implications" -- why the study is important -- usually describes how the research results influence professional practices or future studies

Reading and Evaluating Scholarly Materials

Reading research can be a challenge. However, the tutorials and videos below can help. They explain what scholarly articles look like, how to read them, and how to evaluate them:

• CRAAP Checklist A frequently-used checklist that helps you examine the currency, relevance, authority, accuracy, and purpose of an information source.
• IF I APPLY A newer model of evaluating sources which encourages you to think about your own biases as a reader, as well as concerns about the item you are reading.
• Credo Video: How to Read Scholarly Materials (4 min.)
• Credo Tutorial: How to Read Scholarly Materials
• Credo Tutorial: Evaluating Information
• Credo Video: Evaluating Statistics (4 min.)
• Credo Tutorial: Evaluating for Diverse Points of View
• Next: Finding Empirical Research in Library Databases >>
• Last Updated: Aug 13, 2024 3:16 PM
• URL: https://guides.libraries.psu.edu/emp

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Published: 01 June 2023

Data, measurement and empirical methods in the science of science

• Lu Liu 1 , 2 , 3 , 4 ,
• Benjamin F. Jones   ORCID: orcid.org/0000-0001-9697-9388 1 , 2 , 3 , 5 , 6 ,
• Brian Uzzi   ORCID: orcid.org/0000-0001-6855-2854 1 , 2 , 3 &
• Dashun Wang   ORCID: orcid.org/0000-0002-7054-2206 1 , 2 , 3 , 7  

Nature Human Behaviour volume  7 ,  pages 1046–1058 ( 2023 ) Cite this article

20k Accesses

18 Citations

116 Altmetric

Metrics details

• Scientific community

The advent of large-scale datasets that trace the workings of science has encouraged researchers from many different disciplinary backgrounds to turn scientific methods into science itself, cultivating a rapidly expanding ‘science of science’. This Review considers this growing, multidisciplinary literature through the lens of data, measurement and empirical methods. We discuss the purposes, strengths and limitations of major empirical approaches, seeking to increase understanding of the field’s diverse methodologies and expand researchers’ toolkits. Overall, new empirical developments provide enormous capacity to test traditional beliefs and conceptual frameworks about science, discover factors associated with scientific productivity, predict scientific outcomes and design policies that facilitate scientific progress.

Similar content being viewed by others

SciSciNet: A large-scale open data lake for the science of science research

A dataset for measuring the impact of research data and their curation

Envisioning a “science diplomacy 2.0”: on data, global challenges, and multi-layered networks

Scientific advances are a key input to rising standards of living, health and the capacity of society to confront grand challenges, from climate change to the COVID-19 pandemic 1 , 2 , 3 . A deeper understanding of how science works and where innovation occurs can help us to more effectively design science policy and science institutions, better inform scientists’ own research choices, and create and capture enormous value for science and humanity. Building on these key premises, recent years have witnessed substantial development in the ‘science of science’ 4 , 5 , 6 , 7 , 8 , 9 , which uses large-scale datasets and diverse computational toolkits to unearth fundamental patterns behind scientific production and use.

The idea of turning scientific methods into science itself is long-standing. Since the mid-20th century, researchers from different disciplines have asked central questions about the nature of scientific progress and the practice, organization and impact of scientific research. Building on these rich historical roots, the field of the science of science draws upon many disciplines, ranging from information science to the social, physical and biological sciences to computer science, engineering and design. The science of science closely relates to several strands and communities of research, including metascience, scientometrics, the economics of science, research on research, science and technology studies, the sociology of science, metaknowledge and quantitative science studies 5 . There are noticeable differences between some of these communities, mostly around their historical origins and the initial disciplinary composition of researchers forming these communities. For example, metascience has its origins in the clinical sciences and psychology, and focuses on rigour, transparency, reproducibility and other open science-related practices and topics. The scientometrics community, born in library and information sciences, places a particular emphasis on developing robust and responsible measures and indicators for science. Science and technology studies engage the history of science and technology, the philosophy of science, and the interplay between science, technology and society. The science of science, which has its origins in physics, computer science and sociology, takes a data-driven approach and emphasizes questions on how science works. Each of these communities has made fundamental contributions to understanding science. While they differ in their origins, these differences pale in comparison to the overarching, common interest in understanding the practice of science and its societal impact.

Three major developments have encouraged rapid advances in the science of science. The first is in data 9 : modern databases include millions of research articles, grant proposals, patents and more. This windfall of data traces scientific activity in remarkable detail and at scale. The second development is in measurement: scholars have used data to develop many new measures of scientific activities and examine theories that have long been viewed as important but difficult to quantify. The third development is in empirical methods: thanks to parallel advances in data science, network science, artificial intelligence and econometrics, researchers can study relationships, make predictions and assess science policy in powerful new ways. Together, new data, measurements and methods have revealed fundamental new insights about the inner workings of science and scientific progress itself.

With multiple approaches, however, comes a key challenge. As researchers adhere to norms respected within their disciplines, their methods vary, with results often published in venues with non-overlapping readership, fragmenting research along disciplinary boundaries. This fragmentation challenges researchers’ ability to appreciate and understand the value of work outside of their own discipline, much less to build directly on it for further investigations.

Recognizing these challenges and the rapidly developing nature of the field, this paper reviews the empirical approaches that are prevalent in this literature. We aim to provide readers with an up-to-date understanding of the available datasets, measurement constructs and empirical methodologies, as well as the value and limitations of each. Owing to space constraints, this Review does not cover the full technical details of each method, referring readers to related guides to learn more. Instead, we will emphasize why a researcher might favour one method over another, depending on the research question.

Beyond a positive understanding of science, a key goal of the science of science is to inform science policy. While this Review mainly focuses on empirical approaches, with its core audience being researchers in the field, the studies reviewed are also germane to key policy questions. For example, what is the appropriate scale of scientific investment, in what directions and through what institutions 10 , 11 ? Are public investments in science aligned with public interests 12 ? What conditions produce novel or high-impact science 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 ? How do the reward systems of science influence the rate and direction of progress 13 , 21 , 22 , 23 , 24 , and what governs scientific reproducibility 25 , 26 , 27 ? How do contributions evolve over a scientific career 28 , 29 , 30 , 31 , 32 , and how may diversity among scientists advance scientific progress 33 , 34 , 35 , among other questions relevant to science policy 36 , 37 .

Overall, this review aims to facilitate entry to science of science research, expand researcher toolkits and illustrate how diverse research approaches contribute to our collective understanding of science. Section 2 reviews datasets and data linkages. Section 3 reviews major measurement constructs in the science of science. Section 4 considers a range of empirical methods, focusing on one study to illustrate each method and briefly summarizing related examples and applications. Section 5 concludes with an outlook for the science of science.

Historically, data on scientific activities were difficult to collect and were available in limited quantities. Gathering data could involve manually tallying statistics from publications 38 , 39 , interviewing scientists 16 , 40 , or assembling historical anecdotes and biographies 13 , 41 . Analyses were typically limited to a specific domain or group of scientists. Today, massive datasets on scientific production and use are at researchers’ fingertips 42 , 43 , 44 . Armed with big data and advanced algorithms, researchers can now probe questions previously not amenable to quantification and with enormous increases in scope and scale, as detailed below.

Publication datasets cover papers from nearly all scientific disciplines, enabling analyses of both general and domain-specific patterns. Commonly used datasets include the Web of Science (WoS), PubMed, CrossRef, ORCID, OpenCitations, Dimensions and OpenAlex. Datasets incorporating papers’ text (CORE) 45 , 46 , 47 , data entities (DataCite) 48 , 49 and peer review reports (Publons) 33 , 50 , 51 have also become available. These datasets further enable novel measurement, for example, representations of a paper’s content 52 , 53 , novelty 15 , 54 and interdisciplinarity 55 .

Notably, databases today capture more diverse aspects of science beyond publications, offering a richer and more encompassing view of research contexts and of researchers themselves (Fig. 1 ). For example, some datasets trace research funding to the specific publications these investments support 56 , 57 , allowing high-scale studies of the impact of funding on productivity and the return on public investment. Datasets incorporating job placements 58 , 59 , curriculum vitae 21 , 59 and scientific prizes 23 offer rich quantitative evidence on the social structure of science. Combining publication profiles with mentorship genealogies 60 , 61 , dissertations 34 and course syllabi 62 , 63 provides insights on mentoring and cultivating talent.

This figure presents commonly used data types in science of science research, information contained in each data type and examples of data sources. Datasets in the science of science research have not only grown in scale but have also expanded beyond publications to integrate upstream funding investments and downstream applications that extend beyond science itself.

Finally, today’s scope of data extends beyond science to broader aspects of society. Altmetrics 64 captures news media and social media mentions of scientific articles. Other databases incorporate marketplace uses of science, including through patents 10 , pharmaceutical clinical trials and drug approvals 65 , 66 . Policy documents 67 , 68 help us to understand the role of science in the halls of government 69 and policy making 12 , 68 .

While datasets of the modern scientific enterprise have grown exponentially, they are not without limitations. As is often the case for data-driven research, drawing conclusions from specific data sources requires scrutiny and care. Datasets are typically based on published work, which may favour easy-to-publish topics over important ones (the streetlight effect) 70 , 71 . The publication of negative results is also rare (the file drawer problem) 72 , 73 . Meanwhile, English language publications account for over 90% of articles in major data sources, with limited coverage of non-English journals 74 . Publication datasets may also reflect biases in data collection across research institutions or demographic groups. Despite the open science movement, many datasets require paid subscriptions, which can create inequality in data access. Creating more open datasets for the science of science, such as OpenAlex, may not only improve the robustness and replicability of empirical claims but also increase entry to the field.

As today’s datasets become larger in scale and continue to integrate new dimensions, they offer opportunities to unveil the inner workings and external impacts of science in new ways. They can enable researchers to reach beyond previous limitations while conducting original studies of new and long-standing questions about the sciences.

Measurement

Here we discuss prominent measurement approaches in the science of science, including their purposes and limitations.

Modern publication databases typically include data on which articles and authors cite other papers and scientists. These citation linkages have been used to engage core conceptual ideas in scientific research. Here we consider two common measures based on citation information: citation counts and knowledge flows.

First, citation counts are commonly used indicators of impact. The term ‘indicator’ implies that it only approximates the concept of interest. A citation count is defined as how many times a document is cited by subsequent documents and can proxy for the importance of research papers 75 , 76 as well as patented inventions 77 , 78 , 79 . Rather than treating each citation equally, measures may further weight the importance of each citation, for example by using the citation network structure to produce centrality 80 , PageRank 81 , 82 or Eigenfactor indicators 83 , 84 .

Citation-based indicators have also faced criticism 84 , 85 . Citation indicators necessarily oversimplify the construct of impact, often ignoring heterogeneity in the meaning and use of a particular reference, the variations in citation practices across fields and institutional contexts, and the potential for reputation and power structures in science to influence citation behaviour 86 , 87 . Researchers have started to understand more nuanced citation behaviours ranging from negative citations 86 to citation context 47 , 88 , 89 . Understanding what a citation actually measures matters in interpreting and applying many research findings in the science of science. Evaluations relying on citation-based indicators rather than expert judgements raise questions regarding misuse 90 , 91 , 92 . Given the importance of developing indicators that can reliably quantify and evaluate science, the scientometrics community has been working to provide guidance for responsible citation practices and assessment 85 .

Second, scientists use citations to trace knowledge flows. Each citation in a paper is a link to specific previous work from which we can proxy how new discoveries draw upon existing ideas 76 , 93 and how knowledge flows between fields of science 94 , 95 , research institutions 96 , regions and nations 97 , 98 , 99 , and individuals 81 . Combinations of citation linkages can also approximate novelty 15 , disruptiveness 17 , 100 and interdisciplinarity 55 , 95 , 101 , 102 . A rapidly expanding body of work further examines citations to scientific articles from other domains (for example, patents, clinical drug trials and policy documents) to understand the applied value of science 10 , 12 , 65 , 66 , 103 , 104 , 105 .

Individuals

Analysing individual careers allows researchers to answer questions such as: How do we quantify individual scientific productivity? What is a typical career lifecycle? How are resources and credits allocated across individuals and careers? A scholar’s career can be examined through the papers they publish 30 , 31 , 106 , 107 , 108 , with attention to career progression and mobility, publication counts and citation impact, as well as grant funding 24 , 109 , 110 and prizes 111 , 112 , 113 ,

Studies of individual impact focus on output, typically approximated by the number of papers a researcher publishes and citation indicators. A popular measure for individual impact is the h -index 114 , which takes both volume and per-paper impact into consideration. Specifically, a scientist is assigned the largest value h such that they have h papers that were each cited at least h times. Later studies build on the idea of the h -index and propose variants to address limitations 115 , these variants ranging from emphasizing highly cited papers in a career 116 , to field differences 117 and normalizations 118 , to the relative contribution of an individual in collaborative works 119 .

To study dynamics in output over the lifecycle, individuals can be studied according to age, career age or the sequence of publications. A long-standing literature has investigated the relationship between age and the likelihood of outstanding achievement 28 , 106 , 111 , 120 , 121 . Recent studies further decouple the relationship between age, publication volume and per-paper citation, and measure the likelihood of producing highly cited papers in the sequence of works one produces 30 , 31 .

As simple as it sounds, representing careers using publication records is difficult. Collecting the full publication list of a researcher is the foundation to study individuals yet remains a key challenge, requiring name disambiguation techniques to match specific works to specific researchers. Although algorithms are increasingly capable at identifying millions of career profiles 122 , they vary in accuracy and robustness. ORCID can help to alleviate the problem by offering researchers the opportunity to create, maintain and update individual profiles themselves, and it goes beyond publications to collect broader outputs and activities 123 . A second challenge is survivorship bias. Empirical studies tend to focus on careers that are long enough to afford statistical analyses, which limits the applicability of the findings to scientific careers as a whole. A third challenge is the breadth of scientists’ activities, where focusing on publications ignores other important contributions such as mentorship and teaching, service (for example, refereeing papers, reviewing grant proposals and editing journals) or leadership within their organizations. Although researchers have begun exploring these dimensions by linking individual publication profiles with genealogical databases 61 , 124 , dissertations 34 , grants 109 , curriculum vitae 21 and acknowledgements 125 , scientific careers beyond publication records remain under-studied 126 , 127 . Lastly, citation-based indicators only serve as an approximation of individual performance with similar limitations as discussed above. The scientific community has called for more appropriate practices 85 , 128 , ranging from incorporating expert assessment of research contributions to broadening the measures of impact beyond publications.

Over many decades, science has exhibited a substantial and steady shift away from solo authorship towards coauthorship, especially among highly cited works 18 , 129 , 130 . In light of this shift, a research field, the science of team science 131 , 132 , has emerged to study the mechanisms that facilitate or hinder the effectiveness of teams. Team size can be proxied by the number of coauthors on a paper, which has been shown to predict distinctive types of advance: whereas larger teams tend to develop ideas, smaller teams tend to disrupt current ways of thinking 17 . Team characteristics can be inferred from coauthors’ backgrounds 133 , 134 , 135 , allowing quantification of a team’s diversity in terms of field, age, gender or ethnicity. Collaboration networks based on coauthorship 130 , 136 , 137 , 138 , 139 offer nuanced network-based indicators to understand individual and institutional collaborations.

However, there are limitations to using coauthorship alone to study teams 132 . First, coauthorship can obscure individual roles 140 , 141 , 142 , which has prompted institutional responses to help to allocate credit, including authorship order and individual contribution statements 56 , 143 . Second, coauthorship does not reflect the complex dynamics and interactions between team members that are often instrumental for team success 53 , 144 . Third, collaborative contributions can extend beyond coauthorship in publications to include members of a research laboratory 145 or co-principal investigators (co-PIs) on a grant 146 . Initiatives such as CRediT may help to address some of these issues by recording detailed roles for each contributor 147 .

Institutions

Research institutions, such as departments, universities, national laboratories and firms, encompass wider groups of researchers and their corresponding outputs. Institutional membership can be inferred from affiliations listed on publications or patents 148 , 149 , and the output of an institution can be aggregated over all its affiliated researchers 150 . Institutional research information systems (CRIS) contain more comprehensive research outputs and activities from employees.

Some research questions consider the institution as a whole, investigating the returns to research and development investment 104 , inequality of resource allocation 22 and the flow of scientists 21 , 148 , 149 . Other questions focus on institutional structures as sources of research productivity by looking into the role of peer effects 125 , 151 , 152 , 153 , how institutional policies impact research outcomes 154 , 155 and whether interdisciplinary efforts foster innovation 55 . Institution-oriented measurement faces similar limitations as with analyses of individuals and teams, including name disambiguation for a given institution and the limited capacity of formal publication records to characterize the full range of relevant institutional outcomes. It is also unclear how to allocate credit among multiple institutions associated with a paper. Moreover, relevant institutional employees extend beyond publishing researchers: interns, technicians and administrators all contribute to research endeavours 130 .

In sum, measurements allow researchers to quantify scientific production and use across numerous dimensions, but they also raise questions of construct validity: Does the proposed metric really reflect what we want to measure? Testing the construct’s validity is important, as is understanding a construct’s limits. Where possible, using alternative measurement approaches, or qualitative methods such as interviews and surveys, can improve measurement accuracy and the robustness of findings.

Empirical methods

In this section, we review two broad categories of empirical approaches (Table 1 ), each with distinctive goals: (1) to discover, estimate and predict empirical regularities; and (2) to identify causal mechanisms. For each method, we give a concrete example to help to explain how the method works, summarize related work for interested readers, and discuss contributions and limitations.

Descriptive and predictive approaches

Empirical regularities and generalizable facts.

The discovery of empirical regularities in science has had a key role in driving conceptual developments and the directions of future research. By observing empirical patterns at scale, researchers unveil central facts that shape science and present core features that theories of scientific progress and practice must explain. For example, consider citation distributions. de Solla Price first proposed that citation distributions are fat-tailed 39 , indicating that a few papers have extremely high citations while most papers have relatively few or even no citations at all. de Solla Price proposed that citation distribution was a power law, while researchers have since refined this view to show that the distribution appears log-normal, a nearly universal regularity across time and fields 156 , 157 . The fat-tailed nature of citation distributions and its universality across the sciences has in turn sparked substantial theoretical work that seeks to explain this key empirical regularity 20 , 156 , 158 , 159 .

Empirical regularities are often surprising and can contest previous beliefs of how science works. For example, it has been shown that the age distribution of great achievements peaks in middle age across a wide range of fields 107 , 121 , 160 , rejecting the common belief that young scientists typically drive breakthroughs in science. A closer look at the individual careers also indicates that productivity patterns vary widely across individuals 29 . Further, a scholar’s highest-impact papers come at a remarkably constant rate across the sequence of their work 30 , 31 .

The discovery of empirical regularities has had important roles in shaping beliefs about the nature of science 10 , 45 , 161 , 162 , sources of breakthrough ideas 15 , 163 , 164 , 165 , scientific careers 21 , 29 , 126 , 127 , the network structure of ideas and scientists 23 , 98 , 136 , 137 , 138 , 139 , 166 , gender inequality 57 , 108 , 126 , 135 , 143 , 167 , 168 , and many other areas of interest to scientists and science institutions 22 , 47 , 86 , 97 , 102 , 105 , 134 , 169 , 170 , 171 . At the same time, care must be taken to ensure that findings are not merely artefacts due to data selection or inherent bias. To differentiate meaningful patterns from spurious ones, it is important to stress test the findings through different selection criteria or across non-overlapping data sources.

Regression analysis

When investigating correlations among variables, a classic method is regression, which estimates how one set of variables explains variation in an outcome of interest. Regression can be used to test explicit hypotheses or predict outcomes. For example, researchers have investigated whether a paper’s novelty predicts its citation impact 172 . Adding additional control variables to the regression, one can further examine the robustness of the focal relationship.

Although regression analysis is useful for hypothesis testing, it bears substantial limitations. If the question one wishes to ask concerns a ‘causal’ rather than a correlational relationship, regression is poorly suited to the task as it is impossible to control for all the confounding factors. Failing to account for such ‘omitted variables’ can bias the regression coefficient estimates and lead to spurious interpretations. Further, regression models often have low goodness of fit (small R 2 ), indicating that the variables considered explain little of the outcome variation. As regressions typically focus on a specific relationship in simple functional forms, regressions tend to emphasize interpretability rather than overall predictability. The advent of predictive approaches powered by large-scale datasets and novel computational techniques offers new opportunities for modelling complex relationships with stronger predictive power.

Mechanistic models

Mechanistic modelling is an important approach to explaining empirical regularities, drawing from methods primarily used in physics. Such models predict macro-level regularities of a system by modelling micro-level interactions among basic elements with interpretable and modifiable formulars. While theoretical by nature, mechanistic models in the science of science are often empirically grounded, and this approach has developed together with the advent of large-scale, high-resolution data.

Simplicity is the core value of a mechanistic model. Consider for example, why citations follow a fat-tailed distribution. de Solla Price modelled the citing behaviour as a cumulative advantage process on a growing citation network 159 and found that if the probability a paper is cited grows linearly with its existing citations, the resulting distribution would follow a power law, broadly aligned with empirical observations. The model is intentionally simplified, ignoring myriad factors. Yet the simple cumulative advantage process is by itself sufficient in explaining a power law distribution of citations. In this way, mechanistic models can help to reveal key mechanisms that can explain observed patterns.

Moreover, mechanistic models can be refined as empirical evidence evolves. For example, later investigations showed that citation distributions are better characterized as log-normal 156 , 173 , prompting researchers to introduce a fitness parameter to encapsulate the inherent differences in papers’ ability to attract citations 174 , 175 . Further, older papers are less likely to be cited than expected 176 , 177 , 178 , motivating more recent models 20 to introduce an additional aging effect 179 . By combining the cumulative advantage, fitness and aging effects, one can already achieve substantial predictive power not just for the overall properties of the system but also the citation dynamics of individual papers 20 .

In addition to citations, mechanistic models have been developed to understand the formation of collaborations 136 , 180 , 181 , 182 , 183 , knowledge discovery and diffusion 184 , 185 , topic selection 186 , 187 , career dynamics 30 , 31 , 188 , 189 , the growth of scientific fields 190 and the dynamics of failure in science and other domains 178 .

At the same time, some observers have argued that mechanistic models are too simplistic to capture the essence of complex real-world problems 191 . While it has been a cornerstone for the natural sciences, representing social phenomena in a limited set of mathematical equations may miss complexities and heterogeneities that make social phenomena interesting in the first place. Such concerns are not unique to the science of science, as they represent a broader theme in computational social sciences 192 , 193 , ranging from social networks 194 , 195 to human mobility 196 , 197 to epidemics 198 , 199 . Other observers have questioned the practical utility of mechanistic models and whether they can be used to guide decisions and devise actionable policies. Nevertheless, despite these limitations, several complex phenomena in the science of science are well captured by simple mechanistic models, showing a high degree of regularity beneath complex interacting systems and providing powerful insights about the nature of science. Mixing such modelling with other methods could be particularly fruitful in future investigations.

Machine learning

The science of science seeks in part to forecast promising directions for scientific research 7 , 44 . In recent years, machine learning methods have substantially advanced predictive capabilities 200 , 201 and are playing increasingly important parts in the science of science. In contrast to the previous methods, machine learning does not emphasize hypotheses or theories. Rather, it leverages complex relationships in data and optimizes goodness of fit to make predictions and categorizations.

Traditional machine learning models include supervised, semi-supervised and unsupervised learning. The model choice depends on data availability and the research question, ranging from supervised models for citation prediction 202 , 203 to unsupervised models for community detection 204 . Take for example mappings of scientific knowledge 94 , 205 , 206 . The unsupervised method applies network clustering algorithms to map the structures of science. Related visualization tools make sense of clusters from the underlying network, allowing observers to see the organization, interactions and evolution of scientific knowledge. More recently, supervised learning, and deep neural networks in particular, have witnessed especially rapid developments 207 . Neural networks can generate high-dimensional representations of unstructured data such as images and texts, which encode complex properties difficult for human experts to perceive.

Take text analysis as an example. A recent study 52 utilizes 3.3 million paper abstracts in materials science to predict the thermoelectric properties of materials. The intuition is that the words currently used to describe a material may predict its hitherto undiscovered properties (Fig. 2 ). Compared with a random material, the materials predicted by the model are eight times more likely to be reported as thermoelectric in the next 5 years, suggesting that machine learning has the potential to substantially speed up knowledge discovery, especially as data continue to grow in scale and scope. Indeed, predicting the direction of new discoveries represents one of the most promising avenues for machine learning models, with neural networks being applied widely to biology 208 , physics 209 , 210 , mathematics 211 , chemistry 212 , medicine 213 and clinical applications 214 . Neural networks also offer a quantitative framework to probe the characteristics of creative products ranging from scientific papers 53 , journals 215 , organizations 148 , to paintings and movies 32 . Neural networks can also help to predict the reproducibility of papers from a variety of disciplines at scale 53 , 216 .

This figure illustrates the word2vec skip-gram methods 52 , where the goal is to predict useful properties of materials using previous scientific literature. a , The architecture and training process of the word2vec skip-gram model, where the 3-layer, fully connected neural network learns the 200-dimensional representation (hidden layer) from the sparse vector for each word and its context in the literature (input layer). b , The top two principal components of the word embedding. Materials with similar features are close in the 2D space, allowing prediction of a material’s properties. Different targeted words are shown in different colours. Reproduced with permission from ref. 52 , Springer Nature Ltd.

While machine learning can offer high predictive accuracy, successful applications to the science of science face challenges, particularly regarding interpretability. Researchers may value transparent and interpretable findings for how a given feature influences an outcome, rather than a black-box model. The lack of interpretability also raises concerns about bias and fairness. In predicting reproducible patterns from data, machine learning models inevitably include and reproduce biases embedded in these data, often in non-transparent ways. The fairness of machine learning 217 is heavily debated in applications ranging from the criminal justice system to hiring processes. Effective and responsible use of machine learning in the science of science therefore requires thoughtful partnership between humans and machines 53 to build a reliable system accessible to scrutiny and modification.

Causal approaches

The preceding methods can reveal core facts about the workings of science and develop predictive capacity. Yet, they fail to capture causal relationships, which are particularly useful in assessing policy interventions. For example, how can we test whether a science policy boosts or hinders the performance of individuals, teams or institutions? The overarching idea of causal approaches is to construct some counterfactual world where two groups are identical to each other except that one group experiences a treatment that the other group does not.

Towards causation

Before engaging in causal approaches, it is useful to first consider the interpretative challenges of observational data. As observational data emerge from mechanisms that are not fully known or measured, an observed correlation may be driven by underlying forces that were not accounted for in the analysis. This challenge makes causal inference fundamentally difficult in observational data. An awareness of this issue is the first step in confronting it. It further motivates intermediate empirical approaches, including the use of matching strategies and fixed effects, that can help to confront (although not fully eliminate) the inference challenge. We first consider these approaches before turning to more fully causal methods.

Matching. Matching utilizes rich information to construct a control group that is similar to the treatment group on as many observable characteristics as possible before the treatment group is exposed to the treatment. Inferences can then be made by comparing the treatment and the matched control groups. Exact matching applies to categorical values, such as country, gender, discipline or affiliation 35 , 218 . Coarsened exact matching considers percentile bins of continuous variables and matches observations in the same bin 133 . Propensity score matching estimates the probability of receiving the ‘treatment’ on the basis of the controlled variables and uses the estimates to match treatment and control groups, which reduces the matching task from comparing the values of multiple covariates to comparing a single value 24 , 219 . Dynamic matching is useful for longitudinally matching variables that change over time 220 , 221 .

Fixed effects. Fixed effects are a powerful and now standard tool in controlling for confounders. A key requirement for using fixed effects is that there are multiple observations on the same subject or entity (person, field, institution and so on) 222 , 223 , 224 . The fixed effect works as a dummy variable that accounts for the role of any fixed characteristic of that entity. Consider the finding where gender-diverse teams produce higher-impact papers than same-gender teams do 225 . A confounder may be that individuals who tend to write high-impact papers may also be more likely to work in gender-diverse teams. By including individual fixed effects, one accounts for any fixed characteristics of individuals (such as IQ, cultural background or previous education) that might drive the relationship of interest.

In sum, matching and fixed effects methods reduce potential sources of bias in interpreting relationships between variables. Yet, confounders may persist in these studies. For instance, fixed effects do not control for unobserved factors that change with time within the given entity (for example, access to funding or new skills). Identifying casual effects convincingly will then typically require distinct research methods that we turn to next.

Quasi-experiments

Researchers in economics and other fields have developed a range of quasi-experimental methods to construct treatment and control groups. The key idea here is exploiting randomness from external events that differentially expose subjects to a particular treatment. Here we review three quasi-experimental methods: difference-in-differences, instrumental variables and regression discontinuity (Fig. 3 ).

a – c , This figure presents illustrations of ( a ) differences-in-differences, ( b ) instrumental variables and ( c ) regression discontinuity methods. The solid line in b represents causal links and the dashed line represents the relationships that are not allowed, if the IV method is to produce causal inference.

Difference-in-differences. Difference-in-difference regression (DiD) investigates the effect of an unexpected event, comparing the affected group (the treated group) with an unaffected group (the control group). The control group is intended to provide the counterfactual path—what would have happened were it not for the unexpected event. Ideally, the treated and control groups are on virtually identical paths before the treatment event, but DiD can also work if the groups are on parallel paths (Fig. 3a ). For example, one study 226 examines how the premature death of superstar scientists affects the productivity of their previous collaborators. The control group are collaborators of superstars who did not die in the time frame. The two groups do not show significant differences in publications before a death event, yet upon the death of a star scientist, the treated collaborators on average experience a 5–8% decline in their quality-adjusted publication rates compared with the control group. DiD has wide applicability in the science of science, having been used to analyse the causal effects of grant design 24 , access costs to previous research 155 , 227 , university technology transfer policies 154 , intellectual property 228 , citation practices 229 , evolution of fields 221 and the impacts of paper retractions 230 , 231 , 232 . The DiD literature has grown especially rapidly in the field of economics, with substantial recent refinements 233 , 234 .

Instrumental variables. Another quasi-experimental approach utilizes ‘instrumental variables’ (IV). The goal is to determine the causal influence of some feature X on some outcome Y by using a third, instrumental variable. This instrumental variable is a quasi-random event that induces variation in X and, except for its impact through X , has no other effect on the outcome Y (Fig. 3b ). For example, consider a study of astronomy that seeks to understand how telescope time affects career advancement 235 . Here, one cannot simply look at the correlation between telescope time and career outcomes because many confounds (such as talent or grit) may influence both telescope time and career opportunities. Now consider the weather as an instrumental variable. Cloudy weather will, at random, reduce an astronomer’s observational time. Yet, the weather on particular nights is unlikely to correlate with a scientist’s innate qualities. The weather can then provide an instrumental variable to reveal a causal relationship between telescope time and career outcomes. Instrumental variables have been used to study local peer effects in research 151 , the impact of gender composition in scientific committees 236 , patents on future innovation 237 and taxes on inventor mobility 238 .

Regression discontinuity. In regression discontinuity, policies with an arbitrary threshold for receiving some benefit can be used to construct treatment and control groups (Fig. 3c ). Take the funding paylines for grant proposals as an example. Proposals with scores increasingly close to the payline are increasingly similar in their both observable and unobservable characteristics, yet only those projects with scores above the payline receive the funding. For example, a study 110 examines the effect of winning an early-career grant on the probability of winning a later, mid-career grant. The probability has a discontinuous jump across the initial grant’s payline, providing the treatment and control groups needed to estimate the causal effect of receiving a grant. This example utilizes the ‘sharp’ regression discontinuity that assumes treatment status to be fully determined by the cut-off. If we assume treatment status is only partly determined by the cut-off, we can use ‘fuzzy’ regression discontinuity designs. Here the probability of receiving a grant is used to estimate the future outcome 11 , 110 , 239 , 240 , 241 .

Although quasi-experiments are powerful tools, they face their own limitations. First, these approaches identify causal effects within a specific context and often engage small numbers of observations. How representative the samples are for broader populations or contexts is typically left as an open question. Second, the validity of the causal design is typically not ironclad. Researchers usually conduct different robustness checks to verify whether observable confounders have significant differences between the treated and control groups, before treatment. However, unobservable features may still differ between treatment and control groups. The quality of instrumental variables and the specific claim that they have no effect on the outcome except through the variable of interest, is also difficult to assess. Ultimately, researchers must rely partly on judgement to tell whether appropriate conditions are met for causal inference.

This section emphasized popular econometric approaches to causal inference. Other empirical approaches, such as graphical causal modelling 242 , 243 , also represent an important stream of work on assessing causal relationships. Such approaches usually represent causation as a directed acyclic graph, with nodes as variables and arrows between them as suspected causal relationships. In the science of science, the directed acyclic graph approach has been applied to quantify the causal effect of journal impact factor 244 and gender or racial bias 245 on citations. Graphical causal modelling has also triggered discussions on strengths and weaknesses compared to the econometrics methods 246 , 247 .

Experiments

In contrast to quasi-experimental approaches, laboratory and field experiments conduct direct randomization in assigning treatment and control groups. These methods engage explicitly in the data generation process, manipulating interventions to observe counterfactuals. These experiments are crafted to study mechanisms of specific interest and, by designing the experiment and formally randomizing, can produce especially rigorous causal inference.

Laboratory experiments. Laboratory experiments build counterfactual worlds in well-controlled laboratory environments. Researchers randomly assign participants to the treatment or control group and then manipulate the laboratory conditions to observe different outcomes in the two groups. For example, consider laboratory experiments on team performance and gender composition 144 , 248 . The researchers randomly assign participants into groups to perform tasks such as solving puzzles or brainstorming. Teams with a higher proportion of women are found to perform better on average, offering evidence that gender diversity is causally linked to team performance. Laboratory experiments can allow researchers to test forces that are otherwise hard to observe, such as how competition influences creativity 249 . Laboratory experiments have also been used to evaluate how journal impact factors shape scientists’ perceptions of rewards 250 and gender bias in hiring 251 .

Laboratory experiments allow for precise control of settings and procedures to isolate causal effects of interest. However, participants may behave differently in synthetic environments than in real-world settings, raising questions about the generalizability and replicability of the results 252 , 253 , 254 . To assess causal effects in real-world settings, researcher use randomized controlled trials.

Randomized controlled trials. A randomized controlled trial (RCT), or field experiment, is a staple for causal inference across a wide range of disciplines. RCTs randomly assign participants into the treatment and control conditions 255 and can be used not only to assess mechanisms but also to test real-world interventions such as policy change. The science of science has witnessed growing use of RCTs. For instance, a field experiment 146 investigated whether lower search costs for collaborators increased collaboration in grant applications. The authors randomly allocated principal investigators to face-to-face sessions in a medical school, and then measured participants’ chance of writing a grant proposal together. RCTs have also offered rich causal insights on peer review 256 , 257 , 258 , 259 , 260 and gender bias in science 261 , 262 , 263 .

While powerful, RCTs are difficult to conduct in the science of science, mainly for two reasons. The first concerns potential risks in a policy intervention. For instance, while randomizing funding across individuals could generate crucial causal insights for funders, it may also inadvertently harm participants’ careers 264 . Second, key questions in the science of science often require a long-time horizon to trace outcomes, which makes RCTs costly. It also raises the difficulty of replicating findings. A relative advantage of the quasi-experimental methods discussed earlier is that one can identify causal effects over potentially long periods of time in the historical record. On the other hand, quasi-experiments must be found as opposed to designed, and they often are not available for many questions of interest. While the best approaches are context dependent, a growing community of researchers is building platforms to facilitate RCTs for the science of science, aiming to lower their costs and increase their scale. Performing RCTs in partnership with science institutions can also contribute to timely, policy-relevant research that may substantially improve science decision-making and investments.

Research in the science of science has been empowered by the growth of high-scale data, new measurement approaches and an expanding range of empirical methods. These tools provide enormous capacity to test conceptual frameworks about science, discover factors impacting scientific productivity, predict key scientific outcomes and design policies that better facilitate future scientific progress. A careful appreciation of empirical techniques can help researchers to choose effective tools for questions of interest and propel the field. A better and broader understanding of these methodologies may also build bridges across diverse research communities, facilitating communication and collaboration, and better leveraging the value of diverse perspectives. The science of science is about turning scientific methods on the nature of science itself. The fruits of this work, with time, can guide researchers and research institutions to greater progress in discovery and understanding across the landscape of scientific inquiry.

Bush, V . S cience–the Endless Frontier: A Report to the President on a Program for Postwar Scientific Research (National Science Foundation, 1990).

Mokyr, J. The Gifts of Athena (Princeton Univ. Press, 2011).

Jones, B. F. in Rebuilding the Post-Pandemic Economy (eds Kearney, M. S. & Ganz, A.) 272–310 (Aspen Institute Press, 2021).

Wang, D. & Barabási, A.-L. The Science of Science (Cambridge Univ. Press, 2021).

Fortunato, S. et al. Science of science. Science 359 , eaao0185 (2018).

Article   PubMed   PubMed Central   Google Scholar  

Azoulay, P. et al. Toward a more scientific science. Science 361 , 1194–1197 (2018).

Article   PubMed   Google Scholar  

Clauset, A., Larremore, D. B. & Sinatra, R. Data-driven predictions in the science of science. Science 355 , 477–480 (2017).

Article   CAS   PubMed   Google Scholar  

Zeng, A. et al. The science of science: from the perspective of complex systems. Phys. Rep. 714 , 1–73 (2017).

Article   Google Scholar  

Lin, Z., Yin. Y., Liu, L. & Wang, D. SciSciNet: a large-scale open data lake for the science of science research. Sci. Data, https://doi.org/10.1038/s41597-023-02198-9 (2023).

Ahmadpoor, M. & Jones, B. F. The dual frontier: patented inventions and prior scientific advance. Science 357 , 583–587 (2017).

Azoulay, P., Graff Zivin, J. S., Li, D. & Sampat, B. N. Public R&D investments and private-sector patenting: evidence from NIH funding rules. Rev. Econ. Stud. 86 , 117–152 (2019).

Yin, Y., Dong, Y., Wang, K., Wang, D. & Jones, B. F. Public use and public funding of science. Nat. Hum. Behav. 6 , 1344–1350 (2022).

Merton, R. K. The Sociology of Science: Theoretical and Empirical Investigations (Univ. Chicago Press, 1973).

Kuhn, T. The Structure of Scientific Revolutions (Princeton Univ. Press, 2021).

Uzzi, B., Mukherjee, S., Stringer, M. & Jones, B. Atypical combinations and scientific impact. Science 342 , 468–472 (2013).

Zuckerman, H. Scientific Elite: Nobel Laureates in the United States (Transaction Publishers, 1977).

Wu, L., Wang, D. & Evans, J. A. Large teams develop and small teams disrupt science and technology. Nature 566 , 378–382 (2019).

Wuchty, S., Jones, B. F. & Uzzi, B. The increasing dominance of teams in production of knowledge. Science 316 , 1036–1039 (2007).

Foster, J. G., Rzhetsky, A. & Evans, J. A. Tradition and innovation in scientists’ research strategies. Am. Sociol. Rev. 80 , 875–908 (2015).

Wang, D., Song, C. & Barabási, A.-L. Quantifying long-term scientific impact. Science 342 , 127–132 (2013).

Clauset, A., Arbesman, S. & Larremore, D. B. Systematic inequality and hierarchy in faculty hiring networks. Sci. Adv. 1 , e1400005 (2015).

Ma, A., Mondragón, R. J. & Latora, V. Anatomy of funded research in science. Proc. Natl Acad. Sci. USA 112 , 14760–14765 (2015).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Ma, Y. & Uzzi, B. Scientific prize network predicts who pushes the boundaries of science. Proc. Natl Acad. Sci. USA 115 , 12608–12615 (2018).

Azoulay, P., Graff Zivin, J. S. & Manso, G. Incentives and creativity: evidence from the academic life sciences. RAND J. Econ. 42 , 527–554 (2011).

Schor, S. & Karten, I. Statistical evaluation of medical journal manuscripts. JAMA 195 , 1123–1128 (1966).

Platt, J. R. Strong inference: certain systematic methods of scientific thinking may produce much more rapid progress than others. Science 146 , 347–353 (1964).

Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2 , e124 (2005).

Simonton, D. K. Career landmarks in science: individual differences and interdisciplinary contrasts. Dev. Psychol. 27 , 119 (1991).

Way, S. F., Morgan, A. C., Clauset, A. & Larremore, D. B. The misleading narrative of the canonical faculty productivity trajectory. Proc. Natl Acad. Sci. USA 114 , E9216–E9223 (2017).

Sinatra, R., Wang, D., Deville, P., Song, C. & Barabási, A.-L. Quantifying the evolution of individual scientific impact. Science 354 , aaf5239 (2016).

Liu, L. et al. Hot streaks in artistic, cultural, and scientific careers. Nature 559 , 396–399 (2018).

Liu, L., Dehmamy, N., Chown, J., Giles, C. L. & Wang, D. Understanding the onset of hot streaks across artistic, cultural, and scientific careers. Nat. Commun. 12 , 5392 (2021).

Squazzoni, F. et al. Peer review and gender bias: a study on 145 scholarly journals. Sci. Adv. 7 , eabd0299 (2021).

Hofstra, B. et al. The diversity–innovation paradox in science. Proc. Natl Acad. Sci. USA 117 , 9284–9291 (2020).

Huang, J., Gates, A. J., Sinatra, R. & Barabási, A.-L. Historical comparison of gender inequality in scientific careers across countries and disciplines. Proc. Natl Acad. Sci. USA 117 , 4609–4616 (2020).

Gläser, J. & Laudel, G. Governing science: how science policy shapes research content. Eur. J. Sociol. 57 , 117–168 (2016).

Stephan, P. E. How Economics Shapes Science (Harvard Univ. Press, 2012).

Garfield, E. & Sher, I. H. New factors in the evaluation of scientific literature through citation indexing. Am. Doc. 14 , 195–201 (1963).

Article   CAS   Google Scholar  

de Solla Price, D. J. Networks of scientific papers. Science 149 , 510–515 (1965).

Etzkowitz, H., Kemelgor, C. & Uzzi, B. Athena Unbound: The Advancement of Women in Science and Technology (Cambridge Univ. Press, 2000).

Simonton, D. K. Scientific Genius: A Psychology of Science (Cambridge Univ. Press, 1988).

Khabsa, M. & Giles, C. L. The number of scholarly documents on the public web. PLoS ONE 9 , e93949 (2014).

Xia, F., Wang, W., Bekele, T. M. & Liu, H. Big scholarly data: a survey. IEEE Trans. Big Data 3 , 18–35 (2017).

Evans, J. A. & Foster, J. G. Metaknowledge. Science 331 , 721–725 (2011).

Milojević, S. Quantifying the cognitive extent of science. J. Informetr. 9 , 962–973 (2015).

Rzhetsky, A., Foster, J. G., Foster, I. T. & Evans, J. A. Choosing experiments to accelerate collective discovery. Proc. Natl Acad. Sci. USA 112 , 14569–14574 (2015).

Poncela-Casasnovas, J., Gerlach, M., Aguirre, N. & Amaral, L. A. Large-scale analysis of micro-level citation patterns reveals nuanced selection criteria. Nat. Hum. Behav. 3 , 568–575 (2019).

Hardwicke, T. E. et al. Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition. R. Soc. Open Sci. 5 , 180448 (2018).

Nagaraj, A., Shears, E. & de Vaan, M. Improving data access democratizes and diversifies science. Proc. Natl Acad. Sci. USA 117 , 23490–23498 (2020).

Bravo, G., Grimaldo, F., López-Iñesta, E., Mehmani, B. & Squazzoni, F. The effect of publishing peer review reports on referee behavior in five scholarly journals. Nat. Commun. 10 , 322 (2019).

Tran, D. et al. An open review of open review: a critical analysis of the machine learning conference review process. Preprint at https://doi.org/10.48550/arXiv.2010.05137 (2020).

Tshitoyan, V. et al. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature 571 , 95–98 (2019).

Yang, Y., Wu, Y. & Uzzi, B. Estimating the deep replicability of scientific findings using human and artificial intelligence. Proc. Natl Acad. Sci. USA 117 , 10762–10768 (2020).

Mukherjee, S., Uzzi, B., Jones, B. & Stringer, M. A new method for identifying recombinations of existing knowledge associated with high‐impact innovation. J. Prod. Innov. Manage. 33 , 224–236 (2016).

Leahey, E., Beckman, C. M. & Stanko, T. L. Prominent but less productive: the impact of interdisciplinarity on scientists’ research. Adm. Sci. Q. 62 , 105–139 (2017).

Sauermann, H. & Haeussler, C. Authorship and contribution disclosures. Sci. Adv. 3 , e1700404 (2017).

Oliveira, D. F. M., Ma, Y., Woodruff, T. K. & Uzzi, B. Comparison of National Institutes of Health grant amounts to first-time male and female principal investigators. JAMA 321 , 898–900 (2019).

Yang, Y., Chawla, N. V. & Uzzi, B. A network’s gender composition and communication pattern predict women’s leadership success. Proc. Natl Acad. Sci. USA 116 , 2033–2038 (2019).

Way, S. F., Larremore, D. B. & Clauset, A. Gender, productivity, and prestige in computer science faculty hiring networks. In Proc. 25th International Conference on World Wide Web 1169–1179. (ACM 2016)

Malmgren, R. D., Ottino, J. M. & Amaral, L. A. N. The role of mentorship in protege performance. Nature 465 , 622–626 (2010).

Ma, Y., Mukherjee, S. & Uzzi, B. Mentorship and protégé success in STEM fields. Proc. Natl Acad. Sci. USA 117 , 14077–14083 (2020).

Börner, K. et al. Skill discrepancies between research, education, and jobs reveal the critical need to supply soft skills for the data economy. Proc. Natl Acad. Sci. USA 115 , 12630–12637 (2018).

Biasi, B. & Ma, S. The Education-Innovation Gap (National Bureau of Economic Research Working papers, 2020).

Bornmann, L. Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. J. Informetr. 8 , 895–903 (2014).

Cleary, E. G., Beierlein, J. M., Khanuja, N. S., McNamee, L. M. & Ledley, F. D. Contribution of NIH funding to new drug approvals 2010–2016. Proc. Natl Acad. Sci. USA 115 , 2329–2334 (2018).

Spector, J. M., Harrison, R. S. & Fishman, M. C. Fundamental science behind today’s important medicines. Sci. Transl. Med. 10 , eaaq1787 (2018).

Haunschild, R. & Bornmann, L. How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data. Scientometrics 110 , 1209–1216 (2017).

Yin, Y., Gao, J., Jones, B. F. & Wang, D. Coevolution of policy and science during the pandemic. Science 371 , 128–130 (2021).

Sugimoto, C. R., Work, S., Larivière, V. & Haustein, S. Scholarly use of social media and altmetrics: a review of the literature. J. Assoc. Inf. Sci. Technol. 68 , 2037–2062 (2017).

Dunham, I. Human genes: time to follow the roads less traveled? PLoS Biol. 16 , e3000034 (2018).

Kustatscher, G. et al. Understudied proteins: opportunities and challenges for functional proteomics. Nat. Methods 19 , 774–779 (2022).

Rosenthal, R. The file drawer problem and tolerance for null results. Psychol. Bull. 86 , 638 (1979).

Franco, A., Malhotra, N. & Simonovits, G. Publication bias in the social sciences: unlocking the file drawer. Science 345 , 1502–1505 (2014).

Vera-Baceta, M.-A., Thelwall, M. & Kousha, K. Web of Science and Scopus language coverage. Scientometrics 121 , 1803–1813 (2019).

Waltman, L. A review of the literature on citation impact indicators. J. Informetr. 10 , 365–391 (2016).

Garfield, E. & Merton, R. K. Citation Indexing: Its Theory and Application in Science, Technology, and Humanities (Wiley, 1979).

Kelly, B., Papanikolaou, D., Seru, A. & Taddy, M. Measuring Technological Innovation Over the Long Run Report No. 0898-2937 (National Bureau of Economic Research, 2018).

Kogan, L., Papanikolaou, D., Seru, A. & Stoffman, N. Technological innovation, resource allocation, and growth. Q. J. Econ. 132 , 665–712 (2017).

Hall, B. H., Jaffe, A. & Trajtenberg, M. Market value and patent citations. RAND J. Econ. 36 , 16–38 (2005).

Google Scholar  

Yan, E. & Ding, Y. Applying centrality measures to impact analysis: a coauthorship network analysis. J. Am. Soc. Inf. Sci. Technol. 60 , 2107–2118 (2009).

Radicchi, F., Fortunato, S., Markines, B. & Vespignani, A. Diffusion of scientific credits and the ranking of scientists. Phys. Rev. E 80 , 056103 (2009).

Bollen, J., Rodriquez, M. A. & Van de Sompel, H. Journal status. Scientometrics 69 , 669–687 (2006).

Bergstrom, C. T., West, J. D. & Wiseman, M. A. The eigenfactor™ metrics. J. Neurosci. 28 , 11433–11434 (2008).

Cronin, B. & Sugimoto, C. R. Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact (MIT Press, 2014).

Hicks, D., Wouters, P., Waltman, L., De Rijcke, S. & Rafols, I. Bibliometrics: the Leiden Manifesto for research metrics. Nature 520 , 429–431 (2015).

Catalini, C., Lacetera, N. & Oettl, A. The incidence and role of negative citations in science. Proc. Natl Acad. Sci. USA 112 , 13823–13826 (2015).

Alcacer, J. & Gittelman, M. Patent citations as a measure of knowledge flows: the influence of examiner citations. Rev. Econ. Stat. 88 , 774–779 (2006).

Ding, Y. et al. Content‐based citation analysis: the next generation of citation analysis. J. Assoc. Inf. Sci. Technol. 65 , 1820–1833 (2014).

Teufel, S., Siddharthan, A. & Tidhar, D. Automatic classification of citation function. In Proc. 2006 Conference on Empirical Methods in Natural Language Processing, 103–110 (Association for Computational Linguistics 2006)

Seeber, M., Cattaneo, M., Meoli, M. & Malighetti, P. Self-citations as strategic response to the use of metrics for career decisions. Res. Policy 48 , 478–491 (2019).

Pendlebury, D. A. The use and misuse of journal metrics and other citation indicators. Arch. Immunol. Ther. Exp. 57 , 1–11 (2009).

Biagioli, M. Watch out for cheats in citation game. Nature 535 , 201 (2016).

Jo, W. S., Liu, L. & Wang, D. See further upon the giants: quantifying intellectual lineage in science. Quant. Sci. Stud. 3 , 319–330 (2022).

Boyack, K. W., Klavans, R. & Börner, K. Mapping the backbone of science. Scientometrics 64 , 351–374 (2005).

Gates, A. J., Ke, Q., Varol, O. & Barabási, A.-L. Nature’s reach: narrow work has broad impact. Nature 575 , 32–34 (2019).

Börner, K., Penumarthy, S., Meiss, M. & Ke, W. Mapping the diffusion of scholarly knowledge among major US research institutions. Scientometrics 68 , 415–426 (2006).

King, D. A. The scientific impact of nations. Nature 430 , 311–316 (2004).

Pan, R. K., Kaski, K. & Fortunato, S. World citation and collaboration networks: uncovering the role of geography in science. Sci. Rep. 2 , 902 (2012).

Jaffe, A. B., Trajtenberg, M. & Henderson, R. Geographic localization of knowledge spillovers as evidenced by patent citations. Q. J. Econ. 108 , 577–598 (1993).

Funk, R. J. & Owen-Smith, J. A dynamic network measure of technological change. Manage. Sci. 63 , 791–817 (2017).

Yegros-Yegros, A., Rafols, I. & D’este, P. Does interdisciplinary research lead to higher citation impact? The different effect of proximal and distal interdisciplinarity. PLoS ONE 10 , e0135095 (2015).

Larivière, V., Haustein, S. & Börner, K. Long-distance interdisciplinarity leads to higher scientific impact. PLoS ONE 10 , e0122565 (2015).

Fleming, L., Greene, H., Li, G., Marx, M. & Yao, D. Government-funded research increasingly fuels innovation. Science 364 , 1139–1141 (2019).

Bowen, A. & Casadevall, A. Increasing disparities between resource inputs and outcomes, as measured by certain health deliverables, in biomedical research. Proc. Natl Acad. Sci. USA 112 , 11335–11340 (2015).

Li, D., Azoulay, P. & Sampat, B. N. The applied value of public investments in biomedical research. Science 356 , 78–81 (2017).

Lehman, H. C. Age and Achievement (Princeton Univ. Press, 2017).

Simonton, D. K. Creative productivity: a predictive and explanatory model of career trajectories and landmarks. Psychol. Rev. 104 , 66 (1997).

Duch, J. et al. The possible role of resource requirements and academic career-choice risk on gender differences in publication rate and impact. PLoS ONE 7 , e51332 (2012).

Wang, Y., Jones, B. F. & Wang, D. Early-career setback and future career impact. Nat. Commun. 10 , 4331 (2019).

Bol, T., de Vaan, M. & van de Rijt, A. The Matthew effect in science funding. Proc. Natl Acad. Sci. USA 115 , 4887–4890 (2018).

Jones, B. F. Age and great invention. Rev. Econ. Stat. 92 , 1–14 (2010).

Newman, M. Networks (Oxford Univ. Press, 2018).

Mazloumian, A., Eom, Y.-H., Helbing, D., Lozano, S. & Fortunato, S. How citation boosts promote scientific paradigm shifts and nobel prizes. PLoS ONE 6 , e18975 (2011).

Hirsch, J. E. An index to quantify an individual’s scientific research output. Proc. Natl Acad. Sci. USA 102 , 16569–16572 (2005).

Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E. & Herrera, F. h-index: a review focused in its variants, computation and standardization for different scientific fields. J. Informetr. 3 , 273–289 (2009).

Egghe, L. An improvement of the h-index: the g-index. ISSI Newsl. 2 , 8–9 (2006).

Kaur, J., Radicchi, F. & Menczer, F. Universality of scholarly impact metrics. J. Informetr. 7 , 924–932 (2013).

Majeti, D. et al. Scholar plot: design and evaluation of an information interface for faculty research performance. Front. Res. Metr. Anal. 4 , 6 (2020).

Sidiropoulos, A., Katsaros, D. & Manolopoulos, Y. Generalized Hirsch h-index for disclosing latent facts in citation networks. Scientometrics 72 , 253–280 (2007).

Jones, B. F. & Weinberg, B. A. Age dynamics in scientific creativity. Proc. Natl Acad. Sci. USA 108 , 18910–18914 (2011).

Dennis, W. Age and productivity among scientists. Science 123 , 724–725 (1956).

Sanyal, D. K., Bhowmick, P. K. & Das, P. P. A review of author name disambiguation techniques for the PubMed bibliographic database. J. Inf. Sci. 47 , 227–254 (2021).

Haak, L. L., Fenner, M., Paglione, L., Pentz, E. & Ratner, H. ORCID: a system to uniquely identify researchers. Learn. Publ. 25 , 259–264 (2012).

Malmgren, R. D., Ottino, J. M. & Amaral, L. A. N. The role of mentorship in protégé performance. Nature 465 , 662–667 (2010).

Oettl, A. Reconceptualizing stars: scientist helpfulness and peer performance. Manage. Sci. 58 , 1122–1140 (2012).

Morgan, A. C. et al. The unequal impact of parenthood in academia. Sci. Adv. 7 , eabd1996 (2021).

Morgan, A. C. et al. Socioeconomic roots of academic faculty. Nat. Hum. Behav. 6 , 1625–1633 (2022).

San Francisco Declaration on Research Assessment (DORA) (American Society for Cell Biology, 2012).

Falk‐Krzesinski, H. J. et al. Advancing the science of team science. Clin. Transl. Sci. 3 , 263–266 (2010).

Cooke, N. J. et al. Enhancing the Effectiveness of Team Science (National Academies Press, 2015).

Börner, K. et al. A multi-level systems perspective for the science of team science. Sci. Transl. Med. 2 , 49cm24 (2010).

Leahey, E. From sole investigator to team scientist: trends in the practice and study of research collaboration. Annu. Rev. Sociol. 42 , 81–100 (2016).

AlShebli, B. K., Rahwan, T. & Woon, W. L. The preeminence of ethnic diversity in scientific collaboration. Nat. Commun. 9 , 5163 (2018).

Hsiehchen, D., Espinoza, M. & Hsieh, A. Multinational teams and diseconomies of scale in collaborative research. Sci. Adv. 1 , e1500211 (2015).

Koning, R., Samila, S. & Ferguson, J.-P. Who do we invent for? Patents by women focus more on women’s health, but few women get to invent. Science 372 , 1345–1348 (2021).

Barabâsi, A.-L. et al. Evolution of the social network of scientific collaborations. Physica A 311 , 590–614 (2002).

Newman, M. E. Scientific collaboration networks. I. Network construction and fundamental results. Phys. Rev. E 64 , 016131 (2001).

Newman, M. E. Scientific collaboration networks. II. Shortest paths, weighted networks, and centrality. Phys. Rev. E 64 , 016132 (2001).

Palla, G., Barabási, A.-L. & Vicsek, T. Quantifying social group evolution. Nature 446 , 664–667 (2007).

Ross, M. B. et al. Women are credited less in science than men. Nature 608 , 135–145 (2022).

Shen, H.-W. & Barabási, A.-L. Collective credit allocation in science. Proc. Natl Acad. Sci. USA 111 , 12325–12330 (2014).

Merton, R. K. Matthew effect in science. Science 159 , 56–63 (1968).

Ni, C., Smith, E., Yuan, H., Larivière, V. & Sugimoto, C. R. The gendered nature of authorship. Sci. Adv. 7 , eabe4639 (2021).

Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N. & Malone, T. W. Evidence for a collective intelligence factor in the performance of human groups. Science 330 , 686–688 (2010).

Feldon, D. F. et al. Postdocs’ lab engagement predicts trajectories of PhD students’ skill development. Proc. Natl Acad. Sci. USA 116 , 20910–20916 (2019).

Boudreau, K. J. et al. A field experiment on search costs and the formation of scientific collaborations. Rev. Econ. Stat. 99 , 565–576 (2017).

Holcombe, A. O. Contributorship, not authorship: use CRediT to indicate who did what. Publications 7 , 48 (2019).

Murray, D. et al. Unsupervised embedding of trajectories captures the latent structure of mobility. Preprint at https://doi.org/10.48550/arXiv.2012.02785 (2020).

Deville, P. et al. Career on the move: geography, stratification, and scientific impact. Sci. Rep. 4 , 4770 (2014).

Edmunds, L. D. et al. Why do women choose or reject careers in academic medicine? A narrative review of empirical evidence. Lancet 388 , 2948–2958 (2016).

Waldinger, F. Peer effects in science: evidence from the dismissal of scientists in Nazi Germany. Rev. Econ. Stud. 79 , 838–861 (2012).

Agrawal, A., McHale, J. & Oettl, A. How stars matter: recruiting and peer effects in evolutionary biology. Res. Policy 46 , 853–867 (2017).

Fiore, S. M. Interdisciplinarity as teamwork: how the science of teams can inform team science. Small Group Res. 39 , 251–277 (2008).

Hvide, H. K. & Jones, B. F. University innovation and the professor’s privilege. Am. Econ. Rev. 108 , 1860–1898 (2018).

Murray, F., Aghion, P., Dewatripont, M., Kolev, J. & Stern, S. Of mice and academics: examining the effect of openness on innovation. Am. Econ. J. Econ. Policy 8 , 212–252 (2016).

Radicchi, F., Fortunato, S. & Castellano, C. Universality of citation distributions: toward an objective measure of scientific impact. Proc. Natl Acad. Sci. USA 105 , 17268–17272 (2008).

Waltman, L., van Eck, N. J. & van Raan, A. F. Universality of citation distributions revisited. J. Am. Soc. Inf. Sci. Technol. 63 , 72–77 (2012).

Barabási, A.-L. & Albert, R. Emergence of scaling in random networks. Science 286 , 509–512 (1999).

de Solla Price, D. A general theory of bibliometric and other cumulative advantage processes. J. Am. Soc. Inf. Sci. 27 , 292–306 (1976).

Cole, S. Age and scientific performance. Am. J. Sociol. 84 , 958–977 (1979).

Ke, Q., Ferrara, E., Radicchi, F. & Flammini, A. Defining and identifying sleeping beauties in science. Proc. Natl Acad. Sci. USA 112 , 7426–7431 (2015).

Bornmann, L., de Moya Anegón, F. & Leydesdorff, L. Do scientific advancements lean on the shoulders of giants? A bibliometric investigation of the Ortega hypothesis. PLoS ONE 5 , e13327 (2010).

Mukherjee, S., Romero, D. M., Jones, B. & Uzzi, B. The nearly universal link between the age of past knowledge and tomorrow’s breakthroughs in science and technology: the hotspot. Sci. Adv. 3 , e1601315 (2017).

Packalen, M. & Bhattacharya, J. NIH funding and the pursuit of edge science. Proc. Natl Acad. Sci. USA 117 , 12011–12016 (2020).

Zeng, A., Fan, Y., Di, Z., Wang, Y. & Havlin, S. Fresh teams are associated with original and multidisciplinary research. Nat. Hum. Behav. 5 , 1314–1322 (2021).

Newman, M. E. The structure of scientific collaboration networks. Proc. Natl Acad. Sci. USA 98 , 404–409 (2001).

Larivière, V., Ni, C., Gingras, Y., Cronin, B. & Sugimoto, C. R. Bibliometrics: global gender disparities in science. Nature 504 , 211–213 (2013).

West, J. D., Jacquet, J., King, M. M., Correll, S. J. & Bergstrom, C. T. The role of gender in scholarly authorship. PLoS ONE 8 , e66212 (2013).

Gao, J., Yin, Y., Myers, K. R., Lakhani, K. R. & Wang, D. Potentially long-lasting effects of the pandemic on scientists. Nat. Commun. 12 , 6188 (2021).

Jones, B. F., Wuchty, S. & Uzzi, B. Multi-university research teams: shifting impact, geography, and stratification in science. Science 322 , 1259–1262 (2008).

Chu, J. S. & Evans, J. A. Slowed canonical progress in large fields of science. Proc. Natl Acad. Sci. USA 118 , e2021636118 (2021).

Wang, J., Veugelers, R. & Stephan, P. Bias against novelty in science: a cautionary tale for users of bibliometric indicators. Res. Policy 46 , 1416–1436 (2017).

Stringer, M. J., Sales-Pardo, M. & Amaral, L. A. Statistical validation of a global model for the distribution of the ultimate number of citations accrued by papers published in a scientific journal. J. Assoc. Inf. Sci. Technol. 61 , 1377–1385 (2010).

Bianconi, G. & Barabási, A.-L. Bose-Einstein condensation in complex networks. Phys. Rev. Lett. 86 , 5632 (2001).

Bianconi, G. & Barabási, A.-L. Competition and multiscaling in evolving networks. Europhys. Lett. 54 , 436 (2001).

Yin, Y. & Wang, D. The time dimension of science: connecting the past to the future. J. Informetr. 11 , 608–621 (2017).

Pan, R. K., Petersen, A. M., Pammolli, F. & Fortunato, S. The memory of science: Inflation, myopia, and the knowledge network. J. Informetr. 12 , 656–678 (2018).

Yin, Y., Wang, Y., Evans, J. A. & Wang, D. Quantifying the dynamics of failure across science, startups and security. Nature 575 , 190–194 (2019).

Candia, C. & Uzzi, B. Quantifying the selective forgetting and integration of ideas in science and technology. Am. Psychol. 76 , 1067 (2021).

Milojević, S. Principles of scientific research team formation and evolution. Proc. Natl Acad. Sci. USA 111 , 3984–3989 (2014).

Guimera, R., Uzzi, B., Spiro, J. & Amaral, L. A. N. Team assembly mechanisms determine collaboration network structure and team performance. Science 308 , 697–702 (2005).

Newman, M. E. Coauthorship networks and patterns of scientific collaboration. Proc. Natl Acad. Sci. USA 101 , 5200–5205 (2004).

Newman, M. E. Clustering and preferential attachment in growing networks. Phys. Rev. E 64 , 025102 (2001).

Iacopini, I., Milojević, S. & Latora, V. Network dynamics of innovation processes. Phys. Rev. Lett. 120 , 048301 (2018).

Kuhn, T., Perc, M. & Helbing, D. Inheritance patterns in citation networks reveal scientific memes. Phys. Rev. 4 , 041036 (2014).

Jia, T., Wang, D. & Szymanski, B. K. Quantifying patterns of research-interest evolution. Nat. Hum. Behav. 1 , 0078 (2017).

Zeng, A. et al. Increasing trend of scientists to switch between topics. Nat. Commun. https://doi.org/10.1038/s41467-019-11401-8 (2019).

Siudem, G., Żogała-Siudem, B., Cena, A. & Gagolewski, M. Three dimensions of scientific impact. Proc. Natl Acad. Sci. USA 117 , 13896–13900 (2020).

Petersen, A. M. et al. Reputation and impact in academic careers. Proc. Natl Acad. Sci. USA 111 , 15316–15321 (2014).

Jin, C., Song, C., Bjelland, J., Canright, G. & Wang, D. Emergence of scaling in complex substitutive systems. Nat. Hum. Behav. 3 , 837–846 (2019).

Hofman, J. M. et al. Integrating explanation and prediction in computational social science. Nature 595 , 181–188 (2021).

Lazer, D. et al. Computational social science. Science 323 , 721–723 (2009).

Lazer, D. M. et al. Computational social science: obstacles and opportunities. Science 369 , 1060–1062 (2020).

Albert, R. & Barabási, A.-L. Statistical mechanics of complex networks. Rev. Mod. Phys. 74 , 47 (2002).

Newman, M. E. The structure and function of complex networks. SIAM Rev. 45 , 167–256 (2003).

Song, C., Qu, Z., Blumm, N. & Barabási, A.-L. Limits of predictability in human mobility. Science 327 , 1018–1021 (2010).

Alessandretti, L., Aslak, U. & Lehmann, S. The scales of human mobility. Nature 587 , 402–407 (2020).

Pastor-Satorras, R. & Vespignani, A. Epidemic spreading in scale-free networks. Phys. Rev. Lett. 86 , 3200 (2001).

Pastor-Satorras, R., Castellano, C., Van Mieghem, P. & Vespignani, A. Epidemic processes in complex networks. Rev. Mod. Phys. 87 , 925 (2015).

Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning (MIT Press, 2016).

Bishop, C. M. Pattern Recognition and Machine Learning (Springer, 2006).

Dong, Y., Johnson, R. A. & Chawla, N. V. Will this paper increase your h-index? Scientific impact prediction. In Proc. 8th ACM International Conference on Web Search and Data Mining, 149–158 (ACM 2015)

Xiao, S. et al. On modeling and predicting individual paper citation count over time. In IJCAI, 2676–2682 (IJCAI, 2016)

Fortunato, S. Community detection in graphs. Phys. Rep. 486 , 75–174 (2010).

Chen, C. Science mapping: a systematic review of the literature. J. Data Inf. Sci. 2 , 1–40 (2017).

CAS   Google Scholar  

Van Eck, N. J. & Waltman, L. Citation-based clustering of publications using CitNetExplorer and VOSviewer. Scientometrics 111 , 1053–1070 (2017).

LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521 , 436–444 (2015).

Senior, A. W. et al. Improved protein structure prediction using potentials from deep learning. Nature 577 , 706–710 (2020).

Krenn, M. & Zeilinger, A. Predicting research trends with semantic and neural networks with an application in quantum physics. Proc. Natl Acad. Sci. USA 117 , 1910–1916 (2020).

Iten, R., Metger, T., Wilming, H., Del Rio, L. & Renner, R. Discovering physical concepts with neural networks. Phys. Rev. Lett. 124 , 010508 (2020).

Guimerà, R. et al. A Bayesian machine scientist to aid in the solution of challenging scientific problems. Sci. Adv. 6 , eaav6971 (2020).

Segler, M. H., Preuss, M. & Waller, M. P. Planning chemical syntheses with deep neural networks and symbolic AI. Nature 555 , 604–610 (2018).

Ryu, J. Y., Kim, H. U. & Lee, S. Y. Deep learning improves prediction of drug–drug and drug–food interactions. Proc. Natl Acad. Sci. USA 115 , E4304–E4311 (2018).

Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172 , 1122–1131.e9 (2018).

Peng, H., Ke, Q., Budak, C., Romero, D. M. & Ahn, Y.-Y. Neural embeddings of scholarly periodicals reveal complex disciplinary organizations. Sci. Adv. 7 , eabb9004 (2021).

Youyou, W., Yang, Y. & Uzzi, B. A discipline-wide investigation of the replicability of psychology papers over the past two decades. Proc. Natl Acad. Sci. USA 120 , e2208863120 (2023).

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K. & Galstyan, A. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54 , 1–35 (2021).

Way, S. F., Morgan, A. C., Larremore, D. B. & Clauset, A. Productivity, prominence, and the effects of academic environment. Proc. Natl Acad. Sci. USA 116 , 10729–10733 (2019).

Li, W., Aste, T., Caccioli, F. & Livan, G. Early coauthorship with top scientists predicts success in academic careers. Nat. Commun. 10 , 5170 (2019).

Hendry, D. F., Pagan, A. R. & Sargan, J. D. Dynamic specification. Handb. Econ. 2 , 1023–1100 (1984).

Jin, C., Ma, Y. & Uzzi, B. Scientific prizes and the extraordinary growth of scientific topics. Nat. Commun. 12 , 5619 (2021).

Azoulay, P., Ganguli, I. & Zivin, J. G. The mobility of elite life scientists: professional and personal determinants. Res. Policy 46 , 573–590 (2017).

Slavova, K., Fosfuri, A. & De Castro, J. O. Learning by hiring: the effects of scientists’ inbound mobility on research performance in academia. Organ. Sci. 27 , 72–89 (2016).

Sarsons, H. Recognition for group work: gender differences in academia. Am. Econ. Rev. 107 , 141–145 (2017).

Campbell, L. G., Mehtani, S., Dozier, M. E. & Rinehart, J. Gender-heterogeneous working groups produce higher quality science. PLoS ONE 8 , e79147 (2013).

Azoulay, P., Graff Zivin, J. S. & Wang, J. Superstar extinction. Q. J. Econ. 125 , 549–589 (2010).

Furman, J. L. & Stern, S. Climbing atop the shoulders of giants: the impact of institutions on cumulative research. Am. Econ. Rev. 101 , 1933–1963 (2011).

Williams, H. L. Intellectual property rights and innovation: evidence from the human genome. J. Polit. Econ. 121 , 1–27 (2013).

Rubin, A. & Rubin, E. Systematic Bias in the Progress of Research. J. Polit. Econ. 129 , 2666–2719 (2021).

Lu, S. F., Jin, G. Z., Uzzi, B. & Jones, B. The retraction penalty: evidence from the Web of Science. Sci. Rep. 3 , 3146 (2013).

Jin, G. Z., Jones, B., Lu, S. F. & Uzzi, B. The reverse Matthew effect: consequences of retraction in scientific teams. Rev. Econ. Stat. 101 , 492–506 (2019).

Azoulay, P., Bonatti, A. & Krieger, J. L. The career effects of scandal: evidence from scientific retractions. Res. Policy 46 , 1552–1569 (2017).

Goodman-Bacon, A. Difference-in-differences with variation in treatment timing. J. Econ. 225 , 254–277 (2021).

Callaway, B. & Sant’Anna, P. H. Difference-in-differences with multiple time periods. J. Econ. 225 , 200–230 (2021).

Hill, R. Searching for Superstars: Research Risk and Talent Discovery in Astronomy Working Paper (Massachusetts Institute of Technology, 2019).

Bagues, M., Sylos-Labini, M. & Zinovyeva, N. Does the gender composition of scientific committees matter? Am. Econ. Rev. 107 , 1207–1238 (2017).

Sampat, B. & Williams, H. L. How do patents affect follow-on innovation? Evidence from the human genome. Am. Econ. Rev. 109 , 203–236 (2019).

Moretti, E. & Wilson, D. J. The effect of state taxes on the geographical location of top earners: evidence from star scientists. Am. Econ. Rev. 107 , 1858–1903 (2017).

Jacob, B. A. & Lefgren, L. The impact of research grant funding on scientific productivity. J. Public Econ. 95 , 1168–1177 (2011).

Li, D. Expertise versus bias in evaluation: evidence from the NIH. Am. Econ. J. Appl. Econ. 9 , 60–92 (2017).

Pearl, J. Causal diagrams for empirical research. Biometrika 82 , 669–688 (1995).

Pearl, J. & Mackenzie, D. The Book of Why: The New Science of Cause and Effect (Basic Books, 2018).

Traag, V. A. Inferring the causal effect of journals on citations. Quant. Sci. Stud. 2 , 496–504 (2021).

Traag, V. & Waltman, L. Causal foundations of bias, disparity and fairness. Preprint at https://doi.org/10.48550/arXiv.2207.13665 (2022).

Imbens, G. W. Potential outcome and directed acyclic graph approaches to causality: relevance for empirical practice in economics. J. Econ. Lit. 58 , 1129–1179 (2020).

Heckman, J. J. & Pinto, R. Causality and Econometrics (National Bureau of Economic Research, 2022).

Aggarwal, I., Woolley, A. W., Chabris, C. F. & Malone, T. W. The impact of cognitive style diversity on implicit learning in teams. Front. Psychol. 10 , 112 (2019).

Balietti, S., Goldstone, R. L. & Helbing, D. Peer review and competition in the Art Exhibition Game. Proc. Natl Acad. Sci. USA 113 , 8414–8419 (2016).

Paulus, F. M., Rademacher, L., Schäfer, T. A. J., Müller-Pinzler, L. & Krach, S. Journal impact factor shapes scientists’ reward signal in the prospect of publication. PLoS ONE 10 , e0142537 (2015).

Williams, W. M. & Ceci, S. J. National hiring experiments reveal 2:1 faculty preference for women on STEM tenure track. Proc. Natl Acad. Sci. USA 112 , 5360–5365 (2015).

Collaboration, O. S. Estimating the reproducibility of psychological science. Science 349 , aac4716 (2015).

Camerer, C. F. et al. Evaluating replicability of laboratory experiments in economics. Science 351 , 1433–1436 (2016).

Camerer, C. F. et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav. 2 , 637–644 (2018).

Duflo, E. & Banerjee, A. Handbook of Field Experiments (Elsevier, 2017).

Tomkins, A., Zhang, M. & Heavlin, W. D. Reviewer bias in single versus double-blind peer review. Proc. Natl Acad. Sci. USA 114 , 12708–12713 (2017).

Blank, R. M. The effects of double-blind versus single-blind reviewing: experimental evidence from the American Economic Review. Am. Econ. Rev. 81 , 1041–1067 (1991).

Boudreau, K. J., Guinan, E. C., Lakhani, K. R. & Riedl, C. Looking across and looking beyond the knowledge frontier: intellectual distance, novelty, and resource allocation in science. Manage. Sci. 62 , 2765–2783 (2016).

Lane, J. et al. When Do Experts Listen to Other Experts? The Role of Negative Information in Expert Evaluations for Novel Projects Working Paper #21-007 (Harvard Business School, 2020).

Teplitskiy, M. et al. Do Experts Listen to Other Experts? Field Experimental Evidence from Scientific Peer Review (Harvard Business School, 2019).

Moss-Racusin, C. A., Dovidio, J. F., Brescoll, V. L., Graham, M. J. & Handelsman, J. Science faculty’s subtle gender biases favor male students. Proc. Natl Acad. Sci. USA 109 , 16474–16479 (2012).

Forscher, P. S., Cox, W. T., Brauer, M. & Devine, P. G. Little race or gender bias in an experiment of initial review of NIH R01 grant proposals. Nat. Hum. Behav. 3 , 257–264 (2019).

Dennehy, T. C. & Dasgupta, N. Female peer mentors early in college increase women’s positive academic experiences and retention in engineering. Proc. Natl Acad. Sci. USA 114 , 5964–5969 (2017).

Azoulay, P. Turn the scientific method on ourselves. Nature 484 , 31–32 (2012).

Download references

Acknowledgements

The authors thank all members of the Center for Science of Science and Innovation (CSSI) for invaluable comments. This work was supported by the Air Force Office of Scientific Research under award number FA9550-19-1-0354, National Science Foundation grant SBE 1829344, and the Alfred P. Sloan Foundation G-2019-12485.

Author information

Authors and affiliations.

Center for Science of Science and Innovation, Northwestern University, Evanston, IL, USA

Lu Liu, Benjamin F. Jones, Brian Uzzi & Dashun Wang

Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL, USA

Kellogg School of Management, Northwestern University, Evanston, IL, USA

College of Information Sciences and Technology, Pennsylvania State University, University Park, PA, USA

National Bureau of Economic Research, Cambridge, MA, USA

Benjamin F. Jones

Brookings Institution, Washington, DC, USA

McCormick School of Engineering, Northwestern University, Evanston, IL, USA

• Dashun Wang

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Dashun Wang .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Human Behaviour thanks Ludo Waltman, Erin Leahey and Sarah Bratt for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article.

Liu, L., Jones, B.F., Uzzi, B. et al. Data, measurement and empirical methods in the science of science. Nat Hum Behav 7 , 1046–1058 (2023). https://doi.org/10.1038/s41562-023-01562-4

Download citation

Received : 30 June 2022

Accepted : 17 February 2023

Published : 01 June 2023

Issue Date : July 2023

DOI : https://doi.org/10.1038/s41562-023-01562-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Publication, funding, and experimental data in support of human reference atlas construction and usage.

• Yongxin Kong
• Katy Börner

Scientific Data (2024)

Quantifying attrition in science: a cohort-based, longitudinal study of scientists in 38 OECD countries

• Marek Kwiek
• Lukasz Szymula

Higher Education (2024)

Rescaling the disruption index reveals the universality of disruption distributions in science

• Alex J. Yang
• Hongcun Gong
• Sanhong Deng

Scientometrics (2024)

Signs of Dysconscious Racism and Xenophobiaism in Knowledge Production and the Formation of Academic Researchers: A National Study

• Dina Zoe Belluigi

Journal of Academic Ethics (2024)

Scientific Data (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Empirical Research: A Comprehensive Guide for Academics 

Empirical research relies on gathering and studying real, observable data. The term ’empirical’ comes from the Greek word ’empeirikos,’ meaning ‘experienced’ or ‘based on experience.’ So, what is empirical research? Instead of using theories or opinions, empirical research depends on real data obtained through direct observation or experimentation. 

Why Empirical Research?

Empirical research plays a key role in checking or improving current theories, providing a systematic way to grow knowledge across different areas. By focusing on objectivity, it makes research findings more trustworthy, which is critical in research fields like medicine, psychology, economics, and public policy. In the end, the strengths of empirical research lie in deepening our awareness of the world and improving our capacity to tackle problems wisely. 1,2  

Qualitative and Quantitative Methods

There are two main types of empirical research methods – qualitative and quantitative. 3,4 Qualitative research delves into intricate phenomena using non-numerical data, such as interviews or observations, to offer in-depth insights into human experiences. In contrast, quantitative research analyzes numerical data to spot patterns and relationships, aiming for objectivity and the ability to apply findings to a wider context. 

Steps for Conducting Empirical Research

When it comes to conducting research, there are some simple steps that researchers can follow. 5,6  

• Create Research Hypothesis:  Clearly state the specific question you want to answer or the hypothesis you want to explore in your study. 
• Examine Existing Research:  Read and study existing research on your topic. Understand what’s already known, identify existing gaps in knowledge, and create a framework for your own study based on what you learn. 
• Plan Your Study:  Decide how you’ll conduct your research—whether through qualitative methods, quantitative methods, or a mix of both. Choose suitable techniques like surveys, experiments, interviews, or observations based on your research question. 
• Develop Research Instruments:  Create reliable research collection tools, such as surveys or questionnaires, to help you collate data. Ensure these tools are well-designed and effective. 
• Collect Data:  Systematically gather the information you need for your research according to your study design and protocols using the chosen research methods. 
• Data Analysis:  Analyze the collected data using suitable statistical or qualitative methods that align with your research question and objectives. 
• Interpret Results:  Understand and explain the significance of your analysis results in the context of your research question or hypothesis. 
• Draw Conclusions:  Summarize your findings and draw conclusions based on the evidence. Acknowledge any study limitations and propose areas for future research. 

Advantages of Empirical Research

Empirical research is valuable because it stays objective by relying on observable data, lessening the impact of personal biases. This objectivity boosts the trustworthiness of research findings. Also, using precise quantitative methods helps in accurate measurement and statistical analysis. This precision ensures researchers can draw reliable conclusions from numerical data, strengthening our understanding of the studied phenomena. 4  

Disadvantages of Empirical Research

While empirical research has notable strengths, researchers must also be aware of its limitations when deciding on the right research method for their study.4 One significant drawback of empirical research is the risk of oversimplifying complex phenomena, especially when relying solely on quantitative methods. These methods may struggle to capture the richness and nuances present in certain social, cultural, or psychological contexts. Another challenge is the potential for confounding variables or biases during data collection, impacting result accuracy.  

Tips for Empirical Writing

In empirical research, the writing is usually done in research papers, articles, or reports. The empirical writing follows a set structure, and each section has a specific role. Here are some tips for your empirical writing. 7   

• Define Your Objectives:  When you write about your research, start by making your goals clear. Explain what you want to find out or prove in a simple and direct way. This helps guide your research and lets others know what you have set out to achieve. 
• Be Specific in Your Literature Review:  In the part where you talk about what others have studied before you, focus on research that directly relates to your research question. Keep it short and pick studies that help explain why your research is important. This part sets the stage for your work. 
• Explain Your Methods Clearly : When you talk about how you did your research (Methods), explain it in detail. Be clear about your research plan, who took part, and what you did; this helps others understand and trust your study. Also, be honest about any rules you follow to make sure your study is ethical and reproducible. 
• Share Your Results Clearly : After doing your empirical research, share what you found in a simple way. Use tables or graphs to make it easier for your audience to understand your research. Also, talk about any numbers you found and clearly state if they are important or not. Ensure that others can see why your research findings matter. 
• Talk About What Your Findings Mean:  In the part where you discuss your research results, explain what they mean. Discuss why your findings are important and if they connect to what others have found before. Be honest about any problems with your study and suggest ideas for more research in the future. 
• Wrap It Up Clearly:  Finally, end your empirical research paper by summarizing what you found and why it’s important. Remind everyone why your study matters. Keep your writing clear and fix any mistakes before you share it. Ask someone you trust to read it and give you feedback before you finish. 

References:  

• Empirical Research in the Social Sciences and Education, Penn State University Libraries. Available online at  https://guides.libraries.psu.edu/emp  
• How to conduct empirical research, Emerald Publishing. Available online at  https://www.emeraldgrouppublishing.com/how-to/research-methods/conduct-empirical-research  
• Empirical Research: Quantitative & Qualitative, Arrendale Library, Piedmont University. Available online at  https://library.piedmont.edu/empirical-research  
• Bouchrika, I.  What Is Empirical Research? Definition, Types & Samples  in 2024. Research.com, January 2024. Available online at  https://research.com/research/what-is-empirical-research  
• Quantitative and Empirical Research vs. Other Types of Research. California State University, April 2023. Available online at  https://libguides.csusb.edu/quantitative  
• Empirical Research, Definitions, Methods, Types and Examples, Studocu.com website. Available online at  https://www.studocu.com/row/document/uganda-christian-university/it-research-methods/emperical-research-definitions-methods-types-and-examples/55333816  
• Writing an Empirical Paper in APA Style. Psychology Writing Center, University of Washington. Available online at  https://psych.uw.edu/storage/writing_center/APApaper.pdf  

Paperpal is an AI writing assistant that help academics write better, faster with real-time suggestions for in-depth language and grammar correction. Trained on millions of research manuscripts enhanced by professional academic editors, Paperpal delivers human precision at machine speed.  

Try it for free or upgrade to  Paperpal Prime , which unlocks unlimited access to premium features like academic translation, paraphrasing, contextual synonyms, consistency checks and more. It’s like always having a professional academic editor by your side! Go beyond limitations and experience the future of academic writing.  Get Paperpal Prime now at just US$19 a month!  

Related Reads:

• How to Write a Scientific Paper in 10 Steps 
• What is a Literature Review? How to Write It (with Examples)
• What is an Argumentative Essay? How to Write It (With Examples)
• Ethical Research Practices For Research with Human Subjects

Ethics in Science: Importance, Principles & Guidelines 

Presenting research data effectively through tables and figures, you may also like, academic integrity vs academic dishonesty: types & examples, dissertation printing and binding | types & comparison , what is a dissertation preface definition and examples , the ai revolution: authors’ role in upholding academic..., the future of academia: how ai tools are..., how to write a research proposal: (with examples..., how to write your research paper in apa..., how to choose a dissertation topic, how to write a phd research proposal, how to write an academic paragraph (step-by-step guide).

Understanding the Empirical Method in Research Methodology

Table of Contents

Have you ever wondered how scientists gather evidence to support their theories? Or what steps researchers take to ensure that their findings are reliable and not just based on speculation? The answer lies in a cornerstone of scientific investigation known as the empirical method . This approach to research is all about collecting data and observing the world to form solid, evidence-based conclusions. Let’s dive into the empirical method’s fascinating world and understand why it’s so critical in research methodology.

What is the empirical method?

The empirical method is a way of gaining knowledge by means of direct and indirect observation or experience. It’s fundamentally based on the idea that knowledge comes from sensory experience and can be acquired through observation and experimentation. This method stands in contrast to approaches that rely solely on theoretical or logical means.

The role of observation in the empirical method

Observation is at the heart of the empirical method. It involves using your senses to gather information about the world. This could be as simple as noting the color of a flower or as complex as using advanced technology to observe the behavior of microscopic organisms. The key is that the observations must be systematic and replicable, providing reliable data that can be used to draw conclusions.

Data collection: qualitative and quantitative

Different types of data can be collected using the empirical method:

• Qualitative data – This data type is descriptive and conceptual, often collected through interviews, observations, and case studies.
• Quantitative data – This involves numerical data collected through methods like surveys, experiments, and statistical analysis.

Empirical vs. experimental methods

While the empirical method is often associated with experimentation, it’s important to distinguish between the two. Experimental methods involve controlled tests where the researcher manipulates one variable to observe the effect on another. In contrast, the empirical method doesn’t necessarily involve manipulation. Instead, it focuses on observing and collecting data in natural settings, offering a broader understanding of phenomena as they occur in real life.

Why the distinction matters

Understanding the difference between empirical and experimental methods is crucial because it affects how research is conducted and how results are interpreted. Empirical research can provide a more naturalistic view of the subject matter, whereas experimental research can offer more control over variables and potentially more precise outcomes.

The significance of experiential learning

The empirical method has deep roots in experiential learning, which emphasizes learning through experience. This connection is vital because it underlines the importance of engaging with the subject matter at a practical level, rather than just theoretically. It’s a hands-on approach to knowledge that has been valued since the time of Aristotle.

Developing theories from empirical research

One of the most significant aspects of the empirical method is its role in theory development . Researchers collect and analyze data, and from these findings, they can formulate or refine theories. Theories that are supported by empirical evidence tend to be more robust and widely accepted in the scientific community.

Applying the empirical method in various fields

The empirical method is not limited to the natural sciences. It’s used across a range of disciplines, from social sciences to humanities, to understand different aspects of the world. For instance:

• In psychology , researchers might use the empirical method to observe and record behaviors to understand the underlying mental processes.
• In sociology , it could involve studying social interactions to draw conclusions about societal structures.
• In economics , empirical data might be used to test the validity of economic theories or to measure market trends.

Challenges and limitations

Despite its importance, the empirical method has its challenges and limitations. One major challenge is ensuring that observations and data collection are unbiased. Additionally, not all phenomena are easily observable, and some may require more complex or abstract approaches.

The empirical method is a fundamental aspect of research methodology that has stood the test of time. By relying on observation and data collection, it allows researchers to ground their theories in reality, providing a solid foundation for knowledge. Whether it’s used in the hard sciences, social sciences, or humanities, the empirical method continues to be a critical tool for understanding our complex world.

How do you think the empirical method affects the credibility of research findings? And can you think of a situation where empirical methods might be difficult to apply but still necessary for advancing knowledge? Let’s discuss these thought-provoking questions and consider the breadth of the empirical method’s impact on the pursuit of understanding.

How useful was this post?

Click on a star to rate it!

Average rating / 5. Vote count:

No votes so far! Be the first to rate this post.

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Submit Comment

Research Methodology

1 Introduction to Research in General

• Research in General
• Research Circle
• Tools of Research
• Methods: Quantitative or Qualitative
• The Product: Research Report or Papers

2 Original Unity of Philosophy and Science

• Myth Philosophy and Science: Original Unity
• The Myth: A Spiritual Metaphor
• Myth Philosophy and Science
• The Greek Quest for Unity
• The Ionian School
• Towards a Grand Unification Theory or Theory of Everything
• Einstein’s Perennial Quest for Unity

3 Evolution of the Distinct Methods of Science

• Definition of Scientific Method
• The Evolution of Scientific Methods
• Theory-Dependence of Observation
• Scope of Science and Scientific Methods
• Prevalent Mistakes in Applying the Scientific Method

4 Relation of Scientific and Philosophical Methods

• Definitions of Scientific and Philosophical method
• Philosophical method
• Scientific method
• The relation
• The Importance of Philosophical and scientific methods

5 Dialectical Method

• Introduction and a Brief Survey of the Method
• Types of Dialectics
• Dialectics in Classical Philosophy
• Dialectics in Modern Philosophy
• Critique of Dialectical Method

6 Rational Method

• Understanding Rationalism
• Rational Method of Investigation
• Descartes’ Rational Method
• Leibniz’ Aim of Philosophy
• Spinoza’ Aim of Philosophy

7 Empirical Method

• Common Features of Philosophical Method
• Empirical Method
• Exposition of Empiricism
• Locke’s Empirical Method
• Berkeley’s Empirical Method
• David Hume’s Empirical Method

8 Critical Method

• Basic Features of Critical Theory
• On Instrumental Reason
• Conception of Society
• Human History as Dialectic of Enlightenment
• Substantive Reason
• Habermasian Critical Theory
• Habermas’ Theory of Society
• Habermas’ Critique of Scientism
• Theory of Communicative Action
• Discourse Ethics of Habermas

9 Phenomenological Method (Western and Indian)

• Phenomenology in Philosophy
• Phenomenology as a Method
• Phenomenological Analysis of Knowledge
• Phenomenological Reduction
• Husserl’s Triad: Ego Cogito Cogitata
• Intentionality
• Understanding ‘Consciousness’
• Phenomenological Method in Indian Tradition
• Phenomenological Method in Religion

10 Analytical Method (Western and Indian)

• Analysis in History of Philosophy
• Conceptual Analysis
• Analysis as a Method
• Analysis in Logical Atomism and Logical Positivism
• Analytic Method in Ethics
• Language Analysis
• Quine’s Analytical Method
• Analysis in Indian Traditions

11 Hermeneutical Method (Western and Indian)

• The Power (Sakti) to Convey Meaning
• Three Meanings
• Pre-understanding
• The Semantic Autonomy of the Text
• Towards a Fusion of Horizons
• The Hermeneutical Circle
• The True Scandal of the Text
• Literary Forms

12 Deconstructive Method

• The Seminal Idea of Deconstruction in Heidegger
• Deconstruction in Derrida
• Structuralism and Post-structuralism
• Sign Signifier and Signified
• Writing and Trace
• Deconstruction as a Strategic Reading
• The Logic of Supplement
• No Outside-text

13 Method of Bibliography

• Preparing to Write
• Writing a Paper
• The Main Divisions of a Paper
• Writing Bibliography in Turabian and APA
• Sample Bibliography

14 Method of Footnotes

• Citations and Notes
• General Hints for Footnotes
• Writing Footnotes
• Examples of Footnote or Endnote
• Example of a Research Article

15 Method of Notes Taking

• Methods of Note-taking
• Note Book Style
• Note taking in a Computer
• Types of Note-taking
• Notes from Field Research
• Errors to be Avoided

16 Method of Thesis Proposal and Presentation

• Preliminary Section
• Presenting the Problem of the Thesis
• Design of the Study
• Main Body of the Thesis
• Conclusion Summary and Recommendations
• Reference Material

Share on Mastodon

Summer is here, and so is the sale. Get a yearly plan with up to 65% off today! 🌴🌞

• Form Builder
• Survey Maker
• AI Form Generator
• AI Survey Tool
• AI Quiz Maker
• Store Builder
• WordPress Plugin

HubSpot CRM

Google Sheets

Google Analytics

Microsoft Excel

• Popular Forms
• Job Application Form Template
• Rental Application Form Template
• Hotel Accommodation Form Template
• Online Registration Form Template
• Employment Application Form Template
• Application Forms
• Booking Forms
• Consent Forms
• Contact Forms
• Donation Forms
• Customer Satisfaction Surveys
• Employee Satisfaction Surveys
• Evaluation Surveys
• Feedback Surveys
• Market Research Surveys
• Personality Quiz Template
• Geography Quiz Template
• Math Quiz Template
• Science Quiz Template
• Vocabulary Quiz Template

Try without registration Quick Start

Read engaging stories, how-to guides, learn about forms.app features.

Inspirational ready-to-use templates for getting started fast and powerful.

Spot-on guides on how to use forms.app and make the most out of it.

See the technical measures we take and learn how we keep your data safe and secure.

• Integrations
• Help Center
• Sign In Sign Up Free
• What is empirical research: Methods, types & examples

Defne Çobanoğlu

Having opinions on matters based on observation is okay sometimes. Same as having theories on the subject you want to solve. However, some theories need to be tested. Just like Robert Oppenheimer says, “Theory will take you only so far .” 

In that case, when you have your research question ready and you want to make sure it is correct, the next step would be experimentation. Because only then you can test your ideas and collect tangible information. Now, let us start with the empirical research definition:

• What is empirical research?

Empirical research is a research type where the aim of the study is based on finding concrete and provable evidence . The researcher using this method to draw conclusions can use both quantitative and qualitative methods. Different than theoretical research, empirical research uses scientific experimentation and investigation. 

Using experimentation makes sense when you need to have tangible evidence to act on whatever you are planning to do. As the researcher, you can be a marketer who is planning on creating a new ad for the target audience, or you can be an educator who wants the best for the students. No matter how big or small, data gathered from the real world using this research helps break down the question at hand. 

• When to use empirical research?

Empirical research methods are used when the researcher needs to gather data analysis on direct, observable, and measurable data. Research findings are a great way to make grounded ideas. Here are some situations when one may need to do empirical research:

1. When quantitative or qualitative data is needed

There are times when a researcher, marketer, or producer needs to gather data on specific research questions to make an informed decision. And the concrete data gathered in the research process gives a good starting point.

2. When you need to test a hypothesis

When you have a hypothesis on a subject, you can test the hypothesis through observation or experiment. Making a planned study is a great way to collect information and test whether or not your hypothesis is correct.

3. When you want to establish causality

Experimental research is a good way to explore whether or not there is any correlation between two variables. Researchers usually establish causality by changing a variable and observing if the independent variable changes accordingly.

• Types of empirical research

The aim of empirical research is to collect information about a subject from the people by doing experimentation and other data collection methods. However, the methods and data collected are divided into two groups: one collects numerical data, and the other one collects opinion-like data. Let us see the difference between these two types:

Quantitative research

Quantitative research methods are used to collect data in a numerical way. Therefore, the results gathered by these methods will be numbers, statistics, charts, etc. The results can be used to quantify behaviors, opinions, and other variables. Quantitative research methods are surveys, questionnaires, and experimental research.

Qualitiative research

Qualitative research methods are not used to collect numerical answers, instead, they are used to collect the participants’ reasons, opinions, and other meaningful aspects. Qualitative research methods include case studies, observations, interviews, focus groups, and text analysis.

• 5 steps to conduct empirical research

Necessary steps for empirical research

When you want to collect direct and concrete data on a subject, empirical research is a great way to go. And, just like every other project and research, it is best to have a clear structure in mind. This is even more important in studies that may take a long time, such as experiments that take years. Let us look at a clear plan on how to do empirical research:

1. Define the research question

The very first step of every study is to have the question you will explore ready. Because you do not want to change your mind in the middle of the study after investing and spending time on the experimentation.

2. Go through relevant literature

This is the step where you sit down and do a desk research where you gather relevant data and see if other researchers have tried to explore similar research questions. If so, you can see how well they were able to answer the question or what kind of difficulties they faced during the research process.

3. Decide on the methodology

Once you are done going through the relevant literature, you can decide on which method or methods you can use. The appropriate methods are observation, experimentation, surveys, interviews, focus groups, etc.

4. Do data analysis

When you get to this step, it means you have successfully gathered enough data to make a data analysis. Now, all you need to do is look at the data you collected and make an informed analysis.

5. Conclusion

This is the last step, where you are finished with the experimentation and data analysis process. Now, it is time to decide what to do with this information. You can publish a paper and make informed decisions about whatever your goal is.

• Empirical research methodologies

Some essential methodologies to conduct empirical research

The aim of this type of research is to explore brand-new evidence and facts. Therefore, the methods should be primary and gathered in real life, directly from the people. There is more than one method for this goal, and it is up to the researcher to use which one(s). Let us see the methods of empirical research: 

• Observation

The method of observation is a great way to collect information on people without the effect of interference. The researcher can choose the appropriate area, time, or situation and observe the people and their interactions with one another. The researcher can be just an outside observer or can be a participant as an observer or a full participant.

• Experimentation

The experimentation process can be done in the real world by intervening in some elements to unify the environment for all participants. This method can also be done in a laboratory environment. The experimentation process is good for being able to change the variables according to the aim of the study.

The case study method is done by making an in-depth analysis of already existing cases. When the parameters and variables are similar to the research question at hand, it is wise to go through what was researched before.

• Focus groups

The case study method is done by using a group of individuals or multiple groups and using their opinions, characteristics, and responses. The scientists gather the data from this group and generalize it to the whole population.

Surveys are an effective way to gather data directly from people. It is a systematic approach to collecting information. If it is done in an online setting as an online survey , it would be even easier to reach out to people and ask their opinions in open-ended or close-ended questions.

Interviews are similar to surveys as you are using questions to collect information and opinions of the people. Unlike a survey, this process is done face-to-face, as a phone call, or as a video call.

• Advantages of empirical research

Empirical research is effective for many reasons, and helps researchers from numerous fields. Here are some advantages of empirical research to have in mind for your next research:

• Empirical research improves the internal validity of the study.
• Empirical evidence gathered from the study is used to authenticate the research question.
• Collecting provable evidence is important for the success of the study.
• The researcher is able to make informed decisions based on the data collected using empirical research.
• Disadvantages of empirical research

After learning about the positive aspects of empirical research, it is time to mention the negative aspects. Because this type may not be suitable for everyone and the researcher should be mindful of the disadvantages of empirical research. Here are the disadvantages of empirical research:

• As it is similar to other research types, a case study where experimentation is included will be time-consuming no matter what. It has more steps and variables than concluding a secondary research.
• There are a lot of variables that need to be controlled and considered. Therefore, it may be a challenging task to be mindful of all the details.
• Doing evidence-based research can be expensive if you need to complete it on a large scale.
• When you are conducting an experiment, you may need some waivers and permissions.
• Frequently asked questions about empirical research

Empirical research is one of the many research types, and there may be some questions in mind about its similarities and differences to other research types.

Is empirical research qualitative or quantitative?

The data collected by empirical research can be qualitative, quantitative, or a mix of both. It is up to the aim of researcher to what kind of data is needed and searched for.

Is empirical research the same as quantitative research?

As quantitative research heavily relies on data collection methods of observation and experimentation, it is, in nature, an empirical study. Some professors may even use the terms interchangeably. However, that does not mean that empirical research is only a quantitative one.

What is the difference between theoretical and empirical research?

Empirical studies are based on data collection to prove theories or answer questions, and it is done by using methods such as observation and experimentation. Therefore, empirical research relies on finding evidence that backs up theories. On the other hand, theoretical research relies on theorizing on empirical research data and trying to make connections and correlations.

What is the difference between conceptual and empirical research?

Conceptual research is about thoughts and ideas and does not involve any kind of experimentation. Empirical research, on the other hand, works with provable data and hard evidence.

What is the difference between empirical vs applied research?

Some scientists may use these two terms interchangeably however, there is a difference between them. Applied research involves applying theories to solve real-life problems. On the other hand, empirical research involves the obtaining and analysis of data to test hypotheses and theories.

• Final words

Empirical research is a good means when the goal of your study is to find concrete data to go with. You may need to do empirical research when you need to test a theory, establish causality, or need qualitative/quantitative data. For example, you are a scientist and want to know if certain colors have an effect on people’s moods, or you are a marketer and want to test your theory on ad places on websites. 

In both scenarios, you can collect information by using empirical research methods and make informed decisions afterward. These are just the two of empirical research examples. This research type can be applied to many areas of work life and social sciences. Lastly, for all your research needs, you can visit forms.app to use its many useful features and over 1000 form and survey templates!

Defne is a content writer at forms.app. She is also a translator specializing in literary translation. Defne loves reading, writing, and translating professionally and as a hobby. Her expertise lies in survey research, research methodologies, content writing, and translation.

• Form Features
• Data Collection

Table of Contents

Related posts.

35+ Father's day quiz questions & answers (+Free templates)

Best B2B SaaS marketing strategies

Brand awareness survey: key points you should know

Alladdine Djaidani

Canvas | University | Ask a Librarian

• Library Homepage
• Arrendale Library

Empirical & Non-Empirical Research

• Empirical Research

Introduction: What is Empirical Research?

Quantitative methods, qualitative methods.

• Quantitative vs. Qualitative
• Reference Works for Social Sciences Research
• What is Non-Empirical Research?
• Contact Us!

 Call us at 706-776-0111

  Chat with a Librarian

  Send Us Email

  Library Hours

Empirical research  is based on phenomena that can be observed and measured. Empirical research derives knowledge from actual experience rather than from theory or belief. 

Key characteristics of empirical research include:

• Specific research questions to be answered;
• Definitions of the population, behavior, or phenomena being studied;
• Description of the methodology or research design used to study this population or phenomena, including selection criteria, controls, and testing instruments (such as surveys);
• Two basic research processes or methods in empirical research: quantitative methods and qualitative methods (see the rest of the guide for more about these methods).

(based on the original from the Connelly LIbrary of LaSalle University)

Empirical Research: Qualitative vs. Quantitative

Learn about common types of journal articles that use APA Style, including empirical studies; meta-analyses; literature reviews; and replication, theoretical, and methodological articles.

Academic Writer

© 2024 American Psychological Association.

• More about Academic Writer ...

Quantitative Research

A quantitative research project is characterized by having a population about which the researcher wants to draw conclusions, but it is not possible to collect data on the entire population.

• For an observational study, it is necessary to select a proper, statistical random sample and to use methods of statistical inference to draw conclusions about the population. 
• For an experimental study, it is necessary to have a random assignment of subjects to experimental and control groups in order to use methods of statistical inference.

Statistical methods are used in all three stages of a quantitative research project.

For observational studies, the data are collected using statistical sampling theory. Then, the sample data are analyzed using descriptive statistical analysis. Finally, generalizations are made from the sample data to the entire population using statistical inference.

For experimental studies, the subjects are allocated to experimental and control group using randomizing methods. Then, the experimental data are analyzed using descriptive statistical analysis. Finally, just as for observational data, generalizations are made to a larger population.

Iversen, G. (2004). Quantitative research . In M. Lewis-Beck, A. Bryman, & T. Liao (Eds.), Encyclopedia of social science research methods . (pp. 897-898). Thousand Oaks, CA: SAGE Publications, Inc.

Qualitative Research

What makes a work deserving of the label qualitative research is the demonstrable effort to produce richly and relevantly detailed descriptions and particularized interpretations of people and the social, linguistic, material, and other practices and events that shape and are shaped by them.

Qualitative research typically includes, but is not limited to, discerning the perspectives of these people, or what is often referred to as the actor’s point of view. Although both philosophically and methodologically a highly diverse entity, qualitative research is marked by certain defining imperatives that include its case (as opposed to its variable) orientation, sensitivity to cultural and historical context, and reflexivity. 

In its many guises, qualitative research is a form of empirical inquiry that typically entails some form of purposive sampling for information-rich cases; in-depth interviews and open-ended interviews, lengthy participant/field observations, and/or document or artifact study; and techniques for analysis and interpretation of data that move beyond the data generated and their surface appearances. 

Sandelowski, M. (2004).  Qualitative research . In M. Lewis-Beck, A. Bryman, & T. Liao (Eds.),  Encyclopedia of social science research methods . (pp. 893-894). Thousand Oaks, CA: SAGE Publications, Inc.

• Next: Quantitative vs. Qualitative >>
• Last Updated: Jul 24, 2024 12:04 PM
• URL: https://library.piedmont.edu/empirical-research
• Ebooks & Online Video
• New Materials
• Renew Checkouts
• Faculty Resources
• Library Friends
• Library Services
• Our Mission
• Library History
• Ask a Librarian!
• Making Citations
• Working Online

Arrendale Library Piedmont University 706-776-0111

La Salle University

Connelly library, library main menu.

• Course Reserves
• Interlibrary Loan (ILL)
• Study Room Use & Reservations
• Technology & Printing
• Citation Guides
• Reserve Library Space
• Request Instruction
• Copyright Information
• Guides for Faculty

Special Collections

• University Archives
• Historical & Cultural Collections
• Rare Bibles & Prayer Books
• Historical Research Guides
• Information & Guidelines
• Staff Directory
• Meet with a Librarian
• Directions & Building Maps

Research Hub

• Research Tools
• Research Guides

Qualitative and Quantitative Research

What is "empirical research".

• empirical research
• Locating Articles in Cinahl and PsycInfo
• Locating Articles in PubMed
• Getting the Articles

Empirical research  is based on observed and measured phenomena and derives knowledge from actual experience rather than from theory or belief. 

How do you know if a study is empirical? Read the subheadings within the article, book, or report and look for a description of the research "methodology."  Ask yourself: Could I recreate this study and test these results?

Key characteristics to look for:

• Specific research questions  to be answered
• Definition of the  population, behavior, or   phenomena  being studied
• Description of the  process  used to study this population or phenomena, including selection criteria, controls, and testing instruments (such as surveys)

Another hint: some scholarly journals use a specific layout, called the "IMRaD" format, to communicate empirical research findings. Such articles typically have 4 components:

• Introduction : sometimes called "literature review" -- what is currently known about the topic -- usually includes a theoretical framework and/or discussion of previous studies
• Methodology:  sometimes called "research design" --  how to recreate the study -- usually describes the population, research process, and analytical tools
• Results : sometimes called "findings"  --  what was learned through the study -- usually appears as statistical data or as substantial quotations from research participants
• Discussion : sometimes called "conclusion" or "implications" -- why the study is important -- usually describes how the research results influence professional practices or future studies
• << Previous: Home
• Next: Locating Articles in Cinahl and PsycInfo >>

Chat Assistance

15 Empirical Evidence Examples

Dave Cornell (PhD)

Dr. Cornell has worked in education for more than 20 years. His work has involved designing teacher certification for Trinity College in London and in-service training for state governments in the United States. He has trained kindergarten teachers in 8 countries and helped businessmen and women open baby centers and kindergartens in 3 countries.

Learn about our Editorial Process

Chris Drew (PhD)

This article was peer-reviewed and edited by Chris Drew (PhD). The review process on Helpful Professor involves having a PhD level expert fact check, edit, and contribute to articles. Reviewers ensure all content reflects expert academic consensus and is backed up with reference to academic studies. Dr. Drew has published over 20 academic articles in scholarly journals. He is the former editor of the Journal of Learning Development in Higher Education and holds a PhD in Education from ACU.

The term empirical evidence refers to the attainment of knowledge through observation, measurement, or experimentation ( also known as empiricism ). It is accumulated through systematic observations of phenomena in natural or laboratory setting.

One of the key standards of empirical evidence in academic research is that the results can be tested and verified by others. This can increase confidence in the conclusion and demonstrate that there is substantial evidence to accept something as a natural fact.

By contrast, anecdotal evidence , which is considered unclear and open to bias, might be a form of one-off evidence that wouldn’t be accepted unless it’s replicable. Confidence in the conclusion of anecdotal evidence is minimal.

Empirical Evidence Examples

• Quantitative Data: Quantitative data is numerical data obtained through measurement, counting, or statistical analysis. An example is a person’s test scores, which are used as empirical evidence that can get you into a prestigious university. The strength of this type of data is that it tends to be objective, meaning it is less disputable than qualitative data.
• Qualitative Data : Qualitative data is non-numerical data that captures the characteristics or properties of something. Examples include interview transcripts, field notes, and descriptions of behaviors or events. Its strength is that it can lead to in-depth explanations and understandingsthat quantitative data often cannot support.
• Survey Data (Quantitative): Survey data, such as polling data in the lead up to an election, can be used as a form of empirical evidence if the study is reliable and valid. When the survey size is sufficient and population sample is sufficiently uniform, the data may be able to make population-wide predications about society and social views.
• Data from Naturalistic Observation (Qualitative): Naturalistic observations are observations that we can make in the real world, not just in a lab environment. It can pass as empirical evidence if it’s repeatable over time and renders similar results. However, if the observation is made only once and future studies do not support claims made by the original observer, it may slip into the category of anecdotal evidence.
• Case Study Data (Qualitative): Sometimes the first step to understanding a psychological disorder is to thoroughly examine how it is manifest in a single individual. This is referred to as a case study. This data is empirical and may be very sound, but its lack of population-level relevance limits is usability.
• Textual Data (Quantitative or Qualitative): Although it is not possible to read people’s minds yet, the next best thing is to ask people to write their thoughts down on paper. Those words can then be coded along a variety of dimensions to develop trends across a textual dataset ( also known as textual analysis and thematic analysis).
• Experimental Lab Data (Quantitative) : Randomly assigning research participants to receive different treatments is a hallmark of scientific research. By manipulating the level of one variable and observing changes in another, the researcher can draw conclusions regarding causality. Many consider this type of research as the most scientifically sound method of attaining empirical evidence.  
• Longitudinal Research Data (Quantitative or Qualitative): This strategy for acquiring empirical evidence involves collecting data on a particular trait over a long period of time. Researchers will administer the same measurement tool at different points in the participants’ lifespan. This can provide valuable clues regarding the stability of personality characteristics or intellectual abilities.
• Cross-sectional Data (Quantitative or Qualitative): Cross-sectional data is data collected from different subjects at a single point in time. For example, a national census usually generates a cross-sectional dataset of a nation’s population at a specific point in time by asking everyone to complete the census on the same day.
• Historical Data (Quantitative or Qualitative) : Historical data is empirical evidence collected from past records or documents that can provide valuable context and insight into past events or societal trends. Examples include analyzing economic data from past decades to understand the causes of financial crises or examining the diaries of individuals who lived through significant historical events to gain a deeper understanding of their experiences.
• Meta-Analysis Data (Generally Quantitative) : Meta-analysis is a quantitative technique that combines the results of multiple studies on a similar topic to derive an overall conclusion. This type of empirical evidence can provide a more reliable and generalizable understanding of a phenomenon by aggregating the findings of individual studies, reducing the influence of individual study biases, and increasing statistical power.
• Ethnographic Data (Qualitative) : Ethnographic data is a form of qualitative data that provides evidence recorded by an anthropologist or similar researcher. While this data gives extremely in-depth understandings (often called ‘think descriptions’), its inability to be replicated means it lacks the authority of many other types of empirical research provided in this list. Examples include studying the daily lives of a remote tribe or exploring the workplace culture of a specific organization.
• Computer Simulation Data (Quantitative) : Computer simulations can be used to model complex systems or processes, providing empirical evidence that may not be easily obtained through direct observation or experimentation. Examples include modeling the spread of infectious diseases to inform public health interventions, or simulating the effects of climate change on ecosystems.
• Physiological Measurement Data (Quantitative) : Physiological measurements are the empirical data that result from the recording of physical or biological signals from the body. This can provide evidence about a person’s physical state and whether it fits within physiological norms required for healthy living. Examples include measuring heart rate or skin conductance to assess stress levels.
• Cultural Artifacts (Quantitative or Qualitative) : Cultural artifacts and provide powerful empirical evidence about past cultures. For example, the etchings of Aboriginal rock art in Australia has been valuable in providing clear evidence about the historical longevity of the world’s oldest continuous culture.

Case Studies of Empirical Evidence  

1. evidence on who shares their passwords.

Evidence Collected by: Surveys

We all know that sharing passwords is risky. Experts would like to know who is most likely to engage in this risky behavior. As a group of these professionals sit around the table debating the issue, they quickly realize that everyone can provide good arguments to support their opinion.

So, how can this question be answered objectively?

The answer: through the scientific method .

To this end, Whitty et al. (2015) measured several personality characteristics of 630 internet users in the UK. Participants were administered questionnaires that assessed Impulsivity, Self-monitoring, and Locus of Control.

Age and knowledge of cyber security issues were also measured.

The results were sometimes surprising:

• Younger people were more likely to share passwords than older people.
• Those who scored high on a lack of perseverance (i.e., impulsivity) were more likely to share passwords.
• Knowledge about cybersecurity did not distinguish between those who share passwords and those who do not share passwords.

The researchers concluded that:

“psychology plays an important role in providing answers to why individuals engage in cyber security practices” (p. 6).

It should be noted that several of the researchers’ hypotheses were not supported by the data. This points to a key reason empirical evidence is so valuable.

2. Linguistic Inquiry and Word Count (LIWC)  

Evidence Collected by: Text Analysis

Language is the most common way that people communicate their internal thoughts and emotions. Language is key to business, relationships, scientific innovation and nearly every facet of human existence.

Studying people through language is the way that cognitive, clinical, and social psychologist try to understand human behavior .

Today, communication via texting has never been easier, offering researchers an opportunity like never before. In the old day, text analysis was conducted by hand. Weintraub (1981, 1989) pioneered this approach by hand, analyzing political speeches and medical interviews.

Tausczik and Pennebaker’s (2010) pay respects to Weintraub’s work:

“He noticed that first-person singular pronouns (e.g., I, me, my) were reliably linked to people’s levels of depression. Although his methods were straightforward and his findings consistently related to important outcome measures, his work was largely ignored” (p. 26).

Fortunately, the volume of texts today can be handled with the use of linguistic technology.

The LIWC is unique in that it has the capability of analyzing text that can

“…provide important psychological cues to their thought processes, emotional states, intentions, and motivations…that reflect language correlates of attentional focus, emotional state, social relationships, thinking styles, and individual differences” (p. 37).

Human coding is subject to bias, but empirical evidence via computer technology is much more objective.

3. Measuring Neural Activity as Empirical Evidence

Evidence collected by: Physiological instruments

One of the most common methods of measuring the brain’s neural activity in psychological research is the EEG (electroencephalogram). A few electrodes attached to the scalp can measure this activity in real time.

This kind of data collection technique has allowed researchers to study a wide range of psychological phenomena, such as memory, attention span, and emotions.

In one interesting application, Wong et al. (2007) examined if music-related experience could enhance the processing of a tonal language such as Mandarin.

After all, music involves a lot of tones, as does Mandarin. Therefore, it would seem logical that musicians would be better at processing the sounds of Mandarin than non-musicians.

So, musicians and non-musicians watched a video that contained audio recordings of Mandarin while EEG data were collected. 

The results showed that the auditory brainstem regions of musicians

“…showed more faithful representation of the stimulus…Musicians showed significantly better identification and discrimination” (p. 421).

More simply put, the brains of musicians encoded the tones of Mandarin more accurately than the brains of non-musicians.

When empirical evidence is gathered using high-tech equipment, it lends a lot of credibility to the findings.

4. Reaction Time and Semantic Memory 

Evidence collected by: Computational data

A frequently used measure of cognitive processing is called “reaction time.” This is a precise measurement of how long it takes for a person to process a specific piece of information. 

For example, a research participant is presented with two words that are either related or unrelated.

If the two words are related, then they press one key. If the words are unrelated, then they press a different key.

The computer records how long it takes between the presentation of the words and the key press.

In one of the most influential studies in cognitive psychology , Collins and Loftus (1975) built a foundation of evidence suggesting that semantic information is stored in memory based on strengths of associations.

The reaction time of processing words that were related was much faster than words unrelated. This is because related words are more strongly connected in the memory network:

“The more properties two concepts have in common, the more links there are between the two nodes via these properties and the more closely related are the concepts…When a concept is processed (or stimulated), activation spreads out along the paths of the network in a decreasing gradient” (p. 411).

Empirical evidence in the form of reaction time is not subject to bias and the precision of measurement is quite impressive.  

5. Classroom Décor and Learning

Evidence collected by: Naturalistic observation

Most teachers enjoy decorating the classroom environment with educational posters, student artwork, and theme-based materials. But if you were to tell them that those decorations actually impair learning, it might be a tough sell.

However, Fisher et al. (2014) have empirical evidence suggesting this is a real possibility.

Visual stimuli can be distracting, especially to young children. They already have short attention spans.

To put the hypothesis to the test, 24 kindergarten students participated in six lessons over a two-week period; the classroom was either fully decorate or sparsely decorated.

The lessons were recorded and coded for on-task and off-task behaviors. In addition, students took a test immediately after each lesson.

“…spent significantly more instructional time off task in the decorated-classroom condition than in the sparse-classroom condition…learning scores were higher in the sparse-classroom condition than in the decorated-classroom condition” (p. 6).

Empirical evidence isn’t perfect, and every study has limitations, but it is far more objective than opinions based on subjective judgements.

Empirical evidence is attained objectively through methods that adhere to rigorous scientific standards.

The precision of empirical evidence is quite wide. On one end of the continuum, data collected from surveys simply involves participants circling a number that is supposed to reflect their attitude.

On the other end, computer programs can be designed that track how long it takes a person to process a stimulus down to milliseconds.

But perhaps the greatest value of empirical evidence is that it can resolve debates. Smart people can generate convincing arguments to support their opinions on either side of an issue.

However, evidence that comes from scientific methods can settle those debates. Even if it takes several studies to arrive at a firm conclusion, the end result helps move science forward.

Collins, A. M., & Quillian, M. R. (1969). Retrieval time from semantic memory . Journal of Verbal Learning and Verbal Behavior , 8 (2), 240-247.

Collins, W. M., & Loftus, E. F. (1975). A spreading-activation theory of semantic processing. Psychological Review, 82 (6), 407-428.

Fisher, A. V., Godwin, K. E., & Seltman, H. (2014). Visual environment, attention allocation, and learning in young children: When too much of a good thing may be bad. Psychological Science , 25 (7), 1362-1370.

Hacking, I. (1983). Representing and intervening . Cambridge: Cambridge University Press.

Malik, S. (2017). Observation versus experiment: An adequate framework for analysing scientific experimentation? Journal for General Philosophy of Science, 48 , 71–95. https://doi.org/10.1007/s10838-016-9335-y

Iliev, R., Dehghani, M., & Sagi, E. (2015). Automated text analysis in psychology: Methods, applications, and future developments. Language and Cognition , 7 (2), 265-290.

Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology , 29 (1), 24-54.

Weintraub, W. (1981). Verbal behavior: Adaptation and psychopathology . New York: Springer.

Weintraub, W. (1989). Verbal behavior in everyday life . New York: Springer

Whitty, M., Doodson, J., Creese, S., & Hodges, D. (2015). Individual differences in cyber security behaviors: an examination of who is sharing passwords. Cyberpsychology, Behavior, and Social Networking , 18 (1), 3-7.

Wong, P. C., Skoe, E., Russo, N. M., Dees, T., & Kraus, N. (2007). Musical experience shapes human brainstem encoding of linguistic pitch patterns. Nature Neuroscience , 10 (4), 420–422. https://doi.org/10.1038/nn1872

• Dave Cornell (PhD) https://helpfulprofessor.com/author/dave-cornell-phd/ 23 Achieved Status Examples
• Dave Cornell (PhD) https://helpfulprofessor.com/author/dave-cornell-phd/ 25 Defense Mechanisms Examples
• Dave Cornell (PhD) https://helpfulprofessor.com/author/dave-cornell-phd/ 15 Theory of Planned Behavior Examples
• Dave Cornell (PhD) https://helpfulprofessor.com/author/dave-cornell-phd/ 18 Adaptive Behavior Examples

• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 23 Achieved Status Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 15 Ableism Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 25 Defense Mechanisms Examples
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 15 Theory of Planned Behavior Examples

Leave a Comment Cancel Reply

Your email address will not be published. Required fields are marked *

Empirical Evidence

Information obtained through observation and documentation of certain behavior and patterns or through an experiment

What is Empirical Evidence?

Empirical evidence is the information obtained through observation and documentation of certain behavior and patterns or through an experiment. Empirical evidence is a quintessential part of the scientific method of research that is applicable in many disciplines.

In the scientific method, the pieces of empirical evidence are used to validate or disprove a stated hypothesis , statement, or claim. In the scientific world, a hypothesis can be accepted by the community only if sufficient (empirical) evidence that supports the hypothesis is provided.

Breaking Down the Evidence

Empirical evidence is primarily obtained through observation or experimentation. The observations or experiments are known as primary sources. However, it can also be obtained through various secondary sources, including articles, reports , newspapers, etc. The process of finding empirical evidence is called empirical research.

The main concern with empirical research is the collection of unbiased evidence. Researchers must carefully design the research while minimizing exposure to potential errors. In the scientific world, it is common that several scientists or researchers gather evidence simultaneously by replicating the same study. In addition, peer review is a primary tool in science that is used to validate the evidence provided in a study or research.

Types of Empirical Evidence

The two primary types of empirical evidence are qualitative evidence and quantitative evidence.

1. Qualitative

Qualitative evidence is the type of data that describes non-measurable information. Qualitative data is used in various disciplines, notably in social sciences, as well as in market research and finance . In such fields, the research generally investigates human behavior and its patterns. The non-measurable nature of qualitative data, as well as its subjective analysis, makes it prone to potential biases.

2. Quantitative

Quantitative evidence refers to numerical data that can be further analyzed using mathematical and/or statistical methods . Quantitative data is used in almost every discipline of science.

Unlike qualitative data, the evidence obtained using quantitative data is generally considered to be unbiased as the validity of the data can be easily verified using mathematical/statistical calculations or analysis.

Related Readings

CFI is the official provider of the Business Intelligence & Data Analyst (BIDA)®  certification program, designed to transform anyone into a world-class financial analyst.

To keep learning and developing your knowledge of financial analysis, we highly recommend the additional CFI resources below:

• Basic Statistics Concepts in Finance
• Durability Bias
• Monte Carlo Simulation
• Survivorship Bias
• See all data science resources
• Share this article

Create a free account to unlock this Template

Access and download collection of free Templates to help power your productivity and performance.

Already have an account? Log in

Supercharge your skills with Premium Templates

Take your learning and productivity to the next level with our Premium Templates.

Upgrading to a paid membership gives you access to our extensive collection of plug-and-play Templates designed to power your performance—as well as CFI's full course catalog and accredited Certification Programs.

Already have a Self-Study or Full-Immersion membership? Log in

Access Exclusive Templates

Gain unlimited access to more than 250 productivity Templates, CFI's full course catalog and accredited Certification Programs, hundreds of resources, expert reviews and support, the chance to work with real-world finance and research tools, and more.

Already have a Full-Immersion membership? Log in

What is Empirical Evidence?

Published Jan 5, 2018

Empirical evidence is information that researchers generate to help uncover answers to questions that can have significant implications for our society.

Take seatbelts. Prior to their invention, people were killed or maimed in what today we would think of as minor traffic accidents. So smart engineers put their heads together to try to do something about it.

Let’s try tying people down! Let’s change what the steering wheel is made of! Let’s put an exploding bag of air in the steering wheel! (Imagine how crazy that sounded in a pitch meeting.) These all seem like reasonable ideas (well except that exploding airbag one), so how do we know which one we should do?

The answer is to generate and weigh empirical evidence.

Theory vs. Empirical Evidence

One might have a theory about how something will play out, but what one observes or experiences can be different from what a theory might predict. People want to know the effectiveness of all sorts of things, which means they have to test them.

Social scientists produce empirical evidence in a variety of ways to test theories and measure the ability of A to produce an expected result: B.

Usually, researchers collect data through direct or indirect observation, and they analyze these data to answer empirical questions (questions that can be answered through observation).

Let’s look at our car safety example. Engineers and scientists equipped cars with various safety devices in various configurations, then smashed them into walls, poles and other cars and recorded what happened. Over time, they were able to figure out what types of safety devices worked and which ones didn’t. As it turns out, that whole airbag thing wasn’t so crazy after all.

They didn’t get everything right immediately. For instance, early seatbelts weren’t retractable. Some airbags shot pieces of metal into passengers . But, in fits and in starts, auto safety got better, and even though people are driving more and more miles, fewer and fewer are dying on the road .

How Gathering Empirical Evidence in Social Science is Different

Testing the effects of, say, a public policy on a group of people puts us in the territory of social science.

For instance, education research is not the same as automotive research because children (people) aren’t cars (objects). Education, though, can be made better by attempting new things, gathering data on those efforts, rigorously analyzing that data and then weighing all available empirical evidence to see if those new things accomplish what we hope they do.

Unfortunately, the “rigorously analyzing” bit is often missing from education research. In the labs of automobile engineers, great care is taken to only change one bit of design (a variable) at a time so that each test isolates the individual factor that is making a car more or less safe. OK, for this test, let’s just change the material of the steering wheel and keep everything else the same, so we’ll know if it is the wheel that is hurting people.

Comparing Apples with Apples

In social science and especially in education, trying to isolate variables is challenging, but possible, if researchers can make “apples-to-apples” comparisons.

The best way to get an apples-to-apples comparison is to perform something called a randomized control trial (RCT). You might have heard about these in relation to the testing of medicine. Drug testing uses RCTs all the time.

In an educational RCT, students are divided into two groups by a randomized lottery and half of the students receive whatever the educational “treatment” is (a new reading program, a change in approach to discipline, a school voucher, etc.) while the other does not. Researchers compare the results of those two groups and estimate the “treatment” effect. This approach gives us confidence that the observed effect is caused by the intervention and no other factors.

RCTs are not always possible. Sometimes researchers can get close by using random events that separate kids into two groups, such as school district boundaries that are created by rivers or creeks that split a community more or less by chance or birthday cutoffs for preschool that place a child born on August 31st in one grade but one born September 1st in another even though there is basically no difference between them. Depending on the exact nature of the event, these can be known as “regression discontinuity” or “instrumental variable” analyses, and they can be useful tools to estimate the effects of a program.

Researchers can also follow individual children that receive a treatment if they have data from before and after to see how that child’s educational trajectory changes over time. These are known as “fixed effects” analyses.

All three of these—randomized control trials, regression discontinuity analyses and fixed effects analyses —have their drawbacks.

Very few outside events are truly random. If, as regression discontinuity analysis often does, researchers only look at children just above or just below the cutoff, or, as fixed effects analysis often does, researchers look at only those children who switch from one school to another, those children might not be representative of the population. How would an intervention affect kids who are not close to a cutoff or border? Or kids who do not switch schools?

In the SlideShare below, we present empirical evidence based on rigorous research on private school choice programs as an example of how we, as academics and researchers ourselves, identify and characterize the high-quality empirical evidence in a given area of study.

[Slideshare no longer available]

A Couple Considerations

It’s a lot to wade through, so before you do, we’d like to offer two notes.

First, it is always important to understand the tradeoffs between internal and external validity.

Internal validity refers to how well a study is conducted—it gives us confidence that the effects we observe can be attributed to the intervention or program, not other factors.

For example, when the federal government wanted to know if Washington, D.C.’s school voucher program increased students’ reading and math test scores, researchers took the 2,308 students who applied for the program and randomly assigned 1,387 to get vouchers and 921 not to . They then followed the two groups over time, and when they analyzed the results, they could reasonably conclude that any differences were due to the offer of a voucher, because that is the only thing that was different between the two groups and they were different only because of random chance. This study had high internal validity.

External validity refers to the extent that we can generalize the findings from a study to other settings.

Let’s think about that same study. The D.C. program was unique. The amount of money that students receive, the regulations that participating schools had to agree to, the size of the program, its politically precarious situation and numerous other factors were different in that program than in others, not to mention the fact that Washington, D.C. is not representative of the United States as a whole demographically, politically or in really any way we can possibly imagine. As a result, we have to be cautious when we try to generalize the findings. The study has lower external validity.

To combat issues around lower external validity, researchers can collect and analyze empirical evidence on program design to understand its impact. We can also look at multiple studies to see how similar interventions affect students in different settings.

Second, the respect and use of research does not endorse technocracy. Research and expertise is incredibly useful. When you get on an airplane or head into surgery, you want the person who is doing the work to be an expert. Empirical evidence can help us know more about the world and be better at what we do. But we should also exercise restraint and humility by recognizing the limits of social science.

Public policy involves weighing tradeoffs that social science cannot do for us. Social science can tell us that a program increases reading scores but also increases anxiety and depression in children. Should that program be allowed to continue? Ultimately, that comes down to human judgment and values. That should never be forgotten.

With that, we hope you found this article helpful. Please feel free to reach out with any questions by emailing [email protected] or posting your question in the comments section below.

Michael Q. McShane

Director of national research, edchoice.

Dr. Michael McShane is Director of National Research at EdChoice. He is the author, editor, co-author, or co-editor eleven books on education policy, including his most recent Hybrid Homeschooling: A Guide to the Future of Education (Rowman and Littlefield, 2021) He is currently an opinion contributor to Forbes, and his analyses and commentary have been published widely in the media, including in USA Today, The Washington Post, and the Wall Street Journal. He has also been featured in education-specific outlets such as Teachers College Record, Education Week, Phi Delta Kappan, and Education Next. In addition to authoring numerous white papers, McShane has had academic work published in Education Finance and Policy, The Handbook of Education Politics and Policy, and the Journal of School Choice. A former high school teacher, he earned a Ph.D. in education policy from the University of Arkansas, an M.Ed. from the University of Notre Dame, and a B.A. in English from St. Louis University.

Will Flanders

Education research director, wisconsin institute for law & liberty.

Will Flanders is the education research director at the Wisconsin Institute for Law & Liberty. He holds a Ph.D. in Political Science with a specialization in American Politics and Public Policy, an M.S. in Political Science and an M.S. in Applied American Politics and Policy from Florida State University.

2024 Schooling in America Survey Is Out Now!

Colyn Ritter

Updated August 19, 2024

Cell Phones in Schools Are a Complicated Issue, New Poll Finds

Updated August 8, 2024

RECEIVE EDUCATIONAL CHOICE UPDATES STRAIGHT TO YOUR INBOX

• First Name *
• Last Name *
• © 2024 EdChoice
• Privacy Policy
• Terms of Use

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Psicol Reflex Crit
• v.32; 2019 Dec

Empirically supported interventions in psychology: contributions of Research Domain Criteria

Rita pasion.

1 Laboratory of Neuropsychophysiology, Faculty of Psychology and Educational Sciences, University of Porto, Rua Alfredo Allen, 535, 4200-135 Porto, Portugal

Eva C. Martins

2 Department of Social and Behavioural Sciences (CPUP), University of Porto, Porto, Portugal

3 Maia University Institute (ISMAI), Maia, Portugal

Fernando Barbosa

Associated data.

Not applicable

Empirically supported interventions in psychological disorders should provide (1) evidence supporting the underlying psychological mechanisms of psychopathology to target in the intervention and (2) evidence supporting the efficacy of the intervention. However, research has been dedicated in a greater extent to efficacy than to the acquisition of empirical support for the theoretical basis of therapies. Research Domain Criteria (RDoC) emerges as a new framework to provide empirically based theories about psychological mechanisms that may be targeted in intervention and tested for its efficacy. The current review aims to demonstrate the possible applications of RDoC to design empirically supported interventions for psychological disorders. Two RDoC-inspired interventions are reviewed, and the RDoC framework is broadly explored in terms of its contributions and limitations. From preliminary evidence, RDoC offers many avenues for improving evidence-based interventions in psychology, but some limitations must be anticipated to increase the RDoC applicability to naturalistic settings.

Introduction

The search for empirically supported interventions of psychological disorders has marked a paradigm shift in applied psychology. This movement followed changes in other areas like medicine and resulted from the need to provide clinicians with clear directives regarding the psychological treatments that were effective and the ones that were not, or even worse, were harmful. Hence, APA’s Presidential Task Force on Evidence-Based Practice in Psychology stressed the importance of incorporating empirically supported treatments with clinical expertise in the context of patient characteristics, culture, and preferences (Levant, 2005 ). Although APA accepts many forms of research evidence (Levant, 2005 ), most of the studies are centered on whether the treatments work and in what conditions (Roberts, Blossom, Evans, Amaro, & Kanine, 2017 ).

Nevertheless, the focus on efficacy in evidence-based interventions in psychological disorders (e.g., if the intervention is associated with symptom reduction) has not been paralleled with the acquisition of empirical support for the theoretical basis of these therapies (Emmelkamp et al., 2014 ). Evidence-based interventions should have an empirical background supporting the psychological mechanisms of psychopathology targeted for intervention and not only present the efficacy of the therapeutic package (David & Montgomery, 2011 ).

The smaller emphasis on background evidence may be explained by the state of the art in psychology and the yet incipient comprehension of mental disorders. For one, there is a wide variety of models regarding mental disorders and, subsequently, longstanding debates in the field. For the other, psychological and neurodevelopmental research lacks integration into a comprehensive conceptual framework of mechanisms in mental disorders, which constitutes a roadblock for designing empirically sound interventions in psychology (Emmelkamp et al., 2014 ). In this sense, the Research Domain Criteria (RDoC) movement emerges as a framework for dealing with these limitations.

The goal of this review is to explore the RDoC and discuss its relevance for designing empirically supported interventions in psychological disorders. We will start by introducing the RDoC framework and by examining its practical applications in the clinical setting. Then, two psychological interventions—training for awareness resilience and action (TARA) and Engage—that claim to be inspired by the RDoC framework in terms of its empirically based foundations and preliminary evidence on efficacy will be used to illustrate some of the possible applications of RDoC in psychological clinical settings. Then, the final section of this article aims to provide a critical analysis of RDoC applicability to real-life psychological interventions, from the observed advances and obstacles exposed by TARA and Engage at conceptual, methodological, and efficacy measurement levels.

The RDoC framework

RDoC was launched by the National Institute of Mental Health in 2009 as a new approach towards an empirically based model of psychopathology (Cuthbert & Insel, 2013 ; Insel et al., 2010 ) that is bound beyond experts’ opinion (e.g., DSM) (Appelbaum, 2017 ; Clark, Cuthbert, Lewis-Fernández, Narrow, & Reed, 2017 ). Thus, the RDoC framework was envisioned to provide a comprehensive understanding of psychopathology (and its mechanisms) based on empirical findings, ultimately allowing for the development of reliable diagnosis, as well as evidence-based pharmacological and psychological interventions (Morris, Rumsey, & Cuthbert, 2014 ). Although it may be argued that decision-making may still be biased and influenced by external factors, the decisions of the RDoC Task Forces are necessarily supported by the available research.

These efforts gave rise to a matrix that is the heart of RDoC. The RDoC matrix depicts biopsychological processes (represented by constructs within higher-order functional domains) and their expression (represented by units of analysis) to assess phenotypic expressions of psychopathology alongside with their interconnected mechanisms (for an overview of the RDoC matrix see https://www.nimh.nih.gov/research-priorities/rdoc/constructs/rdoc-matrix.shtml ). At this point, it is important to note that the current matrix is a work in progress and not a final/static description of the human biopsychological aspects (Clark et al., 2017 ). As the framework provided by the RDoC matrix is dependent on the emergent empirical findings, the matrix had several revisions to date. Until recently, there were five functional domains in the RDoC matrix—positive valence systems, negative valence systems, cognitive control systems, arousal and regulatory systems, and social processes systems—but at the beginning of 2019 the Sensory Systems domain was added to encourage research focused on the role of motor systems disruptions across mental disorders (for a full description of the domains and constructs see https://www.nimh.nih.gov/research-priorities/rdoc/constructs/index.shtml ).

Each of these six domains is divided into constructs capturing more specific dimensions (e.g., loss is a construct of the negative valence system domain), which may be used to investigate each mechanism of biopsychological functioning. For this purpose, the constructs are divided into units of analysis. These units of analysis consist of different methods—from molecules and neuronal-circuit activity to behavior and self-reports—to measure the expression of the underlying constructs. The inclusion of distinct unit of analysis represents an effort to encourage translational research and makes the RDoC matrix a multidisciplinary entity, allowing for multi-level analysis (Cuthbert & Insel, 2013 ) and for aggregating research streaming from biology and neuroscience to psychology (Kozak & Cuthbert, 2016 ). Currently, an investigation from these different areas is rarely integrated into a causal model of psychopathology. Most basic and applied research is disconnected leading to major breakthroughs in the development of better interventions (Morris et al., 2014 ).

Importantly, the constructs are defined as having at one end of the continuum its normal range and at the other end of the continuum its psychopathological forms, as well as having a transdiagnostic identity that cuts across mental disorders as they are defined today (e.g., DSM) (Kozak & Cuthbert, 2016 ). In this sense, RDoC rethinks psychopathology from the perspective of domains of function and basic mechanisms and does not consider psychological disorders as currently operationalized in a cluster of symptoms. This new approach may promote a better understanding of psychopathology that will eventually take us to better and earlier diagnosis and improved and tailored interventions (Clark et al., 2017 ).

In sum, by providing a “map” of constructs that can be tested simultaneously along many levels of analysis, RDoC allows to obtain evidence-based validity for mental disorders and its mechanisms, theories about development and developmental psychopathology, and finally, for evidence-based interventions. Specifically, the last topic will be addressed in more detail in the next sections.

RDoC empirically based interventions

As explained in the previous section, the RDoC-inspired research gave way to the construction of the RDoC matrix, a framework for studying relevant biopsychological constructs and its behavioral manifestations. The main goal was to integrate knowledge that was being acquired from different fields of study (e.g., psychology, neuroscience, biology) so that better interventions would be available for the treatment of mental disorders (Cuthbert & Insel, 2013 ; Morris et al., 2014 ).

Of special relevance for the topic of this article, RDoC is a new path towards providing validity evidence for the theory about psychological mechanisms of psychopathology and towards improving the way how efficacy in intervention is measured, in terms of change in the targeted mechanisms featured in the intervention. The absence of the first has been stressed as one of the major limitations in the evaluative framework of evidence-based interventions in psychology (David & Montgomery, 2011 ; Emmelkamp et al., 2014 ).

However, an initial challenge to the design of an RDoC informed intervention emerges from the assumption that RDoC does not recognize diagnosis a priori, for example, DSM categories (Cuthbert & Kozak, 2013 ). RDoC informed researchers of psychopathology are currently working with the RDoC matrix to push the state of art towards the stage 0 (basic science research prior to intervention) (Shoham et al., 2014 ), by including large datasets and by collecting multi-level data across multi-domains (e.g., Clementz et al., 2016 ; Drysdale et al., 2016 ; Van Dam et al., 2017 ). Ultimately, the accumulated scientific evidence will allow to isolate empirically derived clusters (biotypes) that may inform researchers how to design interventions that go beyond the current (heterogeneous) categorical diagnosis and that are focused on more specific-unique profiles of psychopathology (Clark et al., 2017 ; Clementz et al., 2016 ). As such, it will be possible in the long term to refine interventions for groups that are heterogeneous in nature and show divergent responses to the available treatments.

Nonetheless, the process of redefining psychopathology from RDoC-inspired research is still at the beginning and is highly demanding. For now, one possible venue is to test whether the RDoC constructs described in the matrix are mechanisms of change and promote efficacy in psychological interventions. TARA and Engage—the two interventions that will be analyzed in the current article—had started to use the empirical RDoC constructs to conceive new dimensional interventions in psychology.

At this point, it is important to systematize some practical aspects that must be taken into account when operationalizing the RDoC framework in the psychological clinical setting.

RDoC-based practical applications to interventions

Considering the RDoC postulates, there are some aspects that may be expected when designing and testing RDoC evidence-based interventions. As described below, these aspects may provide some guidance to clinicians aiming to design RDoC-inspired interventions.

First, it is important to isolate the RDoC constructs of interest with functional relevance to the psychopathological phenomena. The RDoC matrix was built on the assumption that neurobiological knowledge has reached the point of providing biologically meaningful targets, thus guiding the development of empirically informed interventions (Alexopoulos & Arean, 2014 ). Therefore, this framework informs clinicians about valid theories of psychopathology, allowing to better decide on which RDoC constructs should be used as targets in therapies. From the current standpoint, it is possible to roughly arrange the existing diagnoses into components of the RDoC matrix to select the constructs for intervention with high functional relevance to a disorder or a cluster of interrelated disorders (McKay & Tolin, 2017 ). For example, anxiety disorders are closely associated with the RDoC negative valence systems domain along the constructs of acute, sustained, and potential threat constructs (McKay & Tolin, 2017 ), and also with some constructs of the cognitive control systems domain (e.g., performance-monitoring subconstruct) (Pasion, Paiva, Fernandes, Almeida, & Barbosa, 2018 ).

Second, and from the set of selected constructs, it is important to develop a hierarchy of priority to intervention (Alexopoulos & Arean, 2014 ). This is one of the ways how RDoC promotes tailored interventions at the individual level (Clark et al., 2017 ). Alexopoulos and Arean ( 2014 ) recommended personalizing the intervention by selecting the appropriate psychological, behavioral, and ecosystem modification strategies to target abnormal domains of functioning and the respective constructs while considering additional domains and constructs in patients unresponsive to treatment. As such, the individual variability is taken into account independently of the previous diagnosis. Two individuals with anxiety may show distinct RDoC-related dysfunctions, and subsequently, the intervention must be necessarily distinct (Hershenberg & Goldfried, 2015 ).

Third, and to apply the hierarchical analysis of constructs to intervention, it is recommended to disentangle risk factors from mediation and moderation effects (Hershenberg & Goldfried, 2015 ). Mediator (how a treatment works) and moderator mechanisms (for whom treatment works) of effective treatment need to be accommodated in the intervention design (Hershenberg & Goldfried, 2015 ; McKay & Tolin, 2017 ). To the extent that various treatments operate via different mechanisms, this may facilitate the development of innovative and synergistic treatments, maximizing active therapeutic components and minimizing inactive or antagonistic ingredients, while anticipating which subpopulations will respond to a given intervention.

Forth, some methodological issues should be considered. The conceptual distance from categorical clinical diagnosis argues for the usage of large sample sizes in intervention research and for the inclusion of comorbid conditions that share causal (basic) mechanisms (Clementz et al., 2016 ; Drysdale et al., 2016 ; Van Dam et al., 2017 ). For example, threat-related and cognitive control constructs are not only relevant for anxiety disorders, but also for obsessive-compulsive and post-stress traumatic disorders (McKay & Tolin, 2017 ). In this sense, another common RDoC recommendation is to avoid extensive exclusion criteria that may limit the intended cohort approximation of a representative sample and a misrepresentation of the psychopathological spectrum (Van Dam et al., 2017 ). The inclusion of less severe expressions of psychopathology or even healthy relatives may also be motivated by the continuum analysis of the normal-abnormal spectrum.

Finally, and consistently with empirically based ambitions, it is mandatory to examine whether the efficacy of the new intervention is explained by alterations in the purported mechanisms of change. Since the units of analysis enable to better isolate the indicators of change, this may be demonstrated by collecting multi-level data (i.e., self-report, molecular, cellular, circuit, physiological, and behavioral data) across the selected multi-domains and constructs for intervention. As such, it is important to include the proposed RDoC units of analysis in the study design—along with other measures of interest to researchers. The RDoC matrix intends to be universal and, thus, a consistent application of the same constructs and units of analysis may foster the collection of uniformized data across distinct research groups. RDoC further calls for replication longitudinal analysis to prove the generalization and stability of the findings.

From this demonstration of how RDoC may work in clinical settings and intervention research, the next sections will illustrate how TARA and Engage were conducted to put into practice some of the RDoC postulates.

TARA is a group intervention program for adolescents with depression (14–18-year-olds), but also includes participants with anxiety symptoms. It comprises 12 weekly sessions of 90 min, in groups up to 12 participants. The program is composed of four modules (apiece lasting 3 sessions): Module 1, calming down and creating a sense of safety; Module 2, attending and caring about our inner experience; Module 3, recognizing, regulating and communicating emotions; and finally, Module 4, core values, goal setting, and committed action (Blom et al., 2014 , 2017 ).

The TARA program is one of the first published attempts to develop a psychological intervention that is translational in nature. Put differently, TARA authors were inspired by neuroscience-based research findings on RDoC mechanisms in depression and targeted them to intervention (Blom et al., 2017 ). On first glance, TARA seems to be just another mindfulness-based intervention (Blom et al., 2017 ). Undeniably, the intervention technics used in TARA have long been part of the repertoire of mindfulness, action and commitment therapy, and cognitive-based interventions. However, if we trace back to how these technics were developed and integrated into the aforementioned therapies, we realize that they lacked, in their inception, neuroscience evidence supporting them. On the contrary, empirical-based therapies in RDoC aim to integrate neuroscientific findings to support the goals of the intervention (psychological mechanisms) and the intervention technics tailored to achieve these goals. This will be explored in the next section, when we map the empirical-based RDoC constructs and neurodevelopmental research in depression, which were the basis to provide some guidelines to TARA’s intervention foci. TARA development was influenced by RDoC dimensional/transdiagnostic mindset, by including participants with depressive and/or anxiety symptoms due to the high comorbidity between these disorders and the common underlying mechanisms (e.g., genetic risk) evidenced by neuroscientific findings (Blom et al., 2017 ).

The TARA program targets many RDoC constructs and domains. However, we will focus on the ones reviewed in the last publication (Blom et al., 2017 ): sustained threat , loss , social processes , and reward learning .

Negative valence systems: sustained threat construct

Sustained threat is one of the constructs included in the domain of negative valence systems, associated with the dysregulation of amygdala reactivity and its brain circuits ( units of analysis ). Sustained threat is defined as “An aversive emotional state caused by prolonged (i.e., weeks to months) exposure to internal and/or external condition(s), state(s), or stimuli that are adaptive to escape or avoid. The exposure may be actual or anticipated; the changes in affect, cognition, physiology, and behavior caused by sustained threat persist in the absence of the threat and can be differentiated from those changes evoked by acute threat” (RDoC, 2019 ).

The TARA program focused on the construct of sustained threat as one of the processes involved in adolescent depression because several neuroimaging studies have demonstrated that depressed adolescents show increased hyperactivation of the amygdala and alterations in its connectivity (Blom et al., 2014 , 2017 ). This has been associated with persistent mood and vegetative symptoms experienced by adolescents with depression (e.g., dysphoria, lassitude) potentially via impaired processing of memories and visceral signals (Cullen et al., 2014 ). Hence, the first intervention target of TARA was to increase vagal and sensory afference through breathing practices and synchronized movement (Module 1), aiming at diminishing amygdala hyperactivation through a bottom-up approach. Supporting this option, recent research using neuronal recordings from rats revealed that the activity in the amygdala was altered following vagal nerve stimulation (Alexander et al., 2017 ). The authors of the program expected that these intervention strategies would result in decreased feelings of stress, as well as physical symptoms of depression and anxiety, allowing for improved emotional self-regulatory abilities and sleep (Blom et al., 2017 ).

Negative valence systems: loss construct

Loss is also included in the domain of negative valence systems and is associated with brain circuits (units of analysis) like the default mode network and the amygdala. Loss is defined as “A state of deprivation of a motivationally significant con-specific, object, or situation. Loss may be social or non-social and may include permanent or sustained loss of shelter, behavioral control, status, loved ones, or relationships. The response to loss may be episodic (e.g., grief) or sustained” (RDoC, 2019 ).

The construct of loss was featured in the TARA program as several neuroimaging studies showed alterations in the DMN (default mode network) in depressed adolescents (Blom et al., 2014 , 2017 ), as well as in their amygdala restating state functional connectivity (Clark et al., 2018 ). At the same time, the heightened dominance of the DMN has been associated with higher levels of maladaptive, depressive rumination, and lower levels of adaptive, reflective rumination (Hamilton et al., 2011 ). Also, there is enough evidence for a relation between disrupted amygdala-prefrontal connectivity and stress-related rumination in depressed adolescents (Fowler, Miernicki, Rudolph, & Telzer, 2017 ). This self-criticism and broader negative expectations about the self, future, and others are corner symptoms of depression (Beck & Bredemeier, 2016 ) supporting the role of the construct of loss as one of its basic mechanisms. Therefore, the second intervention target was to help participants shift neuronal activation away from the DMN by firstly noticing negative self-referential thoughts and then focusing on the present sensory and interoceptive input, including practices of identifying, labeling, and expressing emotional processes without judgment (Module 2). Indeed, mindfulness training in adults has been associated with deactivation of the DMN (Berkovich-Ohana, Glicksohn, & Goldstein, 2012 ) and altering of resting-state functional connectivity of the amygdala (Taren et al., 2015 ). Based on these results, the mindful-based practices seem good intervention strategies for decreasing rumination and generalized anxiety, which were the expected outcomes of Module 2 (Blom et al., 2017 ).

Social processes: social communication, perception, and understanding of self and others constructs

The systems for social processes is the functional domain that “mediate[s] responses to interpersonal settings of various types, including perception and interpretation of others’ actions” (RDoC, 2019 ). Many of the constructs of this domain were featured in the TARA intervention, for instance, social communication, perception, and understanding of self and others (Blom et al., 2014 ). They are the focus of Module 3. Authors state that because there are many brain circuits involved in these constructs, they did not emphasize one in particular (Blom et al., 2014 ).

The social processes systems is a domain of interest, as it has been shown that interpersonal problems (e.g., social conflict and social rejection), whether in the family realm or not, are relevant distal factors or proximal precipitants of depression (Beck & Bredemeier, 2016 ; Blom et al., 2017 ). Therefore, the third target of TARA program was to develop in the participants a set of emotion regulation skills for dealing with social interactions. These skills involved recognizing emotional triggers in oneself and others, empathetic listening, effective communication, and compassionate responses to oneself and others (Blom et al., 2017 ). Prior research has demonstrated that mindfulness intervention, which explicitly coaches self-compassion and mindful exposure to social situations, is associated with better social adjustment and lower social anxiety symptoms (Koszycki et al., 2016 ). The authors of the program used diminished levels of social anxiety as evidence that the intervention would have resulted in an improvement in the abovementioned social processes (Blom et al., 2017 ).

Positive valence systems: reward learning construct

Is one of the constructs included in the domain of positive valence systems, associated with brain circuits (units of analysis) that include the ventral and dorsal striatum. Reward learning is defined as “A process by which organisms acquire information about stimuli, actions, and contexts that predict positive outcomes, and by which behavior is modified when a novel reward occurs or outcomes are better than expected. Reward learning is a type of reinforcement learning, and similar processes may be involved in learning related to negative reinforcement” (RDoC, 2019 ).

Reward learning was selected as one of the constructs to be included in TARA because depression has been associated with the disrupted balance of cortico-striatal circuit function in adolescents (Blom et al., 2014 , 2017 ; Kerestes et al., 2015 ). These circuitries have been implicated in emotional processing, reward learning, and cognitive control, which in turn have been highlighted as some of the mechanisms in depression. Indeed, impaired reward learning may hinder changing negative self-referencing thinking based on new positive experiences (Marchand, 2012 ) and contribute to low mood, anhedonia, and psychomotor retardation (Price & Drevets, 2012 ). Hence, the fourth intervention target of TARA was to increase behavioral activation guided by intrinsic reward. These strategies were inspired by acceptance and commitment therapy, which share many of the mindfulness and cognitive restructuration strategies (Module 4). Here, participants had to develop a personal strategy for living their life in coherence with their own values, had to challenge patterns of experiential avoidance, and practiced top-down cognitive control of affective responses (Blom et al., 2014 ). The expected behavioral outcomes of this module were decreased experiential avoidance, increased committed action, and improved behavior during emotional arousing experiences (Blom et al., 2017 ).

Research evidence supporting TARA

There is one pilot study published with TARA. A single-arm trial was conducted with 26 adolescents (14–18-year-olds; 19 females). Participants met inclusion criteria when they had clinically significant depressive and/or anxious symptomatology. There were three assessment time points: before the groups started; after the groups had finished; and 3 months after TARA ended. Outcomes were reduction of adolescent depression (anhedonia/negative affect, dysphoric mood, negative self-evaluation, somatic complaints); anxiety (physical symptoms, harm/avoidance, social anxiety, separation/panic), insomnia severity, and experiential avoidance; and increase in mindfulness.

There is preliminary evidence that TARA may be a feasible and efficacious intervention program. Overall, TARA participation was associated with less symptoms of depression and generalized and social anxiety, even 3 months later (Blom et al., 2017 ).

By analyzing changes in the RDoC domains/constructs, using the self-reports as units of analysis, some validity evidence is obtained not only for the hypothesized mechanisms in depression but also for the efficacy of TARA modules/techniques in changing such processes. Indeed, the pilot study’s results point to some improvements in the RDoC domains/constructs. From baseline to 3 months after the end of the intervention, participants reported (Blom et al., 2017 ) fewer physical symptoms of anxiety/depression, insomnia, and improved self-regulatory skills (Module 1—construct of sustained threat), decrease in rumination (Module 2—construct of loss; decrease in social anxiety symptoms (Module 3—domain of social processes, and reduction in anhedonia and experiential avoidance (Module 4—construct of reward learning).

Engage is a structured, stepped approach therapy for late-life depression with nine sessions of 40–45 min each (Alexopoulos et al., 2015 , 2017 ; Alexopoulos & Arean, 2014 ). The Engage key RDoC constructs for intervention were defined after consider the input provided by professionals in the field. The authors were guided by the principle of simplification when defining the main strategies for intervention, based on the assumption that simplifying the language and strategies when disseminating RDoC-inspired interventions may turn them more accessible to professionals. Moreover, Engage is based on the behavioral activation therapy (Alexopoulos & Arean, 2014 ), as it is aligned with the broad perspective that depression is associated with a loss of reinforcement from the environment leading to inadequate experience of pleasure or meaning from rewards (Ferster, 1973 ). Since individuals lose interest in activities, leading to melancholic mood, anhedonia, and psychomotor retardation, the intervention entails the scheduling and facilitation of meaningful, rewarding activities. “Reward exposure” is, therefore, the main focus of Engage and the therapeutic vehicle is the RDoC positive valence system (Alexopoulos et al., 2015 ; Alexopoulos & Arean, 2014 ).

In line with empirically based ambitions, the authors of Engage relied on the RDoC workshop proceedings and related findings to select intervention targets, assuming that the RDoC framework reflects an empirical organization of the accumulated knowledge. The end result was a simplified intervention targeting domains of function and constructs with high relevance in late-life depression. It is worth noticing that reward learning, a construct of this system, was also a target for intervention in Blom et al. ( 2017 ), but the reviewed evidence and the intervention design account for the specificities of the targeted group (adolescent vs. elderly). Personalized therapies are, in fact, an RDoC assumption. The domains and constructs are not designed to be independent, but rather promote comprehensive and tailored interventions (Clark et al., 2017 ). In other words, it is not intended to isolate specific constructs to intervention, but rather consider multiple RDoC constructs of interest while defining a unique profile of intervention (Alexopoulos & Arean, 2014 ; Geraldo, Azeredo, Pasion, Dores, & Barbosa, 2018 ). Despite the focus on positive valence system—the core system to explain depression in later-life—the Engage program also includes additional constructs that may moderate the efficacy in intervention, namely loss, arousal, and cognitive control (Alexopoulos et al., 2015 , 2016 , 2017 ; Alexopoulos & Arean, 2014 ).

Positive valence systems: approach motivation and reward learning constructs

The RDoC positive valence system is proposed by the authors (Alexopoulos et al., 2015 , 2016 , 2017 ; Alexopoulos & Arean, 2014 ) as a central mechanism to late-life depression, since the alterations on approach motivation and reward learning constructs seem to hinder rewarding activities, strengthening depressive symptoms and beliefs of lack of meaning of life.

Approach motivation is a “multi-faceted construct involving mechanisms/processes that regulate the direction and maintenance of approach behavior influenced by pre-existing tendencies, learning, memory, stimulus characteristics, and deprivation states” (RDoC, 2018 ). Reward valuation and action selection/preference-based decision-making are subconstructs of approach motivation.

It is important to note, however, that in the last version of the RDoC matrix launched at the beginning of 2019, the positive valence systems had been reorganized. Approach motivation has been removed from the matrix and Reward Valuation is now a construct itself (rather than a subconstruct of approach motivation). Reward Valuation is defined as the set of “Processes by which the probability and benefits of a prospective outcome are computed by reference to external information, social context (e.g., group input), and/or prior experience. This computation [that may involve the assignment of incentive salience to stimuli] is influenced by preexisting biases, learning, memory, stimulus characteristics, and deprivation states” (RDoC, 2019 ). Importantly, Reward Valuation includes Delay as a subconstruct that is enrolled in action selection and decision-making as it represents “processes by which the value of a reinforcer is computed as a function of its magnitude and the time interval prior to its expected delivery” (RDoC, 2019 ).

Despite the changes to the matrix in which Engage was based on, the background remains consistent and valuable. As the authors had stated, abnormalities in Reward Valuation are detected in time inconsistency and intertemporal action for gains and losses in delay discounting tasks (Alexopoulos et al., 2015 , 2016 , 2017 ; Alexopoulos & Arean, 2014 ).

In what regards reward learning, a construct previously explored in TARA, low functional activation of the nucleus accumbens, striatum, and the caudate during the processing of positive-rewarding stimuli in monetary paradigms supports deficits in the acquisition of information that predict positive outcomes (Alexopoulos et al., 2015 , 2016 , 2017 ; Alexopoulos & Arean, 2014 ),.

Thus, considering the dysfunction in constructs of the positive valence system, the Engage program uses reward exposure to reignite this system in terms of reinforcing the engagement in social and physical rewarding activities that have been abandoned by depressed patients. The authors acknowledge that rewarding activities have been a central component of most psychotherapies, where therapists guide patients to develop a list of meaningful activities, such as social engagement, physical exercise, and intellectual exchange, and noticed that depressed elderlies are responsive to these activities.

During the first three sessions, and in a context of a supportive relationship, the focus is on a direct facilitation of rewarding activities using a simplified problem-solving approach: (a) select the most feasible and rewarding goal of the activities included in the participants’ list, (b) develop a list of ideas on what to do in order to meet this goal, (c) choose one of these ideas, and (d) make an “action plan” with specific steps to address obstacles. These strategies allow to operationalize approach motivation.

As a therapeutic tool, these strategies also constitute a behavioral probe that allows to identify patients who fail to engage in rewarding activities. At the end of the third session, therapists conduct a structured assessment to determine whether patients (a) learned how to form action plans, (b) engaged in the planned activities, and (c) showed improvement in depression scores. If all conditions are met, patients continue with reward exposure to promote reward learning, moving from easy to more complex activities. If not, therapists identify barriers and strategies to mitigate them, so that “reward exposure” can progress unhindered. Barriers such as negativity bias, apathy, and emotional dysregulation may be behavioral expressions of neurobiological system dysfunctions frequently occurring in late-life depression (Alexopoulos & Arean, 2014 ). Each of these barriers are related to impairment in RDoC domains of functioning—respectively, loss, arousal, and cognitive control—and are recurrent targets for learning-based therapies. Cognitive-behavioral and ecosystem modification strategies that literature has found efficacious are then used, as detailed below. A similar assessment is made at the end of session 6, and for those still experiencing difficulties, therapists assess whether other barriers exist and add strategies to counteract them.

As a general outcome, it is expected an increase in behavioral activation and engagement in rewarding activities associated with a reduction in depressive symptoms.

The negativity bias, that is the tendency to redirect the attention towards negative rather than positive information, is a vulnerability factor for depression thought to reflect the behavioral expression of the RDoC Loss construct described in the negative valence systems (Alexopoulos & Arean, 2014 ) and previously described TARA.

Correlates of negativity bias include neural excitation to signals of potential danger, heightened startle amplitude, higher heart rate, higher functional activation of the amygdala, anterior cingulate cortex, and ventral and dorsomedial prefrontal cortex to conscious and nonconscious fear probes; moreover, the 5-HTTLPR serotonin transporter seems to explain the heightened sensitivity to negative stimuli (Alexopoulos et al., 2015 ; Alexopoulos & Arean, 2014 ).

Several simplified cognitive-behavioral strategies are used to mitigate the negativity bias, by redirecting the attention to neutral or positive aspects of the reward exposure, namely play devil’s advocate for thoughts interfering with engagement, weigh the evidence to motivate to pursue activities, practice the positivity bias, write alternative positive explanations to negative thoughts, and learn how positive people respond to the same negative situations (Alexopoulos et al., 2016 , 2017 ; Alexopoulos & Arean, 2014 ).

Arousal and regulatory system: arousal construct

Apathy, the lack of motivation not attributable to a diminished level of consciousness, cognitive impairment, or emotional distress seems to constitute the behavioral expression of a dysfunction in the arousal and regulatory system, particularly in Arousal (Alexopoulos & Arean, 2014 ).

Arousal is a continuum of sensitivity of the organism to stimuli, both external and internal; facilitates interaction with the environment in a context-specific manner (e.g., under conditions of threat, some stimuli must be ignored while sensitivity to and responses to others is enhanced, as exemplified in the startle reflex), can be evoked by either external/environmental stimuli or internal stimuli (e.g., emotions and cognition); can be modulated by the physical characteristics and motivational significance of stimuli, varies along a continuum that can be quantified in any behavioral state, including wakefulness and low-arousal states including sleep, anesthesia, and coma; can be regulated by homeostatic drives (e.g., hunger, sleep, thirst, sex) (RDoC, 2019 ).

In late-life depression, apathy is accompanied by functional alterations characterized by low resting-state connectivity of the nucleus accumbens with the amygdala, globus pallidus, thalamus, caudate, putamen, and increased connectivity with the dorsomedial prefrontal cortex and insula (Alexopoulos et al., 2015 ; Alexopoulos & Arean, 2014 )

Both cognitive-behavioral and ecosystem modification strategies are used to cope with apathy, since in severe expressions of apathy, the surrounding context of the individual plays a critical role to mitigate it. Prompts to trigger the action plans for reward exposure are required, for example, checklists, reminders, tape recorders, electronic instructions to start tasks, family and friends acting as prompts (Alexopoulos et al., 2016 , 2017 ; Alexopoulos & Arean, 2014 ).

Cognitive Systems: cognitive control

Finally, emotional dysregulation is considered the behavioral expression of Cognitive Systems dysfunction, namely the Cognitive Control construct. Cognitive control “modulates the operation of other cognitive and emotional systems, in the service of goal-directed behavior, when prepotent modes of responding are not adequate to meet the demands of the current context; control processes are engaged in the case of novel contexts, where appropriate responses need to be selected from among competing alternatives” (RDoC, 2019 ).

Emotional dysregulation may result from abnormal patterns of functioning of the ventral-rostral anterior cingulate cortex and medial prefrontal cortical areas responsible for appraisal and regulation of emotional functions, as well as limbic subcortical circuits of emotion processing (Alexopoulos et al., 2015 ; Alexopoulos & Arean, 2014 ).

The coping strategies are selected from previous experiences and may include modulation (e.g., overwhelming disappointment and anxiety) and management of emotions (e.g., distraction, imagery, relaxation exercises, deep breathing, meditation). The selected strategy is practiced during the session and then autonomously, to experience the difficulties when pursuing plans for reward exposure.

Research evidence supporting Engage

The strategy to evaluate Engage targeted the assessment of training time, fidelity to practice sessions, reduction in depression severity, longitudinal maintenance of the gains and remission.

Considering the focus on simplification and applicability, Alexopoulos et al. ( 2015 ) examined the acquisition of therapeutic skills to administer Engage and the post-training fidelity to treatment in relation to problem-solving therapy. Both social workers and therapists required a shorter training time to be certified in Engage (6.92 h) than in problem-solving therapy. Post-training fidelity was examined by reviewing audiotapes of 63 Engage and 93 problem-solving therapy sessions. Only 12.7% of the Engage sessions required corrective feedback (vs. 36.6% in problem-solving therapy). From the results, the simplification of the language made the interventions more accessible to clinicians. The authors pointed out that Engage is the first psychotherapy using RDoC concepts as a guide to simplification and such an approach may promote fidelity (Alexopoulos et al., 2015 ).

The efficacy of Engage was further assessed in 39 depressed older adults over 9 weeks, in comparison with 97 patients treated with the problem-solving therapy (Alexopoulos et al., 2015 ). This therapy was used as a term of comparison given that Engage uses a simplified problem-solving approach (Alexopoulos & Arean, 2014 ). There were no demographic or clinically significant differences at baseline between the two groups, namely in depression severity, medical burden, disability, and cognitive impairment. Engage did not differ from problem-solving therapy in reducing depression severity. Remission rates for Engage at 6 and 9 weeks were 18.2% and 41.1%, respectively, compared to 13.7% and 35.0% for problem-solving therapy. In a subsequent study (Alexopoulos et al., 2016 ), and including a sample size of 48 depressed older adults, depression scores sharply declined by 10 points during the 9 weeks of treatment and increased mildly by 4.8 points during the follow-up phase (36 weeks). Additional interventions for negativity bias, apathy, and emotional dysregulation did not impact the results in both studies (Alexopoulos et al., 2015 , 2016 ). Using self-report as a unit of analysis to measure activation and to capture approach motivation and reward learning, it was found that the number and variety of engaged activities, the ability to remain active in the face of difficulties, the sensitivity to reward, and feelings of accomplishment had a sharp increase during the 9 weeks of Engage treatment and a mild decline during follow-up. Only activation predicted depression severity at the end of each period of assessment. An increase of one standard deviation in activation resulted in a decrease in depression scores.

The main results were considered by the authors as a proof of concept for Engage efficacy in reducing depressive symptoms, promoting remission and the maintenance of gains by using behavioral activation techniques of reward exposure and a simplified, personalized intervention in RDoC positive valence systems that also considers specific differences in response to intervention (Alexopoulos et al., 2015 , 2016 , 2017 ; Alexopoulos & Arean, 2014 ).

Critical analysis of RDoC applicability to empirically supported psychological interventions

Almost a decade has passed since the NIMH launched the RDoC initiative, and there are still few intervention studies published, with some arguing about the difficulties in integrating the matrix into the current modi operandi of psychological intervention (Appelbaum, 2017 ).

Even though there are scarce empirical studies, which limits the evaluation of RDoC applicability, in the next sections we attempted to critically discuss the merits and drawbacks of RDoC at the conceptual, methodological, and efficacy measurement levels, based on the reviewed empirically based interventions. TARA and Engage are illustrative and represent an effort to conduct RDoC-inspired interventions in naturalistic settings and may expose some advances and difficulties in implementing RDoC in clinical practice. These first efforts in applying RDoC to psychological intervention may be a helpful source of recommendations for future and improved RDoC-tailored psychological interventions.

Conceptual level

How to reframe psychological interventions based on the constructs of the RDoC matrix and on its transdiagnostic basic mechanisms, rather than on previous clinical diagnosis, represents an innovative postulate that may foster optimally matched treatments for mental disorders by settling specific profiles for intervention from the differential combination of RDoC constructs as targets (Geraldo et al., 2018 ).

The selection of RDoC constructs as targets in TARA and Engage were guided by the dimensional character of the RDoC matrix, and in the case of TARA, by a transdiagnostic stance. The attribution of a hierarchical value among the selected constructs to define the priorities for intervention illustrated how tailored and less standardized interventions may be conducted (Alexopoulos & Arean, 2014 ). The individual variability was particularly taken into account in Engage, since the strategies for intervention were dependent on the distinct profiles of depression representing different RDoC-related dysfunctions. TARA also introduced therapeutic objectives along the therapy sessions based on the cascade of purported causality between associated mechanisms in adolescent depression.

It is important to note that Engage and TARA selected distinct constructs to intervention to the same psychopathological phenomenon (depression), following a developmental sensitive framework: older adults vs. adolescents. Reward learning was the focus of interest of Engage and supported the designed strategies for social engagement in older adults, while TARA extended to other interrelated constructs. In fact, TARA included more constructs than Engage, which forces us to think on the optimal set of constructs to be included in an intervention that intends to be both comprehensive and feasible.

At this point, it is critical to acknowledge that the dimensional constructs and the differential interaction among them do not necessarily increase the complexity of the intervention. The RDoC matrix offers a structured guidance that intends to help clinicians defining targets for intervention in a clearer and more precise fashion, working towards simpler though comprehensive therapies. Indeed, it is advocated that the simplified/structured RDoC approach to intervention is a necessary first step for large-scale use (Alexopoulos & Arean, 2014 ).

However, operationalizing RDoC principles in naturalistic settings is challenging (Sharp et al., 2016 ).

The RDoC matrix is lacking important psychological constructs (Mittal & Wakschlag, 2017 ), which limits its applicability to the study of psychopathology and to the design of psychological interventions. For example, the social processes domain used in TARA lacks the detail and precision of other constructs (e.g., negative valence systems). In fact, a recent systematic review on RDoC published research highlights that social processes is one of the least studied domains (alongside with arousal and regulatory systems) (Carcone & Ruocco, 2017 ). This is a crucial limitation for clinical research, as social processes are central to understanding how psychopathology develops (e.g., Autism Spectrum Disorder) and many intervention strategies capitalize on these processes, whether by the means of delivery (e.g., face-to-face therapy) or by the inclusion of targets for intervention (e.g., develop social skills, increase social support).

An embedded developmental approach is also missing in the definition of RDoC constructs (Mittal & Wakschlag, 2017 ). As they are currently defined, many are adult based (or at least the units of analysis focus on this age range) or are presented as varying smoothly throughout development (Mittal & Wakschlag, 2017 ;Patrick & Hajcak, 2016 ; Peterson, 2015 ). By disregarding that the constructs may be developmentally discontinuous, considerable challenges are posed for their inclusion in developmental and youth intervention studies (Patrick & Hajcak, 2016 ; Peterson, 2015 ).

Considering all the conceptual limitations of the matrix to clinical practice, it is important to bear in mind that the matrix is a work in progress (Clark et al., 2017 ). This means that constructs—or developmental variations of the constructs—found to be meaningful to understand psychopathology and to improve interventions may be integrated at any time into the matrix. Nevertheless, it may also mean that the data collected from the guidelines of the archived versions of the matrix may be out of date quickly and lose the expected impact for guiding future and improved studies. For instance, the positive valence systems domain used in Engage was reconceptualized, and although it was not the case, this change could have called into question the empirical validity in which the intervention was based on.

Another RDoC feature that may impel psychological researchers and clinicians away from using this framework is its focus on biology. As a first implication, the RDoC constructs become unfamiliar to clinicians and their clinical utility is not obvious. Despite the relevance of some constructs for several disorders, there is no clear connection between psychological symptoms, clinical problems, and the constructs, demanding a deeper specification of clinical problems as targets for RDoC intervention research (Patrick & Hajcak, 2016 ). Moreover, the RDoC matrix assumes that the different units of analyses are equivalent for each construct, which enables researchers to integrate results from different fields and clinicians to use the accumulated knowledge to devise therapeutic interventions (Marková, 2018 ). As such, translational research searches for the meaning of a certain process in one level of analysis (e.g., biochemical changes in the brain) and also for the result of such changes on other levels of analysis (e.g., behavior and symptoms). However, there seems to be no one-to-one correspondence between units of analysis within the same construct (Peterson, 2015 ). As we progress through the different levels of analysis, from more internal (e.g., molecules) to more external (e.g., self-reports), there is no isomorphic transfer of meaning; that is, information is lost and new is added (Marková, 2018 ).

Finally, when RDoC defines mental disorders from a biological framework focused on the neurodevelopmental aspects of psychopathology, there is a risk to claim that alterations in the brain and biology are causing the altered mental states and behavior in a reductionist and linear way (Marková, 2018 ). This could lead to an internalized (mis)conception of psychological disorders as well as to therapeutic interventions focused solely on biological and pharmacological targets. This would deny the importance of translating the research to psychological interventions and would leave behind the role of the environment for the development of psychopathology. Therefore, it is important to keep a continuous critical analysis on how multifactorial causality in psychopathology is being operationalized in RDoC research.

Biological research on basic mechanisms of psychopathology may be the anchor of RDoC, but a closer look to the translational endeavor shows that RDoC interventions inspired do not rely on reductionism. As stated in RDoC assumptions, an overemphasis on biology would neglect the evidence for a multicausal nature of mental illness. Thus, it is proposed that psychopathology is best conceptualized as a confluence of multiple factors—from biological, developmental, psychosocial, to environmental—that are indeed integrated in the RDoC matrix (Clark et al., 2017 ). For example, psychological factors were the focus of the reviewed interventions, but the contextual and social factors were also considered. Engage uses ecosystem modification strategies (e.g., the apathy was reduced by family and friends acting as prompts) and the emotion regulation strategies in TARA are practiced in relevant social interactions. Also, in RDoC agenda it is explained that the definition of the constructs around the identification of neural circuit does not mean a causal relation between them. It arose from the necessity to find a central unit of analysis that could easily aggregate research streaming from biology and neuroscience to psychology (Kozak & Cuthbert, 2016 ), contributing to the definition of testable and valid psychological constructs, which are relevant to human functioning (Clark et al., 2017 ).

In this line, RDoC may be grounded on the identification of basic mechanisms to intervention—as TARA and Engage did—but the transcultural and universal aspects of these basic mechanisms are of high importance as well. RDoC provides tools and calls for research in large datasets including distinct groups and minorities to test the transcultural validity of its constructs. The adaptation of interventions for distinct cultural and social contexts may contribute to improving the tailoring aspects of RDoC-inspired empirically supported interventions for psychological disorders.

Methodological level

An innovation introduced by RDoC at the methodological level concerns the non-conservative inclusion criteria when selecting participants for research and intervention (Van Dam et al., 2017 ). The reduction in inclusion criteria impels towards a real approximation of the psychopathological phenomenon that cuts across diagnostic boundaries and a real representation of the representative sample. For instance, TARA includes individuals with both symptoms of depression and anxiety; two highly comorbid conditions that are usually approached as distinct (and apparently independent) diagnostics. In turn, the intervention in Engage is disorder-specific and reduces the transdiagnostic applicability of the intervention. By not considering psychiatric comorbidity, no conclusions can be drawn about the suitability of Engage in depressed elders with additional psychiatric diagnoses (Alexopoulos et al., 2015 ). In a more general note, neither TARA nor Engage follow the full directives of RDoC concerning inclusion criteria to the extent they should have included other diagnoses, subclinical expressions and even healthy participants to address the normal-to-abnormal spectrum continuum.

In the case of Engage, these methodological limitations end in a group comparison analysis that loses its central importance in the RDoC preconized dimensional/continuous/transdiagnostic approach. Nevertheless, the between-group comparison between Engage and problem-solving therapy provides stronger evidence than TARA’s study, which did not have a comparison group. The conclusions on TARA are based on a pilot study including a small and age-restricted sample of 26 adolescents followed for 3 months. The comparison Engage vs. problem-solving therapy was, however, conducted without randomization and, again, in a short period of time (9 weeks). The follow-up for 36 weeks did not include a control group. Thus, the main results should be seen as exploratory, remaining unclear whether symptoms remission is specifically related to the intervention. More efforts should be allocated to retrieve valid conclusions on the longitudinal, dimensional, and continuous analysis preconized by RDoC.

Furthermore, the use of self/hetero-reports and behavioral assessments may be a trend in many psychological studies but offers limited evidence. The RDoC matrix brings up other units of analysis (i.e., molecules, physiology, circuits) that should be included to get more robust indicators of efficacy (Mckay & Tolin, 2017 ). TARA and Engage did not take advantage of this novel approach to measure efficacy and have not included biological or other units of analysis, although they used neuroscientific evidence to consubstantiate their interventions. Using the RDoC’ units of analysis to search for evidence of validity is a promising new venue, even if including assessment measures from several research fields requires the interplay between areas of knowledge, multidisciplinary teams, and longer assessment procedures, making this a costly and demanding endeavor (Geraldo et al., 2018 ). As such, it is not clear how it will be achieved in mainstream intervention studies.

Efficacy measurement level

Until now, most evidence-based interventions in psychology have been based on repetitive demonstrations of efficacy (Clark et al., 2017 ; David & Montgomery, 2011 ). This means that the efficacy of the psychological interventions has been tested in a vacuum regarding the validity of the psychological theories that gave rise to these same interventions.

Since the RDoC matrix is empirically driven, a major advantage is that it serves as a tool to inform about psychological theories. Moreover, researchers are now able to collect evidence for the efficacy of the intervention by using measures from the different levels of analysis described in the selected constructs for intervention. That is, RDoC not only searches for empirically based theories about psychological mechanisms of psychopathology but also provides units of analysis that allows to measure the efficacy of the intervention. This fosters empirically supported interventions in a double sense. From the RDoC matrix, TARA, and Engage searched for validity evidence on the psychopathological theories to define the constructs of interest, change mechanisms, and consequent intervention strategies, as well as to measure the impact of the intervention by using some of the proposed units of analysis.

Nevertheless, some shortcomings may be exposed when considering the RDoC measures included in each unit of analysis and how they relate to each other.

Efforts undertaken to establish psychometric properties of the psychophysiological measures are scarce in contrast with self-reports. The convergent and divergent validity of the psychophysiological measures, as well as their reliability, need to be empirically established (Patrick & Hajcak, 2016 ; Peterson, 2015 ). This is a clear obstacle to asserting that these measures are valid indices of the psychological constructs.

Patrick and Hajcak ( 2016 ) further call our attention to the difficulties in achieving high or even moderate correlations between different units of analysis included in the same construct, due to the domain-specific/method variance. This taps on our previous discussion regarding the lack of a precise, one-to-one, correspondence between units of analysis within the same construct (Marková, 2018 ); that is, information about the construct is not exactly the same when moving along the units of analysis. Even so, the domain-specific/method variance problem is not specific of the RDoC and is expected to be found in multi-level assessments. Investigators (e.g., Yancey, Venables, & Patrick, 2016 ) have pointed methodological approaches to address this issue. The computation of cross-domain composites using structural equation modeling and multidimensional item-response modeling may be sound alternatives, but demand for large datasets that are required to ensure high statistical power when testing all the hypotheses paths. For instance, the analysis of the interaction between risk factors, moderation, and mediation effects is complex and demanding but is necessary to provide a comprehensive picture of RDoC postulates. From the provided (complex) picture, it remains difficult to accommodate mediation and moderation effects in the analysis of efficacy as measured from different units of analysis, while dissociating these effects from risk factors.

There were some efforts of TARA and Engage in this direction when considering the hierarchical organization of the constructs for intervention and the sample characteristics. Engage clearly dissociates the main construct of interest as the risk factor (i.e., reward learning) from other constructs that may mediate and potentiate the results (i.e., loss, arousal and cognitive control constructs as reducing, respectively, the negativity bias, apathy, and emotional dysregulation). TARA further tested a larger set of symptoms to prove the transdiagnostic applicability of the intervention while prioritizing constructs that purportedly have a greater impact on the development of depression and anxiety in adolescents (e.g., Sustained Threat) vs. other age groups. Considering that different mechanisms are implicated in different interventions, the inclusion of mediation and moderation effects may increase our understanding for whom the intervention works and improve the development of synergistic treatments, contemplating the maximization of active therapeutic components and the minimization of antagonistic ingredients (Hershenberg & Goldfried, 2015 )

Thus, and despite the methodological limitations previously presented, there are some evidence for the efficacy of TARA in what regards the mechanisms of change. The inclusion of the purported psychopathological mechanism—RDoC construct (e.g., sustained threat—amygdala hyperreactivity), and intervention strategies targeting this construct (e.g., increase vagal efference to lower amygdala hyperreactivity through deep breathing) was related with the expected outcomes, both at process (e.g., better regulatory skills and sleep) and symptom levels (e.g., depression). These findings may indicate that the targeted constructs are relevant for empirically supported interventions as they were associated with critical mechanisms of change in the psychopathological phenomenon (McKay & Tolin, 2017 ). Contrarily, Engage was focused uniquely on symptom remission as measured by the included self-reports. Self-report measures focused on symptomology are not foreseen in the RDoC matrix to measure the change on criteria for clinical diagnosis, but rather to assess the change on mechanisms that have a close link with the maladaptive expression of the construct. This limits the analysis of efficacy under a RDoC approach and is associated with the traditional way of conceiving psychopathology. The use of self-reported measures described in the RDoC matrix should be included to test the convergent validity of the constructs and to measure the efficacy in a consistent way with the RDoC proposals of accumulating knowledge in a large (international) dataset. Nonetheless, the RDoC matrix is not intended to be prescriptive, and it may be important to maintain the link between RDoC and symptomatology to keep the results comparable and interpretable, at least at an early stage of research.

In fact, the few and limited evidence to date, alongside with the differences in the methodological and efficacy measurement levels between RDoC and more traditional approaches, may make it difficult to assess if RDoC interventions are better than the gold standard ones (e.g., cognitive-behavioral therapy). Despite the focus of RDoC on continuous and longitudinal data instead of taxonomic analyses, the gap between the two approaches may be bridged, in a first moment, by conducting studies with similar designs (e.g., randomized control trials) (David & Montgomery, 2011 ) or by fostering comprehensive analytical procedures (e.g., regression-based models).

Presently and taking into consideration that there is limited research on RDoC-inspired psychological interventions, the depth and practicality of these applications remains largely hypothetical. The paucity of published RDoC-inspired psychological interventions may relate to the limitations and misconceptions described above at a conceptual, methodological and efficacy measurement levels. The small number of published interventions precludes the ambition of collecting large datasets in real-life intervention contexts and of including representative psychiatric populations with complex and multifaceted symptom profiles (Alexopoulos et al., 2015 ; Hershenberg & Goldfried, 2015 ), that would lead to a real assessment on RDoC applicability.

Nevertheless, RDoC has several contributes to advance our knowledge on psychopathology and holds promise to be a profitable framework for psychological researchers and clinicians. It may inspire more valid understanding of psychological functioning and psychopathology and in turn influence the development of better evidence-based psychological interventions. In this review, we tried to highlight the potentials of the RDoC initiative and explore the main criticisms and expected limitations of its directives in real-life intervention studies, hoping that it will sparkle new attempts in applying RDoC to psychological intervention.

Acknowledgements

The authors would like to acknowledge Dr. Bruce Cuthbert for all the support and availability to discuss the RDoC initiative.

Abbreviations

Authors’ contributions

RP and ECM are first shared co-authors. RP and ECM contributed equally to the work and were responsible for the literature search, literature analysis, critical reflections, and writing the manuscript. FB participated in critical reflections of the literature and reviewed the manuscript while providing meaningful inputs. All authors read and approved the final manuscript.

RP is supported by a doctoral fellowship from the Fundação para a Ciência e a Tecnologia under grant SFRH/BD/117130/2016. The funding had no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

Availability of data and materials

Competing interests.

The authors declare that they have no competing interests.

Rita Pasion and Eva C. Martins are co-first authors.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rita Pasion and Eva C. Martins contributed equally to this work.

• Alexander, G. M., Huang, Y. Z., Soderblom, E. J., He, X. P., Moseley, M. A., & McNamara, J. O. (2017). Vagal nerve stimulation modifies neuronal activity and the proteome of excitatory synapses of amygdala/piriform cortex. Journal of neurochemistry, 140(4), 629-644. [ PMC free article ] [ PubMed ]
• Alexopoulos GS, Arean P. A model for streamlining psychotherapy in the RDoC era: The example of “Engage” Molecular Psychiatry. 2014; 19 (1):14–19. doi: 10.1038/mp.2013.150. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Alexopoulos GS, O’Neil R, Banerjee S, Raue PJ, Victoria LW, Bress JN, et al. “Engage” therapy: Prediction of change of late-life major depression. Journal of Affective Disorders. 2017; 221 :192–197. doi: 10.1016/j.jad.2017.06.037. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Alexopoulos GS, Raue PJ, Gunning F, Kiosses DN, Kanellopoulos D, Pollari C, et al. Engage therapy: Behavioral activation and improvement of late-life major depression. American Journal of Geriatric Psychiatry. 2016; 24 (4):320–326. doi: 10.1016/j.jagp.2015.11.006. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Alexopoulos GS, Raue PJ, Kiosses DN, Seirup JK, Banerjee S, Arean PA. Comparing engage with PST in late-life major depression: A preliminary report. American Journal of Geriatric Psychiatry. 2015; 23 (5):506–513. doi: 10.1016/j.jagp.2014.06.008. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Appelbaum Paul S. Moving Toward the Future in the Diagnosis of Mental Disorders. Psychological Science in the Public Interest. 2017; 18 (2):67–71. doi: 10.1177/1529100617727267. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Beck AT, Bredemeier K. A unified model of depression. Clinical Psychological Science. 2016; 4 (4):596–619. doi: 10.1177/2167702616628523. [ CrossRef ] [ Google Scholar ]
• Berkovich-Ohana A, Glicksohn J, Goldstein A. Mindfulness-induced changes in gamma band activity – implications for the default mode network, self-reference and attention. Clinical Neurophysiology. 2012; 123 (4):700–710. doi: 10.1016/j.clinph.2011.07.048. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Blom EH, Duncan LG, Ho TC, Connolly CG, LeWinn KZ, Chesney M, et al. The development of an RDoC-based treatment program for adolescent depression:“Training for Awareness, Resilience, and Action”(TARA) Frontiers in Human Neuroscience. 2014; 8 :630. doi: 10.3389/fnhum.2014.00630. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Blom, E. H., Tymofiyeva, O., Chesney, M. A., Ho, T. C., Moran, P., Connolly, C. G., … Yang, T. T. (2017). Feasibility and preliminary efficacy of a novel RDoC-based treatment program for adolescent depression: “Training for Awareness Resilience and Action” (TARA)—A Pilot Study. Frontiers in Psychiatry , 7 . 10.3389/fpsyt.2016.00208. [ PMC free article ] [ PubMed ]
• Carcone, D., & Ruocco, A. C. (2017). Six years of research on the National Institute of Mental Health’s Research Domain Criteria (RDoC) Initiative: A systematic review. Frontiers in Cellular Neuroscience , 11 . 10.3389/fncel.2017.00046. [ PMC free article ] [ PubMed ]
• Clark D, Konduru N, Kemp A, Bray S, Brown E, Goodyear B, Ramasubbu R. The impact of age of onset on amygdala intrinsic connectivity in major depression. Neuropsychiatric Disease and Treatment. 2018; 14 :343–352. doi: 10.2147/NDT.S145042. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Clark LA, Cuthbert B, Lewis-Fernández R, Narrow WE, Reed GM. Three Approaches to understanding and classifying mental disorder: ICD-11, DSM-5 , and the National Institute of Mental Health’s Research Domain Criteria (RDoC) Psychological Science in the Public Interest. 2017; 18 (2):72–145. doi: 10.1177/1529100617727266. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Clementz BA, Sweeney JA, Hamm JP, Ivleva EI, Ethridge LE, Pearlson GD, et al. Identification of distinct psychosis biotypes using brain-based biomarkers. American Journal of Psychiatry. 2016; 173 (4):373–384. doi: 10.1176/appi.ajp.2015.14091200. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Cullen KR, Westlund MK, Klimes-Dougan B, Mueller BA, Houri A, Eberly LE, Lim KO. Abnormal amygdala resting-state functional connectivity in adolescent depression. JAMA Psychiatry. 2014; 71 (10):1138–1147. doi: 10.1001/jamapsychiatry.2014.1087. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Cuthbert, B. N., & Insel, T. R. (2013). Toward the future of psychiatric diagnosis: The seven pillars of RDoC. BMC Medicine , 11 (1). 10.1186/1741-7015-11-126. [ PMC free article ] [ PubMed ]
• Cuthbert BN, Kozak MJ. Constructing constructs for psychopathology: The NIMH Research Domain Criteria. Journal of Abnormal Psychology. 2013; 122 (3):928–937. doi: 10.1037/a0034028. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• David D, Montgomery GH. The scientific status of psychotherapies: A new evaluative framework for evidence-based psychosocial interventions. Clinical Psychology: Science and Practice. 2011; 18 (2):89–99. doi: 10.1111/j.1468-2850.2011.01239.x. [ CrossRef ] [ Google Scholar ]
• Drysdale AT, Grosenick L, Downar J, Dunlop K, Mansouri F, Meng Y, et al. Resting-state connectivity biomarkers define neurophysiological subtypes of depression. Nature Medicine. 2016; 23 (1):28–38. doi: 10.1038/nm.4246. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Emmelkamp PMG, David D, Beckers T, Muris P, Cuijpers P, Lutz W, et al. Advancing psychotherapy and evidence-based psychological interventions. International Journal of Methods in Psychiatric Research. 2014; 23 (S1):58–91. doi: 10.1002/mpr.1411. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ferster CB. A functional analysis of depression. American Psychologist. 1973; 28 (10):857–870. doi: 10.1037/h0035605. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Fowler CH, Miernicki ME, Rudolph KD, Telzer EH. Disrupted amygdala-prefrontal connectivity during emotion regulation links stress-reactive rumination and adolescent depressive symptoms. Developmental Cognitive Neuroscience. 2017; 27 :99–106. doi: 10.1016/j.dcn.2017.09.002. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Geraldo Andreia, Azeredo Andreia, Pasion Rita, Dores Artemisa Rocha, Barbosa Fernando. Fostering advances to neuropsychological assessment based on the Research Domain Criteria: The bridge between cognitive functioning and physiology. The Clinical Neuropsychologist. 2018; 33 (2):327–356. doi: 10.1080/13854046.2018.1523467. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hamilton JP, Furman DJ, Chang C, Thomason ME, Dennis E, Gotlib IH. Default-mode and task-positive network activity in major depressive disorder: Implications for adaptive and maladaptive rumination. Biological Psychiatry. 2011; 70 (4):327–333. doi: 10.1016/j.biopsych.2011.02.003. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hershenberg R, Goldfried MR. Implications of RDoC for the research and practice of psychotherapy. Behavior Therapy. 2015; 46 (2):156–165. doi: 10.1016/j.beth.2014.09.014. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Insel T, Cuthbert B, Garvey M, Heinssen R, Pine DS, Quinn K, et al. Research Domain Criteria (RDoC): Toward a new classification framework for research on mental disorders. American Journal of Psychiatry. 2010; 167 (7):748–751. doi: 10.1176/appi.ajp.2010.09091379. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kerestes R, Harrison BJ, Dandash O, Stephanou K, Whittle S, Pujol J, Davey CG. Specific functional connectivity alterations of the dorsal striatum in young people with depression. NeuroImage: Clinical. 2015; 7 :266–272. doi: 10.1016/j.nicl.2014.12.017. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Koszycki D, Thake J, Mavounza C, Daoust J-P, Taljaard M, Bradwejn J. Preliminary investigation of a mindfulness-based intervention for social anxiety disorder that integrates compassion meditation and mindful exposure. The Journal of Alternative and Complementary Medicine. 2016; 22 (5):363–374. doi: 10.1089/acm.2015.0108. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kozak MJ, Cuthbert BN. The NIMH Research Domain Criteria Initiative: background, issues, and pragmatics. Psychophysiology. 2016; 53 (3):286–297. doi: 10.1111/psyp.12518. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Levant RF. Report of the 2005 presidential task force on evidence-based practice . Washington: American Psychological Association; 2005. [ Google Scholar ]
• Marchand William R. Self-Referential Thinking, Suicide, and Function of the Cortical Midline Structures and Striatum in Mood Disorders: Possible Implications for Treatment Studies of Mindfulness-Based Interventions for Bipolar Depression. Depression Research and Treatment. 2012; 2012 :1–15. doi: 10.1155/2012/246725. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Marková IS. Translational neuroscience and psychiatry: A conceptual analysis. Journal of Evaluation in Clinical Practice. 2018; 24 (4):791–796. doi: 10.1111/jep.12914. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• McKay Dean, Tolin David F. Empirically supported psychological treatments and the Research Domain Criteria (RDoC) Journal of Affective Disorders. 2017; 216 :78–88. doi: 10.1016/j.jad.2016.10.018. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Mittal VA, Wakschlag LS. Research Domain Criteria (RDoC) grows up: Strengthening neurodevelopment investigation within the RDoC framework. Journal of Affective Disorders. 2017; 216 :30–35. doi: 10.1016/j.jad.2016.12.011. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Morris SE, Rumsey JM, Cuthbert BN. Rethinking mental disorders: The role of learning and brain plasticity. Restorative Neurology and Neuroscience. 2014; 32 (1):5–23. doi: 10.3233/RNN-139015. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Pasion R, Paiva TO, Fernandes C, Almeida R, Barbosa F. ERN modulation under sustained threat: A pre-registered report. Journal of Research in Personality. 2018; 77 :137–146. doi: 10.1016/j.jrp.2018.10.009. [ CrossRef ] [ Google Scholar ]
• Patrick CJ, Hajcak G. RDoC: Translating promise into progress. Psychophysiology. 2016; 53 (3):415–424. doi: 10.1111/psyp.12612. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Peterson BS. Editorial: Research Domain Criteria (RDoC): a new psychiatric nosology whose time has not yet come. Journal of Child Psychology and Psychiatry. 2015; 56 (7):719–722. doi: 10.1111/jcpp.12439. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Price JL, Drevets WC. Neural circuits underlying the pathophysiology of mood disorders. Trends in Cognitive Sciences. 2012; 16 (1):61–71. doi: 10.1016/j.tics.2011.12.011. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• RDoC. (2018). RDoC matrix - version 4. Retrieved from https://www.nimh.nih.gov/research/research-funded-by-nimh/rdoc/constructs/rdoc-snapshot-version-4-saved-5-30-18.shtml
• RDoC. (2019). RDoC constructs. Retrieved from https://www.nimh.nih.gov/research-priorities/rdoc/constructs/index.shtml
• Roberts MC, Blossom JB, Evans SC, Amaro CM, Kanine RM. Advancing the scientific foundation for evidence-based practice in clinical child and adolescent psychology. Journal of Clinical Child & Adolescent Psychology. 2017; 46 (6):915–928. doi: 10.1080/15374416.2016.1152554. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Sharp C, Christopher Fowler J, Salas R, Nielsen D, Allen J, Oldham J, et al. Operationalizing NIMH Research Domain Criteria (RDoC) in naturalistic clinical settings. Bulletin of the Menninger Clinic. 2016; 80 (3):187–212. doi: 10.1521/bumc.2016.80.3.187. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Shoham V, Rohrbaugh MJ, Onken LS, Cuthbert BN, Beveridge RM, Fowles TR. Redefining clinical science training: Purpose and products of the delaware project. Clinical Psychological Science. 2014; 2 (1):8–21. doi: 10.1177/2167702613497931. [ CrossRef ] [ Google Scholar ]
• Taren AA, Gianaros PJ, Greco CM, Lindsay EK, Fairgrieve A, Brown KW, et al. Mindfulness meditation training alters stress-related amygdala resting state functional connectivity: A randomized controlled trial. Social Cognitive and Affective Neuroscience. 2015; 10 (12):1758–1768. doi: 10.1093/scan/nsv066. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Van Dam NT, O’Connor D, Marcelle ET, Ho EJ, Cameron Craddock R, Tobe RH, et al. Data-driven phenotypic categorization for neurobiological analyses: Beyond DSM-5 labels. Biological Psychiatry. 2017; 81 (6):484–494. doi: 10.1016/j.biopsych.2016.06.027. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Yancey JR, Venables NC, Patrick CJ. Psychoneurometric operationalization of threat sensitivity: Relations with clinical symptom and physiological response criteria. Psychophysiology. 2016; 53 (3):393–405. doi: 10.1111/psyp.12512. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

The Interplay Between Public Procurement of Innovation and R&D Grants: Empirical Evidence From Belgium 

ZEW - Centre for European Economic Research Discussion Paper No. 24-055

36 Pages Posted: 19 Aug 2024

Dirk Czarnitzki

KU Leuven - Department of Managerial Economics, Strategy, and Innovation; Centre for European Economic Research (ZEW)

Malte Prüfer

KU Leuven - Faculty of Economics and Business

Date Written: July 01, 2024

This paper investigates the impact of Public Procurement of Innovation (PPI) and Research and Development (R&D) grants on firms' R&D investment using data from Belgian R&D-active firms over the past decade. Our empirical analysis robustly reveals a non-negligible crowding-out effect between the two instruments, suggesting a substitutive relationship. While each policy individually positively influences R&D investment, their combined implementation diminishes their effectiveness. These results challenge prevailing evidence and emphasize the need for a careful policy implementation, raising policymakers' awareness against a blanket increase in innovation policies without considering potential interactions.

Keywords: Public procurement of innovation, Research and Development, Econometric policy evaluation, Crowding-out

JEL Classification: H57, O38

Suggested Citation: Suggested Citation

Dirk Czarnitzki (Contact Author)

Ku leuven - department of managerial economics, strategy, and innovation ( email ).

Naamsestraat 69 bus 3500 Leuven, 3000 Belgium

Centre for European Economic Research (ZEW) ( email )

P.O. Box 10 34 43 Mannheim, 68034 Germany

KU Leuven - Faculty of Economics and Business ( email )

Korte Nieuwstraat 33 2000 Antwerpen Belgium

Do you have a job opening that you would like to promote on SSRN?

Paper statistics, related ejournals, social entrepreneurship ejournal.

Subscribe to this fee journal for more curated articles on this topic

ZEW - Leibniz Centre for European Economic Research Discussion Paper Series

Subscribe to this free journal for more curated articles on this topic