what is method in literature review

What is a Literature Review? How to Write It (with Examples)

A literature review is a critical analysis and synthesis of existing research on a particular topic. It provides an overview of the current state of knowledge, identifies gaps, and highlights key findings in the literature. 1 The purpose of a literature review is to situate your own research within the context of existing scholarship, demonstrating your understanding of the topic and showing how your work contributes to the ongoing conversation in the field. Learning how to write a literature review is a critical tool for successful research. Your ability to summarize and synthesize prior research pertaining to a certain topic demonstrates your grasp on the topic of study, and assists in the learning process. 

Table of Contents

What is the purpose of literature review , a. habitat loss and species extinction: , b. range shifts and phenological changes: , c. ocean acidification and coral reefs: , d. adaptive strategies and conservation efforts: .

• Choose a Topic and Define the Research Question: 
• Decide on the Scope of Your Review: 
• Select Databases for Searches: 
• Conduct Searches and Keep Track: 
• Review the Literature: 
• Organize and Write Your Literature Review: 
• How to write a literature review faster with Paperpal? 

Frequently asked questions 

What is a literature review .

A well-conducted literature review demonstrates the researcher’s familiarity with the existing literature, establishes the context for their own research, and contributes to scholarly conversations on the topic. One of the purposes of a literature review is also to help researchers avoid duplicating previous work and ensure that their research is informed by and builds upon the existing body of knowledge.

A literature review serves several important purposes within academic and research contexts. Here are some key objectives and functions of a literature review: 2  

1. Contextualizing the Research Problem: The literature review provides a background and context for the research problem under investigation. It helps to situate the study within the existing body of knowledge. 

2. Identifying Gaps in Knowledge: By identifying gaps, contradictions, or areas requiring further research, the researcher can shape the research question and justify the significance of the study. This is crucial for ensuring that the new research contributes something novel to the field.

Find academic papers related to your research topic faster. Try Research on Paperpal

3. Understanding Theoretical and Conceptual Frameworks: Literature reviews help researchers gain an understanding of the theoretical and conceptual frameworks used in previous studies. This aids in the development of a theoretical framework for the current research. 

4. Providing Methodological Insights: Another purpose of literature reviews is that it allows researchers to learn about the methodologies employed in previous studies. This can help in choosing appropriate research methods for the current study and avoiding pitfalls that others may have encountered. 

5. Establishing Credibility: A well-conducted literature review demonstrates the researcher’s familiarity with existing scholarship, establishing their credibility and expertise in the field. It also helps in building a solid foundation for the new research. 

6. Informing Hypotheses or Research Questions: The literature review guides the formulation of hypotheses or research questions by highlighting relevant findings and areas of uncertainty in existing literature. 

Literature review example 

Let’s delve deeper with a literature review example: Let’s say your literature review is about the impact of climate change on biodiversity. You might format your literature review into sections such as the effects of climate change on habitat loss and species extinction, phenological changes, and marine biodiversity. Each section would then summarize and analyze relevant studies in those areas, highlighting key findings and identifying gaps in the research. The review would conclude by emphasizing the need for further research on specific aspects of the relationship between climate change and biodiversity. The following literature review template provides a glimpse into the recommended literature review structure and content, demonstrating how research findings are organized around specific themes within a broader topic. 

Literature Review on Climate Change Impacts on Biodiversity:  

Climate change is a global phenomenon with far-reaching consequences, including significant impacts on biodiversity. This literature review synthesizes key findings from various studies: 

Climate change-induced alterations in temperature and precipitation patterns contribute to habitat loss, affecting numerous species (Thomas et al., 2004). The review discusses how these changes increase the risk of extinction, particularly for species with specific habitat requirements. 

Observations of range shifts and changes in the timing of biological events (phenology) are documented in response to changing climatic conditions (Parmesan & Yohe, 2003). These shifts affect ecosystems and may lead to mismatches between species and their resources. 

The review explores the impact of climate change on marine biodiversity, emphasizing ocean acidification’s threat to coral reefs (Hoegh-Guldberg et al., 2007). Changes in pH levels negatively affect coral calcification, disrupting the delicate balance of marine ecosystems. 

Recognizing the urgency of the situation, the literature review discusses various adaptive strategies adopted by species and conservation efforts aimed at mitigating the impacts of climate change on biodiversity (Hannah et al., 2007). It emphasizes the importance of interdisciplinary approaches for effective conservation planning. 

Strengthen your literature review with factual insights. Try Research on Paperpal for free!

How to write a good literature review 

Writing a literature review involves summarizing and synthesizing existing research on a particular topic. A good literature review format should include the following elements. 

Introduction: The introduction sets the stage for your literature review, providing context and introducing the main focus of your review. 

• Opening Statement: Begin with a general statement about the broader topic and its significance in the field. 
• Scope and Purpose: Clearly define the scope of your literature review. Explain the specific research question or objective you aim to address. 
• Organizational Framework: Briefly outline the structure of your literature review, indicating how you will categorize and discuss the existing research. 
• Significance of the Study: Highlight why your literature review is important and how it contributes to the understanding of the chosen topic. 
• Thesis Statement: Conclude the introduction with a concise thesis statement that outlines the main argument or perspective you will develop in the body of the literature review. 

Body: The body of the literature review is where you provide a comprehensive analysis of existing literature, grouping studies based on themes, methodologies, or other relevant criteria. 

• Organize by Theme or Concept: Group studies that share common themes, concepts, or methodologies. Discuss each theme or concept in detail, summarizing key findings and identifying gaps or areas of disagreement. 
• Critical Analysis: Evaluate the strengths and weaknesses of each study. Discuss the methodologies used, the quality of evidence, and the overall contribution of each work to the understanding of the topic. 
• Synthesis of Findings: Synthesize the information from different studies to highlight trends, patterns, or areas of consensus in the literature. 
• Identification of Gaps: Discuss any gaps or limitations in the existing research and explain how your review contributes to filling these gaps. 
• Transition between Sections: Provide smooth transitions between different themes or concepts to maintain the flow of your literature review. 

Write and Cite as yo u go with Paperpal Research. Start now for free!

Conclusion: The conclusion of your literature review should summarize the main findings, highlight the contributions of the review, and suggest avenues for future research. 

• Summary of Key Findings: Recap the main findings from the literature and restate how they contribute to your research question or objective. 
• Contributions to the Field: Discuss the overall contribution of your literature review to the existing knowledge in the field. 
• Implications and Applications: Explore the practical implications of the findings and suggest how they might impact future research or practice. 
• Recommendations for Future Research: Identify areas that require further investigation and propose potential directions for future research in the field. 
• Final Thoughts: Conclude with a final reflection on the importance of your literature review and its relevance to the broader academic community. 

Conducting a literature review 

Conducting a literature review is an essential step in research that involves reviewing and analyzing existing literature on a specific topic. It’s important to know how to do a literature review effectively, so here are the steps to follow: 1  

Choose a Topic and Define the Research Question:  

• Select a topic that is relevant to your field of study. 
• Clearly define your research question or objective. Determine what specific aspect of the topic do you want to explore? 

Decide on the Scope of Your Review:  

• Determine the timeframe for your literature review. Are you focusing on recent developments, or do you want a historical overview? 
• Consider the geographical scope. Is your review global, or are you focusing on a specific region? 
• Define the inclusion and exclusion criteria. What types of sources will you include? Are there specific types of studies or publications you will exclude? 

Select Databases for Searches:  

• Identify relevant databases for your field. Examples include PubMed, IEEE Xplore, Scopus, Web of Science, and Google Scholar. 
• Consider searching in library catalogs, institutional repositories, and specialized databases related to your topic. 

Conduct Searches and Keep Track:  

• Develop a systematic search strategy using keywords, Boolean operators (AND, OR, NOT), and other search techniques. 
• Record and document your search strategy for transparency and replicability. 
• Keep track of the articles, including publication details, abstracts, and links. Use citation management tools like EndNote, Zotero, or Mendeley to organize your references. 

Review the Literature:  

• Evaluate the relevance and quality of each source. Consider the methodology, sample size, and results of studies. 
• Organize the literature by themes or key concepts. Identify patterns, trends, and gaps in the existing research. 
• Summarize key findings and arguments from each source. Compare and contrast different perspectives. 
• Identify areas where there is a consensus in the literature and where there are conflicting opinions. 
• Provide critical analysis and synthesis of the literature. What are the strengths and weaknesses of existing research? 

Organize and Write Your Literature Review:  

• Literature review outline should be based on themes, chronological order, or methodological approaches. 
• Write a clear and coherent narrative that synthesizes the information gathered. 
• Use proper citations for each source and ensure consistency in your citation style (APA, MLA, Chicago, etc.). 
• Conclude your literature review by summarizing key findings, identifying gaps, and suggesting areas for future research. 

Whether you’re exploring a new research field or finding new angles to develop an existing topic, sifting through hundreds of papers can take more time than you have to spare. But what if you could find science-backed insights with verified citations in seconds? That’s the power of Paperpal’s new Research feature!  

How to write a literature review faster with Paperpal?  

Paperpal, an AI writing assistant, integrates powerful academic search capabilities within its writing platform. With the Research | Cite feature, you get 100% factual insights, with citations backed by 250M+ verified research articles, directly within your writing interface. It also allows you auto-cite references in 10,000+ styles and save relevant references in your Citation Library. By eliminating the need to switch tabs to find answers to all your research questions, Paperpal saves time and helps you stay focused on your writing.   

Here’s how to use the Research feature:  

• Ask a question: Get started with a new document on paperpal.com. Click on the “Research | Cite” feature and type your question in plain English. Paperpal will scour over 250 million research articles, including conference papers and preprints, to provide you with accurate insights and citations. 

• Review and Save: Paperpal summarizes the information, while citing sources and listing relevant reads. You can quickly scan the results to identify relevant references and save these directly to your built-in citations library for later access. 
• Cite with Confidence: Paperpal makes it easy to incorporate relevant citations and references in 10,000+ styles into your writing, ensuring your arguments are well-supported by credible sources. This translates to a polished, well-researched literature review. 

The literature review sample and detailed advice on writing and conducting a review will help you produce a well-structured report. But remember that a good literature review is an ongoing process, and it may be necessary to revisit and update it as your research progresses. By combining effortless research with an easy citation process, Paperpal Research streamlines the literature review process and empowers you to write faster and with more confidence. Try Paperpal Research now and see for yourself.  

A literature review is a critical and comprehensive analysis of existing literature (published and unpublished works) on a specific topic or research question and provides a synthesis of the current state of knowledge in a particular field. A well-conducted literature review is crucial for researchers to build upon existing knowledge, avoid duplication of efforts, and contribute to the advancement of their field. It also helps researchers situate their work within a broader context and facilitates the development of a sound theoretical and conceptual framework for their studies.

Literature review is a crucial component of research writing, providing a solid background for a research paper’s investigation. The aim is to keep professionals up to date by providing an understanding of ongoing developments within a specific field, including research methods, and experimental techniques used in that field, and present that knowledge in the form of a written report. Also, the depth and breadth of the literature review emphasizes the credibility of the scholar in his or her field.  

Before writing a literature review, it’s essential to undertake several preparatory steps to ensure that your review is well-researched, organized, and focused. This includes choosing a topic of general interest to you and doing exploratory research on that topic, writing an annotated bibliography, and noting major points, especially those that relate to the position you have taken on the topic. 

Literature reviews and academic research papers are essential components of scholarly work but serve different purposes within the academic realm. 3 A literature review aims to provide a foundation for understanding the current state of research on a particular topic, identify gaps or controversies, and lay the groundwork for future research. Therefore, it draws heavily from existing academic sources, including books, journal articles, and other scholarly publications. In contrast, an academic research paper aims to present new knowledge, contribute to the academic discourse, and advance the understanding of a specific research question. Therefore, it involves a mix of existing literature (in the introduction and literature review sections) and original data or findings obtained through research methods. 

Literature reviews are essential components of academic and research papers, and various strategies can be employed to conduct them effectively. If you want to know how to write a literature review for a research paper, here are four common approaches that are often used by researchers.  Chronological Review: This strategy involves organizing the literature based on the chronological order of publication. It helps to trace the development of a topic over time, showing how ideas, theories, and research have evolved.  Thematic Review: Thematic reviews focus on identifying and analyzing themes or topics that cut across different studies. Instead of organizing the literature chronologically, it is grouped by key themes or concepts, allowing for a comprehensive exploration of various aspects of the topic.  Methodological Review: This strategy involves organizing the literature based on the research methods employed in different studies. It helps to highlight the strengths and weaknesses of various methodologies and allows the reader to evaluate the reliability and validity of the research findings.  Theoretical Review: A theoretical review examines the literature based on the theoretical frameworks used in different studies. This approach helps to identify the key theories that have been applied to the topic and assess their contributions to the understanding of the subject.  It’s important to note that these strategies are not mutually exclusive, and a literature review may combine elements of more than one approach. The choice of strategy depends on the research question, the nature of the literature available, and the goals of the review. Additionally, other strategies, such as integrative reviews or systematic reviews, may be employed depending on the specific requirements of the research.

The literature review format can vary depending on the specific publication guidelines. However, there are some common elements and structures that are often followed. Here is a general guideline for the format of a literature review:  Introduction:   Provide an overview of the topic.  Define the scope and purpose of the literature review.  State the research question or objective.  Body:   Organize the literature by themes, concepts, or chronology.  Critically analyze and evaluate each source.  Discuss the strengths and weaknesses of the studies.  Highlight any methodological limitations or biases.  Identify patterns, connections, or contradictions in the existing research.  Conclusion:   Summarize the key points discussed in the literature review.  Highlight the research gap.  Address the research question or objective stated in the introduction.  Highlight the contributions of the review and suggest directions for future research.

Both annotated bibliographies and literature reviews involve the examination of scholarly sources. While annotated bibliographies focus on individual sources with brief annotations, literature reviews provide a more in-depth, integrated, and comprehensive analysis of existing literature on a specific topic. The key differences are as follows: 

References 

• Denney, A. S., & Tewksbury, R. (2013). How to write a literature review.  Journal of criminal justice education ,  24 (2), 218-234. 
• Pan, M. L. (2016).  Preparing literature reviews: Qualitative and quantitative approaches . Taylor & Francis. 
• Cantero, C. (2019). How to write a literature review.  San José State University Writing Center . 

Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 22+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.  

Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.  

Experience the future of academic writing – Sign up to Paperpal and start writing for free!  

Related Reads:

• Empirical Research: A Comprehensive Guide for Academics 
• How to Write a Scientific Paper in 10 Steps 
• How Long Should a Chapter Be?
• How to Use Paperpal to Generate Emails & Cover Letters?

6 Tips for Post-Doc Researchers to Take Their Career to the Next Level

Self-plagiarism in research: what it is and how to avoid it, you may also like, academic integrity vs academic dishonesty: types & examples, dissertation printing and binding | types & comparison , what is a dissertation preface definition and examples , the ai revolution: authors’ role in upholding academic..., the future of academia: how ai tools are..., how to write a research proposal: (with examples..., how to write your research paper in apa..., how to choose a dissertation topic, how to write a phd research proposal, how to write an academic paragraph (step-by-step guide).

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Dissertation
• What is a Literature Review? | Guide, Template, & Examples

What is a Literature Review? | Guide, Template, & Examples

Published on 22 February 2022 by Shona McCombes . Revised on 7 June 2022.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research.

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarise sources – it analyses, synthesises, and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Table of contents

Why write a literature review, examples of literature reviews, step 1: search for relevant literature, step 2: evaluate and select sources, step 3: identify themes, debates and gaps, step 4: outline your literature review’s structure, step 5: write your literature review, frequently asked questions about literature reviews, introduction.

• Quick Run-through
• Step 1 & 2

When you write a dissertation or thesis, you will have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and scholarly context
• Develop a theoretical framework and methodology for your research
• Position yourself in relation to other researchers and theorists
• Show how your dissertation addresses a gap or contributes to a debate

You might also have to write a literature review as a stand-alone assignment. In this case, the purpose is to evaluate the current state of research and demonstrate your knowledge of scholarly debates around a topic.

The content will look slightly different in each case, but the process of conducting a literature review follows the same steps. We’ve written a step-by-step guide that you can follow below.

The only proofreading tool specialized in correcting academic writing

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Correct my document today

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research objectives and questions .

If you are writing a literature review as a stand-alone assignment, you will have to choose a focus and develop a central question to direct your search. Unlike a dissertation research question, this question has to be answerable without collecting original data. You should be able to answer it based only on a review of existing publications.

Make a list of keywords

Start by creating a list of keywords related to your research topic. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list if you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can use boolean operators to help narrow down your search:

Read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

To identify the most important publications on your topic, take note of recurring citations. If the same authors, books or articles keep appearing in your reading, make sure to seek them out.

You probably won’t be able to read absolutely everything that has been written on the topic – you’ll have to evaluate which sources are most relevant to your questions.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models and methods? Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• How does the publication contribute to your understanding of the topic? What are its key insights and arguments?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible, and make sure you read any landmark studies and major theories in your field of research.

You can find out how many times an article has been cited on Google Scholar – a high citation count means the article has been influential in the field, and should certainly be included in your literature review.

The scope of your review will depend on your topic and discipline: in the sciences you usually only review recent literature, but in the humanities you might take a long historical perspective (for example, to trace how a concept has changed in meaning over time).

Remember that you can use our template to summarise and evaluate sources you’re thinking about using!

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It’s important to keep track of your sources with references to avoid plagiarism . It can be helpful to make an annotated bibliography, where you compile full reference information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

You can use our free APA Reference Generator for quick, correct, consistent citations.

Prevent plagiarism, run a free check.

To begin organising your literature review’s argument and structure, you need to understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly-visual platforms like Instagram and Snapchat – this is a gap that you could address in your own research.

There are various approaches to organising the body of a literature review. You should have a rough idea of your strategy before you start writing.

Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarising sources in order.

Try to analyse patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organise your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text, your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

If you are writing the literature review as part of your dissertation or thesis, reiterate your central problem or research question and give a brief summary of the scholarly context. You can emphasise the timeliness of the topic (“many recent studies have focused on the problem of x”) or highlight a gap in the literature (“while there has been much research on x, few researchers have taken y into consideration”).

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, make sure to follow these tips:

• Summarise and synthesise: give an overview of the main points of each source and combine them into a coherent whole.
• Analyse and interpret: don’t just paraphrase other researchers – add your own interpretations, discussing the significance of findings in relation to the literature as a whole.
• Critically evaluate: mention the strengths and weaknesses of your sources.
• Write in well-structured paragraphs: use transitions and topic sentences to draw connections, comparisons and contrasts.

In the conclusion, you should summarise the key findings you have taken from the literature and emphasise their significance.

If the literature review is part of your dissertation or thesis, reiterate how your research addresses gaps and contributes new knowledge, or discuss how you have drawn on existing theories and methods to build a framework for your research. This can lead directly into your methodology section.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your  dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. (2022, June 07). What is a Literature Review? | Guide, Template, & Examples. Scribbr. Retrieved 9 September 2024, from https://www.scribbr.co.uk/thesis-dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, how to write a dissertation proposal | a step-by-step guide, what is a theoretical framework | a step-by-step guide, what is a research methodology | steps & tips.

Libraries | Research Guides

Literature reviews, what is a literature review, learning more about how to do a literature review.

• Planning the Review
• The Research Question
• Choosing Where to Search
• Organizing the Review
• Writing the Review

A literature review is a review and synthesis of existing research on a topic or research question. A literature review is meant to analyze the scholarly literature, make connections across writings and identify strengths, weaknesses, trends, and missing conversations. A literature review should address different aspects of a topic as it relates to your research question. A literature review goes beyond a description or summary of the literature you have read. 

• Sage Research Methods Core This link opens in a new window SAGE Research Methods supports research at all levels by providing material to guide users through every step of the research process. SAGE Research Methods is the ultimate methods library with more than 1000 books, reference works, journal articles, and instructional videos by world-leading academics from across the social sciences, including the largest collection of qualitative methods books available online from any scholarly publisher. – Publisher

• Next: Planning the Review >>
• Last Updated: Jul 8, 2024 11:22 AM
• URL: https://libguides.northwestern.edu/literaturereviews

• UConn Library
• Literature Review: The What, Why and How-to Guide
• Introduction

Literature Review: The What, Why and How-to Guide — Introduction

• Getting Started
• How to Pick a Topic
• Strategies to Find Sources
• Evaluating Sources & Lit. Reviews
• Tips for Writing Literature Reviews
• Writing Literature Review: Useful Sites
• Citation Resources
• Other Academic Writings

What are Literature Reviews?

So, what is a literature review? "A literature review is an account of what has been published on a topic by accredited scholars and researchers. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries." Taylor, D.  The literature review: A few tips on conducting it . University of Toronto Health Sciences Writing Centre.

Goals of Literature Reviews

What are the goals of creating a Literature Review?  A literature could be written to accomplish different aims:

• To develop a theory or evaluate an existing theory
• To summarize the historical or existing state of a research topic
• Identify a problem in a field of research 

Baumeister, R. F., & Leary, M. R. (1997). Writing narrative literature reviews .  Review of General Psychology , 1 (3), 311-320.

What kinds of sources require a Literature Review?

• A research paper assigned in a course
• A thesis or dissertation
• A grant proposal
• An article intended for publication in a journal

All these instances require you to collect what has been written about your research topic so that you can demonstrate how your own research sheds new light on the topic.

Types of Literature Reviews

What kinds of literature reviews are written?

Narrative review: The purpose of this type of review is to describe the current state of the research on a specific topic/research and to offer a critical analysis of the literature reviewed. Studies are grouped by research/theoretical categories, and themes and trends, strengths and weakness, and gaps are identified. The review ends with a conclusion section which summarizes the findings regarding the state of the research of the specific study, the gaps identify and if applicable, explains how the author's research will address gaps identify in the review and expand the knowledge on the topic reviewed.

• Example : Predictors and Outcomes of U.S. Quality Maternity Leave: A Review and Conceptual Framework:  10.1177/08948453211037398  

Systematic review : "The authors of a systematic review use a specific procedure to search the research literature, select the studies to include in their review, and critically evaluate the studies they find." (p. 139). Nelson, L. K. (2013). Research in Communication Sciences and Disorders . Plural Publishing.

• Example : The effect of leave policies on increasing fertility: a systematic review:  10.1057/s41599-022-01270-w

Meta-analysis : "Meta-analysis is a method of reviewing research findings in a quantitative fashion by transforming the data from individual studies into what is called an effect size and then pooling and analyzing this information. The basic goal in meta-analysis is to explain why different outcomes have occurred in different studies." (p. 197). Roberts, M. C., & Ilardi, S. S. (2003). Handbook of Research Methods in Clinical Psychology . Blackwell Publishing.

• Example : Employment Instability and Fertility in Europe: A Meta-Analysis:  10.1215/00703370-9164737

Meta-synthesis : "Qualitative meta-synthesis is a type of qualitative study that uses as data the findings from other qualitative studies linked by the same or related topic." (p.312). Zimmer, L. (2006). Qualitative meta-synthesis: A question of dialoguing with texts .  Journal of Advanced Nursing , 53 (3), 311-318.

• Example : Women’s perspectives on career successes and barriers: A qualitative meta-synthesis:  10.1177/05390184221113735

Literature Reviews in the Health Sciences

• UConn Health subject guide on systematic reviews Explanation of the different review types used in health sciences literature as well as tools to help you find the right review type
• << Previous: Getting Started
• Next: How to Pick a Topic >>
• Last Updated: Sep 21, 2022 2:16 PM
• URL: https://guides.lib.uconn.edu/literaturereview

Research Methods

• Getting Started
• Literature Review Research
• Research Design
• Research Design By Discipline
• SAGE Research Methods
• Teaching with SAGE Research Methods

Literature Review

• What is a Literature Review?
• What is NOT a Literature Review?
• Purposes of a Literature Review
• Types of Literature Reviews
• Literature Reviews vs. Systematic Reviews
• Systematic vs. Meta-Analysis

Literature Review  is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

• Summarizes and analyzes previous research relevant to a topic
• Includes scholarly books and articles published in academic journals
• Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

• Help gather ideas or information
• Keep up to date in current trends and findings
• Help develop new questions

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Helps focus your own research questions or problems
• Discovers relationships between research studies/ideas.
• Suggests unexplored ideas or populations
• Identifies major themes, concepts, and researchers on a topic.
• Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
• Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
• Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University 

Keep in mind the following, a literature review is NOT:

Not an essay 

Not an annotated bibliography  in which you summarize each article that you have reviewed.  A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper   where you select resources to support one side of an issue versus another.  A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

• provides thorough knowledge of previous studies; introduces seminal works.
• helps focus one’s own research topic.
• identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
• suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
• identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
• helps the researcher avoid repetition of earlier research.
• suggests unexplored populations.
• determines whether past studies agree or disagree; identifies controversy in the literature.
• tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review      This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review      Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review      Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review      A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review      This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review      The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature."  Educational Researcher  36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015),  Literature reviews vs systematic reviews.  Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

Systematic review or meta-analysis?

A  systematic review  answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A  meta-analysis  is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

• clearly stated objectives with pre-defined eligibility criteria for studies
• explicit, reproducible methodology
• a systematic search that attempts to identify all studies
• assessment of the validity of the findings of the included studies (e.g. risk of bias)
• systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis. 

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review.  More information on meta-analyses can be found in  Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies.  It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings.  Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted.  In that case, an integrative review is an appropriate strategy. 

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

• << Previous: Getting Started
• Next: Research Design >>
• Last Updated: Jul 15, 2024 10:34 AM
• URL: https://guides.lib.udel.edu/researchmethods

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• CAREER FEATURE
• 04 December 2020
• Correction 09 December 2020

How to write a superb literature review

Andy Tay is a freelance writer based in Singapore.

You can also search for this author in PubMed   Google Scholar

Literature reviews are important resources for scientists. They provide historical context for a field while offering opinions on its future trajectory. Creating them can provide inspiration for one’s own research, as well as some practice in writing. But few scientists are trained in how to write a review — or in what constitutes an excellent one. Even picking the appropriate software to use can be an involved decision (see ‘Tools and techniques’). So Nature asked editors and working scientists with well-cited reviews for their tips.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-020-03422-x

Interviews have been edited for length and clarity.

Updates & Corrections

Correction 09 December 2020 : An earlier version of the tables in this article included some incorrect details about the programs Zotero, Endnote and Manubot. These have now been corrected.

Hsing, I.-M., Xu, Y. & Zhao, W. Electroanalysis 19 , 755–768 (2007).

Article   Google Scholar  

Ledesma, H. A. et al. Nature Nanotechnol. 14 , 645–657 (2019).

Article   PubMed   Google Scholar  

Brahlek, M., Koirala, N., Bansal, N. & Oh, S. Solid State Commun. 215–216 , 54–62 (2015).

Choi, Y. & Lee, S. Y. Nature Rev. Chem . https://doi.org/10.1038/s41570-020-00221-w (2020).

Download references

Related Articles

• Research management

The grassroots organizations continuing the fight for Ukrainian science

Career Feature 11 SEP 24

How a struggling biotech company became a university ‘spin-in’

Career Q&A 10 SEP 24

The human costs of the research-assessment culture

Career Feature 09 SEP 24

Why I’m committed to breaking the bias in large language models

Career Guide 04 SEP 24

Binning out-of-date chemicals? Somebody think about the carbon!

Correspondence 27 AUG 24

Publishing nightmare: a researcher’s quest to keep his own work from being plagiarized

News 04 SEP 24

Intellectual property and data privacy: the hidden risks of AI

Faculty Positions at SUSTech School of Medicine

SUSTech School of Medicine offers equal opportunities and welcome applicants from the world with all ethnic backgrounds.

Shenzhen, Guangdong, China

Southern University of Science and Technology, School of Medicine

Staff Scientist - Immunology

Staff Scientist- Immunology

Houston, Texas (US)

Baylor College of Medicine (BCM)

Institute for Systems Genetics, Tenure Track Faculty Positions

The Institute for Systems Genetics at NYU Langone Health has tenure track faculty positions (assistant professor level) at the new SynBioMed Center.

New York City, New York (US)

NYU Langone Health

Faculty Position

The Institute of Cellular and Organismic Biology (ICOB), Academia Sinica, Taiwan, is seeking candidates to fill multiple tenure-track faculty position

Taipei (TW)

Institute of Cellular and Organismic Biology, Academia Sinica

Postdoctoral Associate

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health (m-health) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jun 20, 2024 9:08 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

Research Methods and Design

• Action Research
• Case Study Design

Literature Review

• Quantitative Research Methods
• Qualitative Research Methods
• Mixed Methods Study
• Indigenous Research and Ethics This link opens in a new window
• Identifying Empirical Research Articles This link opens in a new window
• Research Ethics and Quality
• Data Literacy
• Get Help with Writing Assignments

A literature review is a discussion of the literature (aka. the "research" or "scholarship") surrounding a certain topic. A good literature review doesn't simply summarize the existing material, but provides thoughtful synthesis and analysis. The purpose of a literature review is to orient your own work within an existing body of knowledge. A literature review may be written as a standalone piece or be included in a larger body of work.

You can read more about literature reviews, what they entail, and how to write one, using the resources below. 

Am I the only one struggling to write a literature review?

Dr. Zina O'Leary explains the misconceptions and struggles students often have with writing a literature review. She also provides step-by-step guidance on writing a persuasive literature review.

An Introduction to Literature Reviews

Dr. Eric Jensen, Professor of Sociology at the University of Warwick, and Dr. Charles Laurie, Director of Research at Verisk Maplecroft, explain how to write a literature review, and why researchers need to do so. Literature reviews can be stand-alone research or part of a larger project. They communicate the state of academic knowledge on a given topic, specifically detailing what is still unknown.

This is the first video in a whole series about literature reviews. You can find the rest of the series in our SAGE database, Research Methods:

Videos covering research methods and statistics

Identify Themes and Gaps in Literature (with real examples) | Scribbr

Finding connections between sources is key to organizing the arguments and structure of a good literature review. In this video, you'll learn how to identify themes, debates, and gaps between sources, using examples from real papers.

4 Tips for Writing a Literature Review's Intro, Body, and Conclusion | Scribbr

While each review will be unique in its structure--based on both the existing body of both literature and the overall goals of your own paper, dissertation, or research--this video from Scribbr does a good job simplifying the goals of writing a literature review for those who are new to the process. In this video, you’ll learn what to include in each section, as well as 4 tips for the main body illustrated with an example.

• Literature Review This chapter in SAGE's Encyclopedia of Research Design describes the types of literature reviews and scientific standards for conducting literature reviews.
• UNC Writing Center: Literature Reviews This handout from the Writing Center at UNC will explain what literature reviews are and offer insights into the form and construction of literature reviews in the humanities, social sciences, and sciences.
• Purdue OWL: Writing a Literature Review The overview of literature reviews comes from Purdue's Online Writing Lab. It explains the basic why, what, and how of writing a literature review.

Organizational Tools for Literature Reviews

One of the most daunting aspects of writing a literature review is organizing your research. There are a variety of strategies that you can use to help you in this task. We've highlighted just a few ways writers keep track of all that information! You can use a combination of these tools or come up with your own organizational process. The key is choosing something that works with your own learning style.

Citation Managers

Citation managers are great tools, in general, for organizing research, but can be especially helpful when writing a literature review. You can keep all of your research in one place, take notes, and organize your materials into different folders or categories. Read more about citations managers here:

• Manage Citations & Sources

Concept Mapping

Some writers use concept mapping (sometimes called flow or bubble charts or "mind maps") to help them visualize the ways in which the research they found connects.

There is no right or wrong way to make a concept map. There are a variety of online tools that can help you create a concept map or you can simply put pen to paper. To read more about concept mapping, take a look at the following help guides:

• Using Concept Maps From Williams College's guide, Literature Review: A Self-guided Tutorial

Synthesis Matrix

A synthesis matrix is is a chart you can use to help you organize your research into thematic categories. By organizing your research into a matrix, like the examples below, can help you visualize the ways in which your sources connect. 

• Walden University Writing Center: Literature Review Matrix Find a variety of literature review matrix examples and templates from Walden University.
• Writing A Literature Review and Using a Synthesis Matrix An example synthesis matrix created by NC State University Writing and Speaking Tutorial Service Tutors. If you would like a copy of this synthesis matrix in a different format, like a Word document, please ask a librarian. CC-BY-SA 3.0
• << Previous: Case Study Design
• Next: Quantitative Research Methods >>
• Last Updated: May 7, 2024 9:51 AM

CityU Home - CityU Catalog

• University of Texas Libraries

Literature Reviews

• What is a literature review?
• Steps in the Literature Review Process
• Define your research question
• Determine inclusion and exclusion criteria
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support
• Artificial Intelligence (AI) Tools

What is a Literature Review?

A literature or narrative review is a comprehensive review and analysis of the published literature on a specific topic or research question. The literature that is reviewed contains: books, articles, academic articles, conference proceedings, association papers, and dissertations. It contains the most pertinent studies and points to important past and current research and practices. It provides background and context, and shows how your research will contribute to the field. 

A literature review should: 

• Provide a comprehensive and updated review of the literature;
• Explain why this review has taken place;
• Articulate a position or hypothesis;
• Acknowledge and account for conflicting and corroborating points of view

From  S age Research Methods

Purpose of a Literature Review

A literature review can be written as an introduction to a study to:

• Demonstrate how a study fills a gap in research
• Compare a study with other research that's been done

Or it can be a separate work (a research article on its own) which:

• Organizes or describes a topic
• Describes variables within a particular issue/problem

Limitations of a Literature Review

Some of the limitations of a literature review are:

• It's a snapshot in time. Unlike other reviews, this one has beginning, a middle and an end. There may be future developments that could make your work less relevant.
• It may be too focused. Some niche studies may miss the bigger picture.
• It can be difficult to be comprehensive. There is no way to make sure all the literature on a topic was considered.
• It is easy to be biased if you stick to top tier journals. There may be other places where people are publishing exemplary research. Look to open access publications and conferences to reflect a more inclusive collection. Also, make sure to include opposing views (and not just supporting evidence).

Source: Grant, Maria J., and Andrew Booth. “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies.” Health Information & Libraries Journal, vol. 26, no. 2, June 2009, pp. 91–108. Wiley Online Library, doi:10.1111/j.1471-1842.2009.00848.x.

Librarian Assistance

For help, please contact the librarian for your subject area.  We have a guide to library specialists by subject .

• Last Updated: Aug 26, 2024 5:59 AM
• URL: https://guides.lib.utexas.edu/literaturereviews

Research Methods: Literature Reviews

• Annotated Bibliographies
• Literature Reviews
• Scoping Reviews
• Systematic Reviews
• Scholarship of Teaching and Learning
• Persuasive Arguments
• Subject Specific Methodology

A literature review involves researching, reading, analyzing, evaluating, and summarizing scholarly literature (typically journals and articles) about a specific topic. The results of a literature review may be an entire report or article OR may be part of a article, thesis, dissertation, or grant proposal. A literature review helps the author learn about the history and nature of their topic, and identify research gaps and problems.

Steps & Elements

Problem formulation

• Determine your topic and its components by asking a question
• Research: locate literature related to your topic to identify the gap(s) that can be addressed
• Read: read the articles or other sources of information
• Analyze: assess the findings for relevancy
• Evaluating: determine how the article are relevant to your research and what are the key findings
• Synthesis: write about the key findings and how it is relevant to your research

Elements of a Literature Review

• Summarize subject, issue or theory under consideration, along with objectives of the review
• Divide works under review into categories (e.g. those in support of a particular position, those against, those offering alternative theories entirely)
• Explain how each work is similar to and how it varies from the others
• Conclude which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of an area of research

Writing a Literature Review Resources

• How to Write a Literature Review From the Wesleyan University Library
• Write a Literature Review From the University of California Santa Cruz Library. A Brief overview of a literature review, includes a list of stages for writing a lit review.
• Literature Reviews From the University of North Carolina Writing Center. Detailed information about writing a literature review.
• Undertaking a literature review: a step-by-step approach Cronin, P., Ryan, F., & Coughan, M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), p.38-43

Literature Review Tutorial

• << Previous: Annotated Bibliographies
• Next: Scoping Reviews >>
• Last Updated: Jul 8, 2024 3:13 PM
• URL: https://guides.auraria.edu/researchmethods

1100 Lawrence Street Denver, CO 80204 303-315-7700 Ask Us Directions

Which review is that? A guide to review types

• Which review is that?
• Review Comparison Chart
• Decision Tool
• Critical Review
• Integrative Review
• Narrative Review
• State of the Art Review
• Narrative Summary
• Systematic Review
• Meta-analysis
• Comparative Effectiveness Review
• Diagnostic Systematic Review
• Network Meta-analysis
• Prognostic Review
• Psychometric Review
• Review of Economic Evaluations
• Systematic Review of Epidemiology Studies
• Living Systematic Reviews
• Umbrella Review
• Review of Reviews
• Rapid Review
• Rapid Evidence Assessment
• Rapid Realist Review
• Qualitative Evidence Synthesis
• Qualitative Interpretive Meta-synthesis
• Qualitative Meta-synthesis
• Qualitative Research Synthesis
• Framework Synthesis - Best-fit Framework Synthesis
• Meta-aggregation
• Meta-ethnography
• Meta-interpretation
• Meta-narrative Review
• Meta-summary
• Thematic Synthesis
• Mixed Methods Synthesis
• Narrative Synthesis
• Bayesian Meta-analysis
• EPPI-Centre Review
• Critical Interpretive Synthesis
• Realist Synthesis - Realist Review
• Scoping Review
• Mapping Review
• Systematised Review
• Concept Synthesis
• Expert Opinion - Policy Review
• Technology Assessment Review

Methodological Review

• Systematic Search and Review

A methodological review is a type of systematic secondary research (i.e., research synthesis) which focuses on summarising the state-of-the-art methodological practices of research in a substantive field or topic" (Chong et al, 2021).

Methodological reviews "can be performed to examine any methodological issues relating to the design, conduct and review of research studies and also evidence syntheses". Munn et al, 2018)

Further Reading/Resources

Clarke, M., Oxman, A. D., Paulsen, E., Higgins, J. P. T., & Green, S. (2011). Appendix A: Guide to the contents of a Cochrane Methodology protocol and review. Cochrane Handbook for systematic reviews of interventions . Full Text PDF

Aguinis, H., Ramani, R. S., & Alabduljader, N. (2023). Best-Practice Recommendations for Producers, Evaluators, and Users of Methodological Literature Reviews. Organizational Research Methods, 26(1), 46-76. https://doi.org/10.1177/1094428120943281 Full Text

Jha, C. K., & Kolekar, M. H. (2021). Electrocardiogram data compression techniques for cardiac healthcare systems: A methodological review. IRBM . Full Text

References Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC medical research methodology , 18 (1), 1-9. Full Text Chong, S. W., & Reinders, H. (2021). A methodological review of qualitative research syntheses in CALL: The state-of-the-art. System , 103 , 102646. Full Text

• << Previous: Technology Assessment Review
• Next: Systematic Search and Review >>
• Last Updated: Aug 19, 2024 1:08 PM
• URL: https://unimelb.libguides.com/whichreview

• StudySkills@Sheffield
• Academic writing skills
• Critical writing

How to write a literature review

Are you writing a literature review as part of a final year project, dissertation, or thesis, or as a standalone piece of work? This page will work through a process of organising and synthesising your sources and then writing a clear and critical final review.

What is a literature review?

A literature review is an account of the current thinking in a specific area of study. Its purpose is to introduce the reader to what has gone before and often to provide you with a foundation that you can build on with your own research. This traditional form of review is sometimes also referred to as a narrative review.

A literature review will often form a section or chapter of a larger piece of research work, such as a dissertation, thesis, or final year project.  It can also be a standalone piece of work.  

A literature review will usually do some or all of the following:

• Introduce the reader to a specific area of interest.
• Organise relevant sources thematically, starting with the more general, broader themes and narrowing towards the most specific themes.
• Introduce key theories relevant to the area of study.
• Define your understanding of important terms or language used in the research.
• Include only the most relevant, important or influential sources, carefully selected. It is about quality not quantity!
• Identify gaps or limitations in existing research.

Considering a body of scholarship as a whole (or in relation to each of your themes) will allow you to 'synthesise' multiple sources and produce an overall summary.

Developing a literature review will help you to develop a level of expertise in your chosen area. By consulting and including a unique combination of sources, you will be able to formulate an informed and original perspective.  Where relevant, this can drive forward your ongoing research.

Writing a Literature Review workshop: book here

A systematic review is a research methodology, often following a standardised and replicable search method and reporting structure that is specific to your discipline. Visit our guidance on systematic reviews for more information.

Organising your sources

As you encounter more and more relevant sources, you will face an ever-expanding amount of reading for yourself. It would take years to read through all of the literature in a specific field from start to finish.

Academic reading, and particularly the process of 'reading around' a topic, is about selective, or targeted reading. Visit our Reading and understanding information Hub to explore approaches to reading for different purposes.

Creating a Literature Matrix can help you to identify the key things that you want to take away from each source. A literature matrix is a simple spreadsheet where you select column titles to suit the aims of your literature review. Are you interested in the research methodology, the scale of the research, the main conclusions, or something else entirely?

Once you have scanned through a source and pulled out the points you are interested in, you can move onto the next source. Organising your reading in this way will also allow you to identify key themes that are emerging in your reading, which you will be able to use later on to plan your review.

You may want to use a reference management tool to help organise and produce your bibliography. Visit the University of Sheffield Library Reference Management pages here .

Make a copy of our Literature matrix template (Google Sheet) and add/delete columns based on the information you want to collect during your search.  Using a spreadsheet means that you can filter and sort your sources, for example, into chronological order, or alphabetically by author.

This downloadable example literature matrix shows how you can lay out your columns.

Synthesising your sources

Once you have a number of sources to work with, you will start to identify key themes emerging. At this point you can start to organise your sources systematically to develop and explore those themes. Can you organise your themes from the broadest to the narrowest and most specific?

A synthesis matrix will help you to identify a thematic structure for your literature review and to understand how the sources that you have found relate to one another. A synthesis matrix is a further spreadsheet that organises your sources by theme and includes a synthesis column, where you can begin to draw out comparisons between the sources. 

Once you have identified a number of sources for each theme in your matrix, you should be able to identify the following:

• Do the sources build on or develop one another? This may be a chronological process.
• Do the sources challenge or contradict one another? Do they reveal a debate within the field?
• Do the sources identify an area of particular interest or a gap in the field?
• Do the sources help to fill in gaps or complete a bigger picture?

Your synthesis column provides an opportunity for you to comment on multiple sources considered as a whole. It is a space for your critical voice and interpretation, which is a key part of writing a successful literature review.

Make a copy of our synthesis matrix (Google Sheet) to organise your themes and plan how the relevant sources can be synthesised.

Download a completed example synthesis matrix from NC State University (PDF, 34Kb)

Visit our Producing a literature review interactive tutorial - for further guidance.

Writing your review

Once you have done the background reading and organised your sources using a synthesis matrix, the job of writing your review is simply about adding flesh to the bones. You will need to write your review as a narrative account, but you can use your matrix as a framework to help you do so.

A literature review will usually follow a simple structure:

• Introduction: what is the overall topic area and how have you broken your review down into themes?
• Theme 1: the broadest, most top-level area (perhaps including some background theory that may have influenced your thinking).
• Theme 2, theme 3, theme 4, etc. Your themes should get progressively more specific and closer to the focus of your research.
• Conclusion: how has this informed your thinking and (if the review is part of a bigger project) what are your research aims and objectives? 

Your review may be broken down by section headings or be a continuous flow with themes clearly separated in a paragraph structure. Each section or paragraph will describe that theme and finish by summarising your overview of a theme (the synthesis part of the matrix above, which includes your critical analysis). 

Our web page How to structure a paragrap h has further guidance to ensure your paragraphs are clear and contain your synthesis and critical analysis.

For advice and feedback on your own review, including referencing, synthesis and academic arguments, please book a writing advisory service appointment.

Make an appointment (student login required)

• How to plan an effective information search
• How to plan a dissertation or final year project
• How to write critically

Use your mySkills portfolio to discover your skillset, reflect on your development, and record your progress.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism. Run a free check.

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved September 10, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Service update: Some parts of the Library’s website will be down for maintenance on August 11.

Secondary menu

• Log in to your Library account
• Hours and Maps
• Connect from Off Campus
• UC Berkeley Home

Search form

Psychology 194: honors seminar: the literature review.

• Finding Articles
• The Literature Review
• Citations & Bibliographic Software (Zotero)
• Doing Original Research

Quick Links

• Google Scholar This link opens in a new window Search across many disciplines and sources including articles, theses, books, abstracts and court opinions, from academic publishers, professional societies, online repositories, universities and other web sites. more... less... Lists journal articles, books, preprints, and technical reports in many subject areas (though more specialized article databases may cover any given field more completely). Can be used with "Get it at UC" to access the full text of many articles.

• UC Library Search Start here to search for articles, books and more, in the collections of the University of California

What is a Literature Review?

A literature review is a survey of research on a given topic. It allows you see what has already been written on a topic so that you can draw on that research in your own study. By seeing what has already been written on a topic you will also know how to distinguish your research and engage in an original area of inquiry.

Why do a Literature Review?

A literature review helps you explore the research that has come before you, to see how your research question has (or has not) already been addressed.

You will identify:

• core research in the field
• experts in the subject area
• methodology you may want to use (or avoid)
• gaps in knowledge -- or where your research would fit in

Elements of a Successful Literature Review

According to Byrne's  What makes a successful literature review? you should follow these steps:

• Identify appropriate search terms.
• Search appropriate databases to identify articles on your topic.
• Identify key publications in your area.
• Search the web to identify relevant grey literature. (Grey literature is often found in the public sector and is not traditionally published like academic literature. It is often produced by research organizations.)
• Scan article abstracts and summaries before reading the piece in full.
• Read the relevant articles and take notes.
• Organize by theme.
• Write your review .

from Byrne, D. (2017). What makes a successful literature review?. Project Planner . 10.4135/9781526408518. (via SAGE Research Methods )

Research help

Email : Email your research questions to the Library.

Appointments : Schedule a 30-minute research meeting with a librarian. 

Find a subject librarian : Find a library expert in your specific field of study.

Research guides on your topic : Learn more about resources for your topic or subject.

• << Previous: Finding Articles
• Next: Citations & Bibliographic Software (Zotero) >>
• Last Updated: Sep 13, 2024 7:17 PM
• URL: https://guides.lib.berkeley.edu/Psych194-HonorsSeminar

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Usability Evaluation Methods Used in Electronic Discharge Summaries: Literature Review

Affiliations.

• 1 The University of Sydney School of Pharmacy, Sydney, Australia.
• 2 The University of Queensland School of Pharmacy, Brisbane, Australia.
• 3 The University of Sydney School of Medicine, Sydney, Australia.
• 4 Nepean Kidney Research Centre, Nepean Hospital, Sydney, Australia.
• 5 Australian Commission on Safety and Quality in Health Care, Sydney, Australia.
• PMID: 39264712
• DOI: 10.2196/55247

Background: With the widespread adoption of digital health records, including electronic discharge summaries (eDS), it is important to assess their usability in order to understand whether they meet the needs of the end users. While there are established approaches for evaluating the usability of electronic health records, there is a lack of knowledge regarding suitable evaluation methods specifically for eDS.

Objective: This literature review aims to identify the usability evaluation approaches used in eDS.

Methods: We conducted a comprehensive search of PubMed, CINAHL, Web of Science, ACM Digital Library, MEDLINE, and ProQuest databases from their inception until July 2023. The study information was extracted and reported in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). We included studies that assessed the usability of eDS, and the systems used to display eDS.

Results: A total of 12 records, including 11 studies and 1 thesis, met the inclusion criteria. The included studies used qualitative, quantitative, or mixed methods approaches and reported the use of various usability evaluation methods. Heuristic evaluation was the most used method to assess the usability of eDS systems (n=7), followed by the think-aloud approach (n=5) and laboratory testing (n=3). These methods were used either individually or in combination with usability questionnaires (n=3) and qualitative semistructured interviews (n=4) for evaluating eDS usability issues. The evaluation processes incorporated usability metrics such as user performance, satisfaction, efficiency, and impact rating.

Conclusions: There are a limited number of studies focusing on usability evaluations of eDS. The identified studies used expert-based and user-centered approaches, which can be used either individually or in combination to identify usability issues. However, further research is needed to determine the most appropriate evaluation method which can assess the fitness for purpose of discharge summaries.

Keywords: adoption; digital health; discharge summary; electronic; electronic discharge summaries; end users; evaluation; heuristic evaluation; heuristics, think-aloud; usability; usability testing; user-centered.

©Wubshet Tesfaye, Margaret Jordan, Timothy F Chen, Ronald Lynel Castelino, Kamal Sud, Racha Dabliz, Parisa Aslani. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 12.09.2024.

PubMed Disclaimer

Publication types

• Search in MeSH

LinkOut - more resources

Full text sources.

• JMIR Publications

Miscellaneous

• NCI CPTAC Assay Portal
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Understanding the wellbeing needs of first nations children in out-of-home care in australia: a comprehensive literature review.

1. Introduction

2. materials and methods, 2.1. research team, 2.2. search strategy, 2.3. eligibility criteria, 2.4. study selection, 2.5. data collection and analysis, 3.1. paper characteristics, 3.2. qualitative results, 3.3. being seen, being heard, 3.3.1. having autonomy and agency.

Talk to the actual kids, they’ll be very honest about their placement they don’t hide nothing get new workers to build trust with them kids and they will just spill their guts really . (Sally, worker) [ 47 ]

I think it’s like ya get dropped out of the loop… But basically we all are feeling so left out of things, these things are happening, we might be young but some sort of explanation would just go a mile… (Caroline, post-care, 19, Aboriginal) [ 33 ]

In some cases, while FACS [Family and Community Services] involved the children in some decision-making, the reviewer identified that this consultation was not ongoing. Children have the right to be involved in decisions that affect them and impact their lives, and this failure to consult on an ongoing basis was identified as disempowering practice . (Case file reviewer) [ 38 ]

3.3.2. Being Treated Like a Child, Not a Number

It is so important to know the kids you are working with; each person is an individual. It is just bloody critical that these kids are seen, known and not just a number. This is the work I am so passionate about . (Child Protection staff member) [ 33 ]

They should listen to young people. Hear what they have to say to ask them what they think and if they are safe . (15-year-old First Nations girl in care) [ 52 ]

I was lucky ’cos my grandparents made me feel like I was part of the family. I was never, ever introduced as their foster child and that made me feel loved and appreciated. I would encourage foster carers to try and do that, that was the most important thing . (Caden, post-care, 19, Aboriginal) [ 33 ]

My foster carers…were Aboriginal. They taught me stuff about culture. They helped me keep in contact with family. I stuck with one for most of it… They listened to me… [Carer] was really understanding. He understood why I was misbehaving sometimes . (Phoebe, returned home, 16, Aboriginal) [ 33 ]

3.4. A Sense of Stability

3.4.1. experiencing placement stability.

Stability for Aboriginal people is grounded in their sense of identity in connection to family, kin, culture and country. In our view, permanent care/adoption potentially places an emphasis on achieving stability of living arrangements and a secure legal status potentially at the cost of the child’s identity and enduring relationships with their extended family and connection with community and culture . (Victorian Aboriginal Child Care Agency) [ 34 ]

3.4.2. Receiving Support in School

I always tell them [First Nations children in OOHC engaging in education] who I am and what I’m there to do. And then I ask them if they want to. Because I make sure that they are involved in the decision making of being involved. And I did have two kids go, ‘No. I’m not quite sure’. But then in the end, became involved. I think part of it is also listening to them about what they want . (Education engagement intervention program teacher/mentor) [ 62 ]

3.4.3. Being on a Pathway to Culturally Appropriate Permanency

SNAICC submitted that permanency for Aboriginal children was ‘tied to existing identity, kinship relationships, and connections to culture and country’, and that it was important not to permanently deprive children of these connections through the application of ‘inflexible permanency planning measures . (SNAICC) [ 38 ]

3.5. Holistic Health Support

3.5.1. fulfilment of basic needs.

Most of us kids, the reason why we are in care is because our families are not reliable. You know, money problems, food, clothes, safety problems… The whole reason why they took us off our family was because we feel unsafe, we don’t feel much protected, there’s no food, and we’re not getting clothes… we’re not getting anything. But what’s the point of that if they do exactly the same in all these houses. It’s not better either way: living with our family, living with DCP [Department for Child Protection], government homes… or living on the streets… it’s not good anywhere . (17-year-old Aboriginal male, residential care) [ 35 ]

3.5.2. Receiving Care for Health and Physical Wellbeing

I think another trend that we found is that we’ve got a number of young people who have gone through the care system to be diagnosed as foetal alcohol syndrome at 18. And they’ve already been in and out of detention and they’ve got involvement with the justice system, and now they’re 18, it’s the adult justice system, which is a real concern. One young fella in particular I’m thinking of, was actually in residential care and wasn’t diagnosed until he was 18 . (Western Australian NGO) [ 49 ]

3.5.3. Provision of Trauma-Informed Care

FACS fails to acknowledge that the removal of Aboriginal children from their families often exposes them to danger and ‘immense trauma’, as opposed to ‘protection’, (National Congress of Australia’s First Peoples) and that FACS intervention in and of itself is an extremely arduous, traumatic process that is actively harmful to all involved, particularly children . (Grandmothers Against Removals New South Wales) [ 38 ]

A lot of kids have had severe trauma, been too exposed to a lot of negative experiences, and you can see it, like behavioural change. A lot of the kids are getting suspended all the time, they’re acting out, they just show all the different traits, like physically, emotionally. You can see, spiritually, that they’re impacted too, on a lot of different levels. Their confidence is low, self-esteem, yeah, just a lot of different things . (NSW ACCO) [ 49 ]

While in placement, with the support of a strong and therapeutic care team, an appropriate cultural support plan and a KESO [Koorie Engagement Support Officer], Molly’s [Aboriginal girl in OOHC] behaviours have settled. Molly has told child protection she feels safe and secure with her carers . (caseworker/reviewer) [ 42 ]

3.6. Social and Cultural Connections

3.6.1. fostering interconnected relationships.

The major difficulty in the urban setting was appropriately placing children culturally, working out where they belonged . (ACCO staff) [ 65 ]

3.6.2. Maintaining Cultural Knowledge and Identity

Being Aboriginal is the proudest thing in my life, to know that that’s my people. It made me so proud to see what we’ve actually done and how far we’ve come to this day. It taught me that no matter what, I can still get up and do what I want . (Aboriginal child in OOHC) [ 36 ]

They [First Nations children and youth upon entering cultural camps] didn’t know their connections to communities, didn’t know about the language, didn’t eat Aboriginal food, they knew nothing at all [of their culture] . (Aboriginal education officer) [ 36 ]

3.6.3. Feeling Connected to Community and Country

Aboriginal children coming into care should be placed in their own country. Just because they’re Aboriginal, isn’t good enough. You need to be placed with people who know your identity . (Non-First Nations carer) [ 54 ]

Participants identified a strong cultural identity and effective connection with community as a powerful source of resilience for Indigenous young people during and post transition from care . (ACCO and Government OOHC workers) [ 50 ]

3.6.4. Continued Links to Family and Kin

Give Aboriginal kids back to their home, their family, after you’ve gone through and made sure everything is all safe and all good. If not the mother and father, then maybe the kid has sisters, aunties, or an Aboriginal carer is available . (Aboriginal caregiver) [ 36 ]

Living there [in kinship care] feels like a family . (Shane, kinship care, 15, Aboriginal) [ 33 ]

Well, the strength [of kinship care] is that children remain within their extended family, which supports our philosophy around self-determination, self-management. The family best knows the family circumstances . (Jenny, worker) [ 37 ]

His [First Nations, 8 year old boy in relative care] older siblings were scattered geographically but it was clear from his narrative that he wanted regular contact with his older siblings . (OOHC team leader and art therapist) [ 52 ]

If I need to talk to someone now, my brother would be the first person I would talk to . (Ellie, residential care, 16, Aboriginal) [ 33 ]

I had someone sit down with me and go through everything, my mob, my family. There is nothing else I need to know . (Female, First Nations, 17 years) [ 48 ]

I want to find out if I have a cultural support plan so I can get help finding more info about my culture and where my family was from . (Female, First Nations, 14 years) [ 48 ]

Few months ago I asked [Department of Health and Human Services, Victoria] if I could find my dad. Haven’t seen him since I was one. Part of my life I’ve never met, so not good. My dad is the only actual family I know . (Evan, foster care, 15, Aboriginal) [ 33 ]

3.6.5. Being Supported by Friends

Q: Who do you go to for support? My friends, but more like my best friends. I’ve known them since I was like three and we’ve always stayed in contact and if I have a problem on my mind, I can always just go to his house . (Ethan, kinship care, 15, Aboriginal) [ 33 ]

3.7. Culturally Safe OOHC Providers

3.7.1. supported by oohc organizations trusted by first nations peoples.

We understand where people [Aboriginal families] come from you can’t just have a mainstream organisation culturally competent, its philosophy is driven by white people, how they were raised, how they understand programs and services . (ACCO staff) [ 47 ]

3.7.2. Provision of Support Services Grounded in Culturally Safe Approaches

Aboriginal community-controlled agencies are best placed to support Aboriginal children and young people in OOHC, including maintaining their connection to family, community, culture and Country that is central to identity development and wellbeing . (New South Wales Council of Social Service) [ 38 ]

3.8. Preparedness for Transitioning Out of Care

3.8.1. given adequate opportunities for reunification with family.

He [14-year-old Aboriginal/South Sea Islander boy in OOHC] showed little attachment to the carer in that he talked of running away and not needing anyone . (OOHC team leader and art therapist) [ 52 ]

We’ve got lots of kids walking from care and leaving at 15. And particularly going back to Country or trying to find Country . (New South Wales NGO) [ 49 ]

3.8.2. Provided with Life Skills for after Care

We know there are 16, 17, 18-year-olds out there that can’t even boil water, you know, yet they want to fall pregnant; so if you can get it in there early enough to get these old people to teach these children survival skills, and not just Indigenous (skills), but also how to cook a meal and sew a button on . (Carer) [ 39 ]

4. Discussion

Strengths and limitations, 5. conclusions, supplementary materials, author contributions, data availability statement, acknowledgments, conflicts of interest.

• Commonwealth of Australia. An Outline of National Standards for Out-of-Home Care ; Department of Families, Housing, Community Services and Indigenous Affairs, National Framework Implementation Working Group: Canberra, Australia, 2011. [ Google Scholar ]
• Australian Institute of Health and Welfare. Child protection Australia 2020–2021. Available online: https://www.aihw.gov.au/reports/child-protection/child-protection-australia-2020-21 (accessed on 30 September 2022).
• Liddle, C.; Gray, P.; Burton, J.; Prideaux, C.; Solomon, N.; Cackett, J.; Jones, M.; Bhathal, A.; Corrales, T.; Parolini, A.; et al. The Family Matters Report 2021 ; SNAICC: Melbourne, Australia, 2021. [ Google Scholar ]
• Productivity Commission. Closing the Gap Annual Data Compilation Report July 2023 ; Productivity Commission: Canberra, Australia, 2023. [ Google Scholar ]
• Australian Government Productivity Commission. Report on Government Services Part F, Section 16: Child Protection Services 2024 ; Australian Government, Productivity Commission: Melbourne, Australia, 2024. [ Google Scholar ]
• Commonwealth of Australia. Aboriginal and Torres Strait Islander First Action Plan 2023–2026 under Safe and Supported: The National Framework for Protecting Australia’s Children 2021–2031 ; Commonwealth of Australia: Canberra, Australia, 2022. [ Google Scholar ]
• Kickett, G.; Stubley, K. Moving to Child Safety and Family and Community Wellbeing ; Centre for Social Impact (University of Western Australia) Department of Communities Western Australia: Nedlands, Australia, 2020. [ Google Scholar ]
• Angell, B.; Muhunthan, J.; Eades, A.-M.; Cunningham, J.; Garvey, G.; Cass, A.; Howard, K.; Ratcliffe, J.; Eades, S.; Jan, S. The health-related quality of life of Indigenous populations: A global systematic review. Qual. Life Res. 2016 , 25 , 2161–2178. [ Google Scholar ] [ CrossRef ]
• Butler, T.L.; Anderson, K.; Garvey, G.; Cunningham, J.; Ratcliffe, J.; Tong, A.; Whop, L.J.; Cass, A.; Dickson, M.; Howard, K. Aboriginal and Torres Strait Islander people’s domains of wellbeing: A comprehensive literature review. Soc. Sci. Med. 2019 , 233 , 138–157. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Garvey, G.; Anderson, K.; Gall, A.; Butler, T.L.; Cunningham, J.; Whop, L.J.; Dickson, M.; Ratcliffe, J.; Cass, A.; Tong, A.; et al. What Matters 2 Adults (WM2Adults): Understanding the Foundations of Aboriginal and Torres Strait Islander Wellbeing. Int. J. Environ. Res. Public Health 2021 , 18 , 6193. [ Google Scholar ] [ CrossRef ]
• Garvey, G.; Anderson, K.; Gall, A.; Butler, T.L.; Whop, L.J.; Arley, B.; Cunningham, J.; Dickson, M.; Cass, A.; Ratcliffe, J.; et al. The Fabric of Aboriginal and Torres Strait Islander Wellbeing: A Conceptual Model. Int. J. Environ. Res. Public Health 2021 , 18 , 7745. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Anderson, K.; Elder-Robinson, E.; Gall, A.; Ngampromwongse, K.; Connolly, M.; Letendre, A.; Willing, E.; Akuhata-Huntington, Z.; Howard, K.; Dickson, M.; et al. Aspects of Wellbeing for Indigenous Youth in CANZUS Countries: A Systematic Review. Int. J. Environ. Res. Public Health 2022 , 19 , 13688. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gall, A.; Anderson, K.; Howard, K.; Diaz, A.; King, A.; Willing, E.; Connolly, M.; Lindsay, D.; Garvey, G. Wellbeing of Indigenous Peoples in Canada, Aotearoa (New Zealand) and the United States: A Systematic Review. Int. J. Environ. Res. Public Health 2021 , 18 , 5832. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Dudgeon, P.; Bray, A.; Smallwood, G.; Walker, R.; Dalton, T. Wellbeing and Healing through Connection and Culture ; Lifeline Australia: Sydney, Australia, 2020. [ Google Scholar ] [ CrossRef ]
• United Nations (General Assembly). Convention on the Rights of the Child ; United Nations: New York, NY, USA, 1989. [ Google Scholar ]
• United Nations (General Assembly). Declaration on the Rights of Indigenous People ; United Nations: New York, NY, USA, 2007. [ Google Scholar ]
• Jackomos, A. Being strong in aboriginal identity and culture protects young people leaving care. Parity 2016 , 29 , 14–15. [ Google Scholar ]
• Burton, J.; Young, J.; Jayakody, N.; Ruggiero, E.; Thwaites, R. The Aboriginal and Torres Strait Islander Child Placement Principle: A Guide to Support Implementation ; SNAICC: Melbourne, Australia, 2019. [ Google Scholar ]
• Gatwiri, K.; McPherson, L.; Parmenter, N.; Cameron, N.; Rotumah, D. Indigenous Children and Young People in Residential Care: A Systematic Scoping Review. Trauma Violence Abus. 2019 , 22 , 829–842. [ Google Scholar ] [ CrossRef ]
• Richardson, N.; Bromfield, L. Cultural Considerations in Out-of-Home Care ; Australian Institute of Family Studies: Melbourne, Australia, 2007; Volume 8. [ Google Scholar ]
• Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021 , 372 , n71. [ Google Scholar ] [ CrossRef ]
• Gharabaghi, K.; Anderson-Nathe, B. Strength-based research in a deficits-oriented context. Child Youth Serv. 2017 , 38 , 177–179. [ Google Scholar ] [ CrossRef ]
• Forsman, H.; Vinnerljung, B. Interventions aiming to improve school achievements of children in out-of-home care: A scoping review. Child. Youth Serv. Rev. 2012 , 34 , 1084–1091. [ Google Scholar ] [ CrossRef ]
• Kääriälä, A.; Hiilamo, H. Children in out-of-home care as young adults: A systematic review of outcomes in the Nordic countries. Child. Youth Serv. Rev. 2017 , 79 , 107–114. [ Google Scholar ] [ CrossRef ]
• Sinha, V.; Caldwell, J.; Paul, L.; Fumaneri, P.R. A Review of Literature on the Involvement of Children from Indigenous Communities in Anglo Child Welfare Systems: 1973–2018. Int. Indig. Policy J. 2021 , 12 , 1–43. [ Google Scholar ] [ CrossRef ]
• The EndNote Team. EndNote, EndNote 20 ; Clarivate: Philadelphia, PA, USA, 2013. [ Google Scholar ]
• Veritas Health Information. Covidence Systematic Review Software. Available online: https://www.covidence.org/ (accessed on 9 September 2022).
• Microsoft Corporation. Microsoft Excel. Available online: https://office.microsoft.com/excel (accessed on 9 September 2022).
• Lumivero. NVivo (Version 13) [Computer Software]. Available online: https://lumivero.com/products/nvivo/ (accessed on 24 October 2022).
• Braun, V.; Clarke, V. Reflecting on reflexive thematic analysis. Qual. Res. Sport. Exerc. Health 2019 , 11 , 589–597. [ Google Scholar ] [ CrossRef ]
• Byrne, D. A worked example of Braun and Clarke’s approach to reflexive thematic analysis. Qual. Quant. 2022 , 56 , 1391–1412. [ Google Scholar ] [ CrossRef ]
• Naeem, M.; Ozuem, W.; Howell, K.; Ranfagni, S. A Step-by-Step Process of Thematic Analysis to Develop a Conceptual Model in Qualitative Research. Int. J. Qual. Methods 2023 , 22 , 16094069231205789. [ Google Scholar ] [ CrossRef ]
• Commission for Children and Young People. ‘In Our Own Words’: Systemic Inquiry into the Lived Experience of Children and Young People in the Victorian Out-of-Home Care System ; Commission for Children and Young People: Melbourne, Australia, 2019. [ Google Scholar ]
• Turnbull-Roberts, V.; Salter, M.; Newton, B.J. Trauma then and now: Implications of adoption reform for first nations children. Child Fam. Soc. Work 2021 , 27 , 163–172. [ Google Scholar ] [ CrossRef ]
• Commissioner for Children and Young People WA. Speaking out about Raising Concerns in Care ; Commissioner for Children and Young People WA: Perth, Australia, 2016. [ Google Scholar ]
• AbSec—NSW Child Family and Community Peak Aboriginal Corporation. Hearing the Voices of Aboriginal People in Child Welfare ; Absec—NSW Child, Family and Community Peak Aboriginal Corporation: Sydney, Australia, 2020. [ Google Scholar ]
• Kiraly, M.; James, J.; Humphreys, C. “It”s a family responsibility’: Family and cultural connection for Aboriginal children in kinship care. Child. Aust. 2014 , 40 , 23–32. [ Google Scholar ] [ CrossRef ]
• Davis, M. Family Is Culture: Independent Review into Aboriginal Out-of-Home Care in NSW ; New South Wales Department of Family and Community Services: Sydney, Australia, 2019. [ Google Scholar ]
• McMahon, A.; Reck, L.; Walker, M. Defining well-being for Indigenous children in care. Child. Aust. 2007 , 32 , 15–20. [ Google Scholar ] [ CrossRef ]
• Advocate for Children Young People (NSW). The Voices of Children and Young People in Out-of-Home Care 2021 ; Advocate for Children and Young People (NSW): Sydney, Australia, 2022. [ Google Scholar ]
• Bamblett, M.; Frederico, M.; Harrison, J.; Jackson, A.; Lewis, P. ’Not One Size Fits All’ Understanding the Social & Emotional Wellbeing of Aboriginal Children ; La Trobe University: Bundoora, Australia, 2012. [ Google Scholar ]
• Commission for Children and Young People. ‘Always Was Always Will Be Koori Children’: Systemic Inquiry into Services Provided to Aboriginal Children and Young People in Out-of-Home Care in Victoria ; Commission for Children and Young People: Melbourne, Australia, 2016. [ Google Scholar ]
• Hermeston, W.; McDougall, J.; Burton, J.; Smith, F.; Sydenham, E. Achieving Stability for Aboriginal and Torres Strait Islander Children in Out-of-Home Care ; SNAICC: Melbourne, Australia, 2016. [ Google Scholar ]
• Higgins, D.J.; Bromfield, L.M.; Higgins, J.R.; Richardson, N. Protecting Indigenous children: Views of carers and young people on “out-of-home care”. Fam. Matters 2006 , 75 , 42–49. [ Google Scholar ]
• Kiraly, M.; Humphreys, C. ‘It Is the Story of All Us’. Learning from Aboriginal Communities about Supporting Family Connection ; State of Victoria: Melbourne, Australia, 2011. [ Google Scholar ]
• Lewis, N.; Weston, R.; Burton, J.; Young, J.; Jayakody, N.; Mastroianni, A.; Tan, W.W.; Parolini, A.; Shlonsky, A.; Tilbury, C. The Family Matters Report 2019 ; SNAICC: Melbourne, Australia, 2020. [ Google Scholar ]
• Libesman, T. Cultural Care for Aboriginal and Torres Strait Islander Children in Out of Home Care ; SNAICC: Melbourne, Australia, 2011. [ Google Scholar ]
• McDowall, J.J. Out-of-Home Care in Australia: Children and Young People’s Views after Five Years of National Standards ; CREATE Foundation: Sydney, Australia, 2018. [ Google Scholar ]
• Mendes, P.; Standfield, R.; Saunders, B.; McCurdy, S.; Walsh, J.; Turnbull, L. Indigenous youth transitioning from out-of-home care in Australia: A study of key challenges and effective practice responses. J. Child. Serv. 2022 , 17 , 16–32. [ Google Scholar ] [ CrossRef ]
• Mendes, P.; Standfield, R.; Saunders, B.; McCurdy, S.; Walsh, J.; Turnbull, L. Aboriginal and Torres Strait Islander (Indigenous) young people leaving out-of-home care in Australia: A national scoping study. Child. Youth Serv. Rev. 2021 , 121 , 105848. [ Google Scholar ] [ CrossRef ]
• Moore, T.; Bennett, B.; McArthur, M. They’ve Gotta Listen: Aboriginal and Torres Strait Islander Young People in Out of Home Care ; 192123914X; Australian Catholic University: Dickson, Australia, 2007. [ Google Scholar ]
• Moss, M. Broken circles to a different identity: An exploration of identity for children in out-of-home care in Queensland, Australia. Child. Fam. Soc. Work. 2009 , 14 , 311–321. [ Google Scholar ] [ CrossRef ]
• Spence, N. Kinship Care in Australia. Child Abus. Rev. 2004 , 13 , 263–276. [ Google Scholar ] [ CrossRef ]
• Higgins, D.; Bromfield, L.; Richardson, N.; National Child Protection Clearinghouse. Enhancing Out-of-Home Care for Aboriginal and Torres Strait Islander Young People ; Australian Institute of Family Studies: Melbourne, Australia, 2005. [ Google Scholar ]
• Liddle, C.; Gray, P.; Burton, J.; Taylor, M.; Young, G.; Kumar, R.; Turner, L.; Hutchins, L.; French, B.; Jones, L.A.; et al. The Family Matters Report 2023 ; SNAICC: Melbourne, Australia, 2023. [ Google Scholar ]
• Black, C.; Frederico, M.; Bamblett, M. ‘Healing through culture’: Aboriginal young people’s experiences of social and emotional wellbeing impacts of cultural strengthening programs. Child Abus. Negl. 2023 , 148 , 106206. [ Google Scholar ] [ CrossRef ]
• Creamer, S.; Blair, S.; Toombs, M.; Brolan, C.E. Indigenous services leading the way for Aboriginal and Torres Strait Islander children in out-of-home care. Soc. Work Educ. 2024 , 43 , 913–937. [ Google Scholar ] [ CrossRef ]
• Jau, J.; Mendes, P.; Chavulak, J.; Martin, R. The Housing Pathways and Experiences of Aboriginal and Torres Strait Islander Youth as They Transition from out of Home Care in Victoria and Western Australia. Int. J. Child Maltreatment 2022 , 5 , 319–336. [ Google Scholar ] [ CrossRef ]
• Krakouer, J. Journeys of culturally connecting: Aboriginal young people’s experiences of cultural connection in and beyond out-of-home care. Child Fam. Soc. Work 2023 , 28 , 822–832. [ Google Scholar ] [ CrossRef ]
• Liddle, C.; Gray, P.; Burton, J.; Kumar, R.; Tunny, T.; Prideaux, C.; Armstrong, E.; McCracken, A.; French, B.; Attenborough, I.; et al. The Family Matters Report 2022 ; SNAICC: Melbourne, Australia, 2022. [ Google Scholar ]
• Newton, B.J.; Katz, I.; Gray, P.; Frost, S.; Gelaw, Y.; Hu, N.; Lingam, R.; Stephensen, J. Restoration from out-of-home care for Aboriginal children: Evidence from the pathways of care longitudinal study and experiences of parents and children. Child Abus. Negl. 2023 , 149 , 106058. [ Google Scholar ] [ CrossRef ]
• Roche, S.; Dunk-West, P.; Otarra, C.; Taylor, R.; Moss, M. Exploring strategies for re-engaging children and young people in learning while living in out-of-home care in the Northern Territory, Australia. Child Fam. Soc. Work 2023 , 28 , 1022–1034. [ Google Scholar ] [ CrossRef ]
• Clarke, A.; Healy, K.; Lynch, D.; Featherstone, G. Stability in statutory kinship care: A grounded theory study of placement stability in Australia. Child Youth Serv. Rev. 2024 , 156 , 107289. [ Google Scholar ] [ CrossRef ]
• Wright, A.C.; Collings, S. Conceptual meanings of permanency: Photovoice with care-experienced youth. J. Youth Stud. 2023 , 1–18. [ Google Scholar ] [ CrossRef ]
• Raman, S.; Ruston, S.; Irwin, S.; Tran, P.; Hotton, P.; Thorne, S. Taking culture seriously: Can we improve the developmental health and well-being of Australian aboriginal children in out-of-home care? Child Care Health Dev. 2017 , 43 , 899–905. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Kor, K.; Park, J.; Dear, R.; Fabrianesi, B. Responding to children’s ambiguous loss in out-of-home care: The HEAR practice model. Child Fam. Soc. Work 2024 , 29 , 248–258. [ Google Scholar ] [ CrossRef ]
• Bamblett, M.; Lewis, P. Detoxifying the Child and Family Welfare System for Australian Indigenous Peoples: Self-determination, Rights and Culture as the Critical Tools. First Peoples Child Fam. Rev. 2007 , 3 , 43–56. [ Google Scholar ] [ CrossRef ]
• Hassall, A.; Olsen, A.; Bourke, S.; Pasalich, D.S. How do kinship and foster caregivers differ in their conceptualization of family connectedness? Child Abus. Negl. 2023 , 145 , 106391. [ Google Scholar ] [ CrossRef ]
• Chamberlain, C.; Gray, P.; Bennet, D.; Elliott, A.; Jackomos, M.; Krakouer, J.; Marriott, R.; O’Dea, B.; Andrews, J.; Andrews, S.; et al. Supporting Aboriginal and Torres Strait Islander Families to Stay Together from the Start (SAFeST Start): Urgent call to action to address crisis in infant removals. Aust. J. Soc. Issues 2022 , 57 , 252–273. [ Google Scholar ] [ CrossRef ]
• Fernandez, E.; Lee, J.-S.; McNamara, P. Understanding the Experience and Outcomes of Aboriginal and Torres Strait Islander Children in Out-of-Home Care during the Twentieth Century ; University of New South Wales: Kensington, Australia, 2018. [ Google Scholar ]
• Mendes, P. “The most significant child welfare reform in a generation”: An examination of the strategies used by the Home Stretch campaign. Aust. J. Soc. Issues 2024 , 59 , 328–343. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Garvey, D.; Carter, K.; Anderson, K.; Gall, A.; Howard, K.; Venables, J.; Healy, K.; Bill, L.; Letendre, A.; Dickson, M.; et al. Understanding the Wellbeing Needs of First Nations Children in Out-of-Home Care in Australia: A Comprehensive Literature Review. Int. J. Environ. Res. Public Health 2024 , 21 , 1208. https://doi.org/10.3390/ijerph21091208

Garvey D, Carter K, Anderson K, Gall A, Howard K, Venables J, Healy K, Bill L, Letendre A, Dickson M, et al. Understanding the Wellbeing Needs of First Nations Children in Out-of-Home Care in Australia: A Comprehensive Literature Review. International Journal of Environmental Research and Public Health . 2024; 21(9):1208. https://doi.org/10.3390/ijerph21091208

Garvey, Darren, Ken Carter, Kate Anderson, Alana Gall, Kirsten Howard, Jemma Venables, Karen Healy, Lea Bill, Angeline Letendre, Michelle Dickson, and et al. 2024. "Understanding the Wellbeing Needs of First Nations Children in Out-of-Home Care in Australia: A Comprehensive Literature Review" International Journal of Environmental Research and Public Health 21, no. 9: 1208. https://doi.org/10.3390/ijerph21091208

Article Metrics

Article access statistics, supplementary material.

ZIP-Document (ZIP, 138 KiB)

Further Information

Mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

What is non-formal learning (and how do we know it when we see it)? A pilot study report

• Open access
• Published: 12 September 2024
• Volume 3 , article number  148 , ( 2024 )

Cite this article

You have full access to this open access article

• Martin Johnson   ORCID: orcid.org/0000-0001-8035-4754 1 &
• Dominika Majewska   ORCID: orcid.org/0000-0002-0374-3737 1  

In this paper we consider the issue of formality in learning. Both formal and informal learning appear to be well-defined in educational discourse, in contrast with non-formal learning which is less clearly articulated. This lack of clarity around non-formal learning has consequences for its recognition by teachers and learners. To contribute to the conceptualisation of non-formal learning we carried out a literature review that helped us to identify dimensions of learning formality. This framework allowed us to discriminate the ways in which non-formal learning differed or matched with aspects of formal or informal learning. It also suggested to us that formality is fluid, and that learning with different formalities might co-exist in some learning environments. We then used our framework to develop tools for capturing evidence of non-formal learning. We describe these tools and how we used them in a pilot study to explore the features of non-formal learning in a formal, Year 12 classroom learning context in an English school. We use our pilot study outcomes to reflect on the utility of our data gathering methods. We also use the study outcomes to update our non-formal learning framework.

Similar content being viewed by others

Non-formale und informelle Bildung

Introduction: Learning Theory and Its Relationship to Research in Informal Science Learning Environments

Explore related subjects.

• Artificial Intelligence

Avoid common mistakes on your manuscript.

1 Introduction

Sociocultural learning theories suggest that interaction transforms a learner’s thinking [ 1 ], with learning being experiential and social [ 2 ]. At the same time, the conditions of learning may differ across contexts. For example, Rogoff et al. [ 3 ] observe that learning in indigenous contexts relies more on learners’ intent participation in community activities where they employ listening-in and observation processes. This contrasts with westernised concepts of ‘assembly-line instruction’, where learning is generally decontextualised and more passive.

These cases show that learning can be organised in many ways [ 4 ], with some of these differences being captured through the language of ‘formal, non-formal, and informal’ learning. Whilst there has been extensive research exploring informal (or everyday) learning and how it differs from formal learning (e.g., see [ 1 ]), there is a relative lack of clarity around the meaning of non-formal learning. For some, this category encompasses informal learning (e.g., see [ 5 ]), and for others it forms a distinct category of learning organisation (e.g., see [ 6 ]).

According to Schugurensky [ 2 ], it is only once the conceptual challenge of defining learning is complete that the research and recognition challenges of gathering systematic information about it are possible. Defining and identifying non-formal learning appears highly pertinent at the present time since some education systems are exploring ways of organising learning where students can exercise greater control over their learning. For example, in South Australia there are proposals to develop an Activating Identities and Futures initiative which recognises students’ self-directed learning [ 7 ]. This initiative parallels suggestions for a Scottish Diploma of Achievement that offers students a range of opportunities to develop skills in areas of interest and significance to them [ 8 ].

In this paper we briefly outline a review of research literature that we carried out to construct a picture of non-formal learning. We then describe a non-formal learning data gathering approach that we developed based on our analysis of the research literature. Through this data collection process, we wanted to explore the interaction between learning processes and learning environments. For example, it could let us explore how formal learning environments include aspects of non-formal learning and open up opportunities for it to be recognised and respected alongside more formal learning outcomes. We conclude our paper by describing and critically evaluating a small-scale, pilot data collection exercise where we used the data gathering approach in a school.

2 Literature review

In 2022 we carried out a literature review to explore the various ways that formal, non-formal, and informal learning were conceptualised [ 9 ]. Our review had two aims. We were interested in finding out how formal, non-formal, and informal learning were defined, and how non-formal learning is researched and recognised.

Our review process had three steps: literature source identification, search criteria definition, and document coding. We searched six research databases that covered a broad range of published, peer-reviewed, research literature. Footnote 1 We searched for non-formality Footnote 2 in conjunction with ‘curriculum’, ‘learning’, or ‘education’ in documents that fell within the educational science and research field. We also stipulated that documents were to be published between 2002 and 2022 (although we ignored this if a paper appeared to be particularly important, for instance, if it provided definitions of non-formal learning that were frequently cited by others or that were very insightful). Our search identified 134 documents.

We developed and used a coding framework that could separate out and capture definitions for formal, non-formal, and informal learning concepts. Footnote 3 Document coding was carried out separately by two researchers using MAXQDA 2022 software [ 11 ]. These analyses were combined, and meetings convened between the researchers to consider the coherence of the narrative and to validate the messages contained in the analysis.

Our review had two principal findings. We found that the definition of non-formal learning is contested and lacks consensus, with many favouring a continuum model (see next section). We also found very few examples of explicit methods that could be used to research non-formal learning. This methodological challenge is recognised by Eraut [ 12 ] when he states that “the problems faced by researchers investigating non-formal learning are very considerable” [ 12 , p.119].

In the next section we discuss how non-formal learning is defined, before exploring the importance of its recognition, and the steps we took to develop a method for researching it in a classroom learning environment.

2.1 Defining non-formal learning

Despite there being longstanding interest in the demarcation between formal, non-formal, and informal learning since the 1960s (for example, see [ 13 ]), some of the reviewed literature highlights how the language and concepts relating to these types of learning are contested and imprecise [ 14 , p.30]. We found that much of the reviewed literature draws on Coombs and Ahmed’s [ 15 ] definition of non-formal learning. They define non-formal learning as “any organized, systematic, educational activity carried on outside the framework of the formal system to provide selected types of learning to particular subgroups in the population, adults as well as children” [ 15 , p.8].

Often, non-formal learning can be defined in terms of what it is not, appearing to be “a ‘negative’ concept in the sense that it is a negation of something else. It gives little positive indication of content, profile or quality” [ 5 , p.22]. This point is reiterated by the Organisation for Economic Co-operation and Development (OECD) who observe that there is fair consensus around the concepts of formal and informal learning, but much less around non-formal learning [ 16 ]. They go on to conceptualise a continuum model, with non-formal learning being a midpoint between formal and informal learning (Fig.  1 ). It might be argued that the OECD leave non-formal learning relatively ill-defined when they state that ‘for the majority of authors, it seems clear that non-formal learning is rather organised and can have learning objectives’ [our emphasis] [ 16 ].

Representation of OECD [ 16 ] definitions of learning

The continuum model of learning is found in other literature which notes that non-formal learning has overlaps with other learning forms. For example, in common with formal learning it can be systematically planned [ 17 ] and structured around learning objectives [ 18 ]. On the other hand, and in common with informal learning, it can take place outside of compulsory educational provision [ 19 ] and learning motivation may be intrinsic to the learner [ 20 ].

Through our analysis of the literature, we identified 14 dimensions that we felt discriminated between formal, non-formal, and informal learning (Table  1 ). Although we need to point out that all of the characteristics of a learning type do not need to be present at one time to categorise learning as being either formal, non-formal or informal, our representation of non-formal learning suggests that it occupies a space between formal and informal learning which makes it particularly interesting.

Non-formal learning has some specific characteristics. It can be structured, written down, and planned, and it can also occur in settings where formal, compulsory education is provided. It is intentional from the student’s perspective and motivation for learning is more likely to come from the student than from other extrinsic factors, although extrinsic factors may be present (e.g., receiving a grade for playing a musical instrument).

Non-formal learning also has a hybrid quality. Non-formal learning can share characteristics of formal learning and curricula, which make it possible for it to be recognised and accredited. On the other hand, non-formal learning tends to be more flexible than learning in formal contexts [ 21 ]. This means that non-formal curricula can focus on content that relates to learners’ interests (e.g., focusing on content use in contexts that are meaningful to learners, or where learners exercise some choice in learning content). It also means that non-formal curricula can make it more possible to flexibly pace learning based on learner needs (e.g., address aspects that may be more relevant to learners’ lives at the time, rather than focusing on covering set content or objectives in a very strict fashion). Non-formal learning also has a cognitive, emotional, social and behavioural emphasis, all of which can support the development of various skills such as communication, organisational, leadership, and social and inter-cultural skills [ 22 ].

2.2 ‘Noticing’ non-formal learning and developing professional vision

Our review had two linked aims: to clarify the language and concepts related to learning formality, and to gather information on methods that researchers or teachers might use to capture evidence of non-formal learning in classroom contexts. We were happy that we had achieved the first of these aims, but we were disappointed in our second aim since we could find no strong evidence of methods or tools that were developed in research projects that we could use to identify non-formal learning.

This gap in the literature is problematic because of the close relationship between recognition and value. For example, recognition through assessment is one process of giving value to learning through capturing evidence of its processes and outcomes that are open to wider scrutiny. It is also noted in informal learning literature that some assessment processes can stifle opportunities. Rogoff et al. [ 3 ] outline how the narrow learning indicators that can be associated with formal, school-based assessments can restrict the abilities of learners to develop and show initiative or interest in learning.

A lack of a systematic method for capturing evidence of non-formal learning therefore threatens its status and undermines recognition processes that might give credit to learners engaged in non-formal learning. To develop tools for capturing evidence of non-formal learning we thought it was important to look closely at ‘recognition’ as an activity, and how tools support this process. This close look involved consideration of interactions between language, personal experience, professional vision, and noticing.

The anthropologist Charles Goodwin [ 23 ] highlights how the development of professional vision enables groups of people to make consistent observations, through structuring their capacity to notice things. In this way, experience is a major influence on noticing and meaning making. For example, a study [ 24 ] shows how teachers can notice learning in non-formal contexts, with experienced teachers’ noticing being more rich than that of novice teachers.

In addition to the link between experience and professional noticing, mechanisms of noticing have individual and social dimensions that are influenced by language. For example, Goodwin [ 23 ] analyses police officers’ systematic use of language to articulate their professional vision, allowing others to understand the way that they notice things based on their accrued experience.

At an individual level, noticing involves self-reflection in-action, which is ‘a starting point in exploring the issue of being a professional’ [ 25 , p.90]. In the case of teachers, this individual reflection would be implicated when they make sense of what learners demonstrate in a learning task. The social dimension of noticing occurs where an individual’s reflections are coordinated with those of other professionals [ 26 ]. Having access to a shared language framework allows professionals to rationalise and reach resolution [ 27 ], and to validate their shared thinking through activities such as peer working and receiving feedback [ 24 ]. Sydorenko et al. [ 28 ] “consider formal and non-formal education technologies as a factor in improving the quality, efficiency and effectiveness of continuing education” [ 28 , p.4]. Providing professionals with the language and tools to notice non-formal learning could encourage self-reflection and recognition of their own development. It could also give space to recognise the benefits that can come from less formalised types of learning.

It is perhaps no surprise that alongside the ‘contested, imprecise, and under-researched’ picture of non-formal learning that emerged in our literature review, we also found relatively little evidence of any specific, systematic research approaches that should be used for capturing non-formal learning in schools. At a general level the outcomes of the literature review suggested that the hybrid quality of non-formal learning (i.e., that it possesses characteristics of both formal and informal learning) would require the use of a diverse set of approaches to capture it, but the lack of a recognised approach suggested to us that there was a need for an organized framework that could inform future research.

3 Developing a data gathering approach

The 14-dimension model of learning formality that we identified through our literature review provided us with a basis for conceptualising non-formal learning, and a focus for developing an approach to data collection. A closer look at the dimensions shows that they can be grouped into three superordinate factors (Table  2 ). Half of the dimensions relate to the character of learning itself, such as the structure and emphasis of learning. Four dimensions relate to the learner, linking to aspects such as motivation and intention. The final category of dimensions relates to the learning environment, such as learning location and pedagogy.

The differences in focus of the non-formality factors have implications for research method design since data collection involves capturing evidence about learning, about the learner, and about teaching. The hybrid nature of non-formal learning involves a mixture of approaches to capture the interplay of structure and flexibility. Some of the more formal characteristics of non-formal learning, such as structured learning plans or learner intentionality and motivation, may lend themselves to interview methods as these dimensions have explicit and considered qualities of which participants in the learning process demonstrate awareness. Some of the less formal characteristics of non-formal learning require an approach that relies less on participant self-awareness, as these may defy articulation by those involved in the learning process. Observation methods are useful as they can give insight into the existence of in-the-moment indirect teaching methods and learning interaction patterns.

To capture non-formal learning in schools we wanted to trial a combination of interview and observation methods to capitalise on the strengths of each. To consider the 14-dimensions of non-formal learning we designed two interview schedules, one for teachers and one for students, which gathered information about learning intentions and experiences. We also developed an observation schedule to support classroom observations that captured reflections on learning content and pedagogic approaches (see Online Appendix).

3.1 Pilot data collection

To evaluate the interview and observation schedules we piloted them in a school that was teaching the Theory of Knowledge (ToK) component of the International Baccalaureate ® (IB) programme. This programme has prima facie non-formal elements, transcending the idea of learning through academic subjects and includes engagement with learning that has broader community service elements.

We observed a class of students, aged 17–18, in a comprehensive school in Eastern England. The class included around 60 students, three teachers, and a classroom assistant. The lesson lasted for 60 min and focused on the nature of knowledge and truths. Two researchers carried out the observation simultaneously. For reasons of student identity protection and ethics we did not video record the classroom session. Informed consent was obtained from all individual participants included in the study.

Following the classroom session, we carried out two group interviews with students from the class (12 students in total) and an interview with two of the teachers. These interviews were audio recorded with student and teacher identities anonymised.

Prior to analysis we transcribed the three audio recorded interviews. We also combined our classroom observations into a single observation schedule. We used MAXQDA software [ 11 ] to code evidence of the 14 non-formal learning dimensions in the interviews and the observation schedule. Both researchers coded all of the data independently and met to resolve differences. Reaching consensus is a challenge for qualitative research. In our resolution meetings we confronted varying interpretations of codes, regardless of the inclusion of memos which outlined the definition of specific codes. This highlights how text is rarely unambiguous in interpretation, requiring research designers to include systematic checks to ensure common interpretations are held by researchers. Our resolution meetings resulted in a single coded master file, including 122 codes, which was the basis for final analysis. In the following section we outline our evaluation of our method before sharing what we found out about non-formal learning through our method use.

4.1 Method evaluation

4.1.1 which methods captured which features of non-formal learning.

By looking at whether the different methods allowed diverse or similar insights to each other, we were exploring whether there were any advantages of adopting a mixed methods approach to data collection. To do this, we compared the extent to which evidence of the non-formal dimensions were found through interview and observation methods. Figure  2 shows the extent to which each dimension was evidenced through each method. For example, motivation for learning [LR-Mo] was evidenced exclusively through interviews, whilst evidence about the structuring of learning [LG-St] was captured through interview and observation approaches.

Non-formal learning dimensions captured through observation and interview methods. Note: Abbreviations refer to the 14 dimensions represented in Table  2

The interview method evidenced all dimensions, it captured a higher frequency of codes than observation, and it was also more effective than the observation method for gathering ‘other’, unanticipated data (see below). This difference in capacity to evidence a broad range of data may reflect the way that interview probes can search for evidence. This contrasts with the naturally occurring data that is captured in the observations, with the method capturing things that are directly visible to the observer. For example, when considering learning outside of the classroom it was necessary to rely on reported evidence.

Researcher: Does the student learning always happen in the classroom? Or does it happen somewhere else? Teacher 2: It it’s led in the classroom. We encourage the outside... we don’t necessarily set homework… Teacher 1: So, it’s also drip feeding into all the other subjects… Teacher 2: Yeah, but it’s really clear that the students are thinking about it when they’re outside and interacting with the world, and you see that when it comes back to assessments ... Or you’ll get those comments in the corridor when those students will say, “aha, I read that this thing you know, [I] listen to that podcast and then I read this thing”. (Teacher Interview)

On this evidence it might be argued that observation methods lack a degree of utility as there is a clear opportunity cost to using them in terms of time and analytical effort. Despite this, we also think that there is a benefit of use. When observing, a researcher has less of a role in agenda setting compared with using interviews, and this has implications for the validity of the data. We feel that observation is particularly useful when considering a new area of study. For example, our review of literature [ 9 ] suggested that there was no established method we could rely on, so our two-layer approach gave us an element of reassurance to see whether dimensions were present without our prompting.

Most of the non-formal learning dimensions were evidenced in both teacher and student interviews, suggesting that the use of separate teacher and student interviews allowed a degree of triangulation (discussed further in the concluding section of this paper). An affordance of our methodology, where we sequenced the observation to immediately precede the interviews, was that we could use the methods to complement each other. We could use the interviews to probe aspects of interest from the observation session, and this allowed us to make sense of or verify our earlier reflections. For example, when evidencing the learning structure [LG-St] dimension, our observation evidence noted that the lesson started by paying attention to language issues. In the subsequent interview the teachers were able to rationalise this observation.

‘Language + knowledge initial focus.’ (Lesson Observation). Researcher: Do you have a lot of control about the way that you construct [the lesson]? Teacher 2: We’ll have a kind of time where we have those on the board and say, right, these are the key concepts, how can you actually use them? We just keep kind of bringing that, it’s back to the language use more than anything. (Teacher Interview)

The combination of methods also allowed us to gain a richer insight into the qualitative aspects of some non-formal learning dimensions. For example, the learning relates to/complements formal curricula [LG-Re] dimension was actualised in both direct and indirect ways. In interviews students and teachers gave insight into the direct links between non-formal and formal subject curricula, whilst the observation showed that some of the linking was more oblique (i.e., linking to vaccines and broader science knowledge in general).

Student 4: Some subjects actually, like in psychology … sometimes in the beginning of the lesson, [the teacher] will actually put a Theory of Knowledge link on the board and then you can sort of add that into your journal and then connect it back to Psychology. (Student Interview) Teacher 2: The other thing we do is we do it across different subjects. Also, like we did it with science, didn’t we, when they did their Group 4 Project … Teacher 1: Yes, yeah… Teacher 2: …which was basically them going out and taking pictures of different things in the world, in school, and then coming back and saying like “what’s the science behind this picture?” (Teacher Interview) ‘[Teacher] relating learning to real life (MMR vaccine).’ (Lesson Observation (Page 1).

We found that observation provided evidence of how teachers’ or students’ thinking was realised in practice. For example, when talking about group working, a student’s reflection on teacher encouragement to communicate was complemented by the observation data.

Student 3: [Group work is] definitely encouraged, like at the beginning, a lot of […] lessons. A lot of what was happening was someone would say something, everyone would start writing, but we were encouraged, like repeatedly, to communicate and to discuss. (Student Interview) ‘[Teacher] enforcing respectful listening.’ (Lesson Observation)

We also found that some non-formal learning dimensions tended to be represented more in teacher or in student interviews (Fig.  3 ).

Non-formal learning dimensions found in teacher and student interviews. Note: Abbreviations refer to the 14 dimensions represented in Table  2

For example, learning codification [LG-Co] was evidenced only through teacher interview, whilst learner recognition (of what they have learned) [LR-Re] was evidenced only through student interview.

Researcher: Are there any documents which are central or foundational for planning? Teacher 2: Well, the guide. It’s something I read, the guide, and we read it and read it. We read it all of the time. And then the assessment criteria is something that I look at just to kind of keep grounding. Teacher 1: And within each subject area they’ve got suggestions, so I might cover topic economics, they’ll have suggestions for Theory and Knowledge, so it’s within each subject. (Teacher Interview) Student 3: I think I’ve become through the course a lot more open minded like even in my personal media consumption too. Consuming information from people I don’t necessarily agree with, or I don’t like on social media. (Student Interview)

Similarly, some dimensions tended to be represented more by students, such as learning emphasis (e.g., learning has emotional or social learning elements) [LG-Em], or learner motivation [LR-Mo].

Student 3: There is a lot more discussion in here than other groups, so that [for] learning [we are] put randomly into groups, so we’re learning to talk with people you’re not as comfortable with. (Student Interview) Student 2: I’ve learned a lot of the topics on my own independently just because it’s interesting, like especially through YouTubers like PhilosophyTube . (Student Interview)

The variance between how the dimensions were distributed in the teacher and the student interview data suggests that it was important to capture both perspectives. It is likely that students or teachers have privileged insight into some dimensions, making it important to carry out interviews with both groups.

4.1.2 What features of non-formal learning were captured?

Taken together, our observation and interview methods captured evidence of all of the non-formal dimensions related to learning, the learner, and teaching (Fig.  4 ).

Non-formal learning dimensions and frequencies captured by the research methods. Note: Abbreviations refer to the 14 dimensions represented in Table  2

The figure also shows that the methods were capable of capturing evidence of ‘other’ categories of learning, or learner or teaching behaviours. This meant that the methodology was useful for exploring whether there were some aspects of non-formality that we had not anticipated when we devised our 14-dimensions.

When we analysed the ‘other’ codes (text that did not apparently link to any of our 14 non-formal learning dimensions) we noticed that teachers and students referred to learning links with students’ experiences beyond the classroom.

Researcher: Do you think [where learning happens is] different from other areas you have taught before? Teacher 2: Yes, definitely one hundred percent. I think even you know, in the other subjects that it’s that wide, that kind of real-life bit of it that really makes it. Teacher 1: Yeah, and then encourage so much to think about think about real life, yeah. (Teacher Interview) Researcher: Are the skills you have learned different from other courses that you currently study or that you have studied? Student 2: It does feel like what you’re learning is very much... I mean like it’s meant to be like it’s a thing outside the subject that you use to understand the subject a bit better. (Student Interview)

The association between non-formality and experience-based learning is found elsewhere (e.g., [ 13 ]). Some of this association centres on learner agency and motivation, with concerns that learner motivation is threatened where formal learning programmes treat concepts in isolation from learner experience [ 29 ].

In contrast with informal learning—which is intrinsically engaging, experience-based, but also non-conscious—non-formal learning seeks to harness the links between experience and context in an explicit way so that the learning concepts are explicit and meaningful. Our data suggests that the teachers in our study encouraged non-formal learning through an appeal to students to bring in and draw upon their background experiences, and to relate them to established learning goals. In this way, non-formal learning mirrors an affordance of informal learning where learning is ‘a by-product of other experiences’ [17, p.20]. It also invokes a degree of student agency as the learner is responsible for making meaningful links between their out of school context and their learning.

Closer analysis of this new ‘student experience’ code showed that it linked with data from across all three non-formal learning factors (learning characteristics, learner, and learning environment). Figure  5 shows that non-formal learning relates learner experience to the curriculum [LG-Re] and to knowledge [LG-Kn], it involves the learner being able to recognise their learning [LR-Re], and it implicates learning in different locations (e.g., beyond the classroom) [TE-Lo].

Experience links to learning, the learner, and the learning environment. Note: Dashed lines show associations that are not immediately relevant to this analysis

4.2 Non-formal learning evaluation

4.2.1 what did we learn about non-formal learning (in a formal environment).

We analysed the observation and interview data to identify the dominant intersections between codes (i.e., where two codes co-occurred together at least four times). Figure  6 shows these intersections as well as the methods that evidenced them (lesson observation, student interview, or teacher interview).

Dominant intersections between non-formal learning codes

Indirect teaching behaviours [TE-Te] such as encouraging peer-to-peer learning linked with learning socialisation [LG-So], by which we mean how learners come to conform to school norms and rules. This finding mirrors those of Jung and Choi [ 30 ] who note that “the concept of indirect teaching behaviours can be a valuable addition to the research into the effect of teacher behaviour on the social and moral development of students” [ 30 , p.122].

Indirect teaching behaviours [TE-Te] also intersected with cognitive, emotional, social, and behavioural learning [LG-Em] , which included elements such as learners’ emotions, empathy, relationship management, social awareness and decision making.

Student 4: It’s a huge part of the course, I think for it to be to be successful in ToK, you have to communicate. Because if you don't talk to anyone else around you, then the only perspective you’re gonna have is your own. (Student Interview) Student 2: …they had that whole thing of like how to be a good listener, that they did with the observer and such, where one person had to like look and see if people were, yeah yeah, doing the things that show you’re a good listener. (Student Interview)

Again, this mirrors other findings where indirect teaching behaviours were important in encouraging the development of effective student-peer relationships [ 30 ].

Observations and student interviews captured an intersection between indirect teaching behaviours [TE-Te] and teaching that was directed towards the student and their needs [TE-Di]. The link between these two teaching behaviours appears plausible. Pedagogy that focuses on the learners’ needs would be expected to employ teaching behaviours that encourage things like peer-to-peer learning and task-based learning, where students are at the centre.

Interview data showed that teaching that was directed towards the student and their needs [TE-Di], and this also linked with learners’ motivation [LR-Mo] , which had intrinsic elements.

Student 3: I think I also enjoy questioning like information and it’s very much like you get to explore your own thoughts and it helps you verbalise things, and it gives you words like specific vocabulary you can use to verbalise your thoughts. (Student Interview) Teacher 2: I mean I’d go right back to kind of basic educational theory. That if they, if they can see themselves and they have their voice in it, and if they’re represented within what they’re learning, then it will have a major impact on them. (Teacher Interview)

This link between learner need, motivation, and non-formal learning is noted elsewhere. Eshach [ 20 ] highlights how motivation for learning may be intrinsic to the learner in non-formal learning because it involves a shift from institutionalised control over knowledge (e.g., schools) to individualised control and self-directed learning.

Teaching that was directed towards the student and their needs [TE-Di] also intersected with learner voluntarism [LR-Vo] , suggesting that student choice was an element of the non-formal learning process. It is perhaps noteworthy that this link was evidenced only through the teacher interview, suggesting that it reflected the teachers’ privileged and more extended experience of the ToK course.

Researcher: Is there space for students to determine their own learning goals? Teacher 2: [The students] do have very much say on [the] focus on what it’s gonna be on. Teacher 1: Yeah, a big voice in the curriculum… Teacher 2: …and I mean that’s not just, you know, we can look at that in terms of the IB Diploma programme in terms of all the lessons, but also other elements of the course, like the extended essay in the Diploma programme or the Reflective Project in the… career related programme where they actually have sole, have real autonomy. Teacher 1: Even in classes. Teacher 2: Yeah, yeah, over what it is that they focus on… Teacher 1: …Community projects they want to get involved in, and their goals, what they want to get out of doing them so… Teacher 2: …In the ToK, it’s the exhibition. (Teacher Interview).

The presence of learner choice is commonly recognised as a feature of non-formal learning (e.g., [ 31 ]), and alludes to the space for student agency in the learning process. The link between teacher choice and learning in non-formal contexts is less common [ 9 ], but we found some evidence of this in our data.

Teacher 2: The open mindedness is part of the [course]. I always found the A Level [qualification] to be, I mean the fact that they very much even dictate the text that you can read. I think you’re then already in a box to be looking at literature, GCSE [qualification] literature, A Level literature. Yeah, the fact that there is a canon. I mean the IB has a prescribed list of authors, but it is enormous. I mean, I have never once found an author that is not on it, and I think that’s encouragement. (Teacher Interview)

Students noted the link between cognitive, emotional, social, and behavioural learning [LG-Em], propositional and procedural knowledge learning [LG-Kn], and indirect teaching behaviours [TE-Te]. Through this linking the students seem to be highlighting the holistic nature of learning and recognising that non-formal learning extends beyond conceptual knowledge.

Researcher: Is this [learning] different from other lessons in how you engage or talk with your peers? Student 5: Sort of, as there’s a lot more debates than even in like history or some of the humanities courses. It’s a lot more about group discussion and evaluating information and stuff like that, rather than just making notes and listening. Student 2: And about challenging each other, it’s not like someone says something [and] everyone’s like “oh yes!”. There’s a lot of you know, “well, I actually think this” and “have you thought about this?” Student 3: ...Yeah and yeah, so I think we’re encouraged to bring ToK into other lessons and that’s something that’s really encouraged, like by both our subject teachers and ToK teachers. (Student Interview)

These connections between indirect teaching behaviour and social skills are also highlighted by Sadler [ 32 ] who notes that “few physical, intellectual or social skills can be acquired simply through being told about them. Most require practice in a supportive environment which incorporates feedback loops” [ 32 , p.120]. These intersections suggest that indirect teaching behaviours (e.g., peer collaboration, modelling, demonstration, feedback) are capable of encouraging students’ social learning skills development.

Making space for non-formal learning in the classroom arguably requires teachers to relinquish some of the techniques through which they exert hierarchic power on the learning process, allowing more room for learners to take an active role in their learning. Our data suggested some evidence that teachers were socialising the students to be able to learn through discursive interaction.

Teacher 2: It’s getting them to feel after years of GCSE [qualifications], as well as getting to feel confident with that discussion, that discursive nature of it as well. (Teacher Interview) Student 1: I suppose you can choose whether you want to be having a discussion or whether you just want to be reading, but then they come around and tell you “you should be talking now” or “you should be reading now”. They do that. (Student Interview)

It appears perhaps counter-intuitive, but this encouragement into a new, less formal learning culture involves the teachers initially taking away some learner agency. Through mandating group working and discursive interactions, the teachers are shifting the learning focus from an individualist to a group-learning mindset.

5 Discussion and conclusions

Our broader project had two linked aims. The first aim was to clarify the language and concepts related to learning formality. We wanted to counter some of the limitations of previous definitions of non-formal learning which considered it in terms of ‘what it is not’. Jens Bjørnåvold, who has done much to raise awareness around the importance of non-formal learning—particularly in relation to validating the non-formal learning components of qualifications (e.g., [ 33 , 34 ]) and supporting learning recognition across national borders (e.g., [ 35 ]) notes that it can often be seen as a negative concept [ 5 ]. Our framework of non-formal learning dimensions, based on reviewed literature, clarified the concept through language that showed where non-formal learning differed or matched with aspects of formal or informal learning. Drawing on some of our empirical data we were able to extend the dimensions that we identified through our interpretation of this literature. We became more aware that a key dimension of non-formal learning in formal contexts is its links with students’ experiences beyond the classroom (see below for more on this).

Our second project aim was to augment the limited base of information that exists around appropriate research methods for capturing evidence of non-formal learning, particularly in formal learning contexts. We found that the use of interviews was effective at evidencing all of our dimensions of interest. This links with the way that interviews allow researchers to probe or explore further any issues of interest. We also found that observation method added value as it augmented evidence gathered through interviews and avoided some well-known issues around the weaknesses of self-report data about learning [ 2 ]. For example, we could capture evidence of indirect teacher behaviours, which Jung and Choi [ 30 ] note is important for encouraging learner motivation and developing effective peer relationships.

It was important that our data gathering tools incorporated both student and teacher perspectives because non-formal learning is intentional. Students’ learning intentions spur their agency and teachers’ goals guide their planning. The perspectives captured through student and teacher interviews also allowed us to construct a richer picture of non-formal learning. The student interviews drew our attention to the holistic nature of non-formal learning. Whilst our review work had spent time disambiguating the dimensions of this learning, the student interviews made us very aware of the number of intersecting links that bind these dimensions, and which make non-formal learning complex. This perspective is supported by Badger [ 36 ] who highlights claims that non-formal learning has a holistic impact on learners’ cognitive, affective and social development. We already knew from the literature that students’ exercise of choice in the learning process was a feature of non-formal learning (e.g., [ 37 ]); our empirical pilot study data also broadened this idea out further to draw our attention to how it may also implicate teacher choice.

Our pilot study gave us some insight into how non-formal learning in formal contexts may involve the need for learning culture change. Our teachers and students described how the organisation of learning in this context differed from other contexts, and how this involved transitioning towards more group-focused learning interactions. Linking back to our aims, we were also able to test out ‘a priori’ ideas about non-formal learning dimensions that derived from a literature review. By leaving space to capture evidence of other elements, our empirical evidence suggested that student experience is a key part of non-formal learning, allowing us to re-think our 14-dimension framework and to incorporate this additional dimension.

Table 3 shows our refined framework, highlighting this dimension alongside the others. This point is supported by literature that foregrounds experience in non-formal learning. Norqvist and Leffler [ 13 ] note that non-formal learning tends to capitalise on the use of experience-based learning approaches. Students’ value of knowledge is also linked to experience [ 38 ], with learning in ‘authentic’ contexts (i.e., involving real-world problems in settings that are relevant to the learner) motivating learners (e.g., [ 39 ]).

We think that the model of non-formality that we present here could provide professionals with a useful way of conceptualising learning. Eraut [ 12 ] highlights the importance of tacit knowledge Footnote 4 for professionals, and particularly how this knowledge is gained through socialisation, observation, and participation rather than formal inquiry. Eraut [ 12 ] also highlights that professionals’ use of intuition depends more on prior and tacit knowledge rather than theory. As tacit knowledge is gathered through experience, such experience is crucial for any professional. Our 15-dimension model of learning formality emphasises the role that experience plays in non-formal learning. Our conceptualisation of non-formality is therefore important, as it can give professionals the awareness and language to identify when non-formal learning is occurring, thus validating learning that would not fall into the ‘formal’ category, but which is essential to any professional developing their skills and knowledge.

As various educational jurisdictions explore learning and recognition models that incorporate students’ interests, self-direction, and co-agency (e.g., [ 7 , 8 ]) it appears timely to consider the nature of less-formal learning arrangements that may encourage some of these aspirations. In the Australian context, the importance of student learning engagement is accepted through the way that all states and territories address it in their provisions [ 40 ], with this appearing to encourage a variety of initiatives to increase student agency (e.g., [ 41 , 42 , 43 ]). We suggest that our refined framework and data gathering method have the potential to support the recognition of student agency and other features of non-formal learning through enabling teachers’ and students’ noticing. Our framework provides a language that enables noticing at a personal level through supporting individual teacher reflection on their learning context. The framework also gives teachers a language that supports noticing at a social level through allowing idea transfer between teaching professionals.

At this point it is important to consider some limitations of our pilot study. The pilot study was small in scale and focused on a specific course, which was quite different from more conventional subjects, so we cannot draw conclusions about what would happen if we used our tools in another context, such as a more conventional GCSE subject. We therefore recommend using our methods to explore non-formal learning in other contexts, such as afterschool clubs, museums, and field trips, as we were not in a position to explore these contexts in our pilot study. It would also be interesting to see if students from a later year group, with broader learning experience and who would be closer to finishing their course, would have expressed similar views to our cohort. At the same time, we could argue that being relatively new to the course meant our students were in a good position to contrast the formality of their current and prior learning experiences.

We also need to acknowledge that the amount of observed lesson time was limited and needs further study. The relative lack of data captured by the observation method compared with the interview method might be a consequence of the fact that we only observed the students for an hour. This means that the triangulation between interview and observation data is limited and requires further exploration with more extended observation. A final challenge we need to confront relates to the inter-relationship between our method, our observation outcomes, and the environment in which our method is used. If we fail to observe evidence about a dimension of non-formal learning in a classroom, we cannot be sure whether it is a failure of the method (i.e., the learning occurred but was missed by the method), or whether it is a valid reflection that this learning did not exist (and so could not be observed). This reinforces why it is important to extend the number and variety of cases where the method is used beyond the limited pilot phase to fully evaluate its robustness. These cases should include non-lesson activities that take place beyond the classroom, such as field trips or club/sport activities.

To conclude, understanding non-formal learning is complex and capturing it is challenging. Non-formal learning is important for many different contexts (e.g., museums and Botanical Gardens) and allows access to education for learners who may be disengaged or anxious about learning in more formal settings. By providing professionals with the language around non-formal learning and the tools to capture it, we might facilitate the recognition of development and learning that may otherwise go unnoticed.

Data availability

The datasets generated and analysed during the current study are not publicly available due to the sensitive nature of the information, which includes responses from case studies conducted in a classroom setting. These activities were carried out with ethical approval and consent, adhering to strict confidentiality and privacy standards to protect participant anonymity. As such, the raw data are subject to ethical restrictions to prevent any potential breach of confidentiality. Aggregated data and findings that ensure participant anonymity can be made available upon reasonable request to the corresponding author, subject to compliance with ethical guidelines. Requests for access will be reviewed on a case-by-case basis to ensure they meet ethical standards for data sharing.

Web of Science Core Collection; University of Cambridge Library iDiscover; Taylor & Francis journal database; Scopus/Elsevier database; Wiley Online Library; ERIC.ed.

To identify ‘non-formality’ we used a series of terms that are commonly associated with this area of curriculum enquiry (e.g., see [ 10 ]). These terms were: Non-formal, Informal, Non-taught, Unstudied, Hidden, Implicit, Invisible, Unwritten, or Covert.

These low inference codes included: ‘Non-formal curriculum definition’, ‘Non-formal curriculum benefits’, ‘Non-formal curriculum disadvantages’, ‘Non-formal learning definition’, ‘Non-formal learning benefits’, ‘Non-formal learning disadvantages’… The codes were repeated for formal and informal curricula and learning.

Tacit knowledge is gained through living experience in personal life and professional development [ 12 ].

Bourke R, O’Neill J, Loveridge J. Children’s conceptions of informal and everyday learning. Oxf Rev Educ. 2018;44(6):771–86.

Article   Google Scholar  

Schugurensky D. On informal learning, informal teaching, and informal education: addressing conceptual, methodological, institutional, and pedagogical issues. In: Meijuni O, Cranton P, Táíwò O, editors. Measuring and analyzing informal learning in the digital age. IGI Global; 2015. p. 18–36.

Chapter   Google Scholar  

Rogoff B, Mejía-Arauz R, Correa-Chávez M. A cultural paradigm—learning by observing and pitching in. Adv Child Dev Behav. 2015;1(49):1–22.

Google Scholar  

Rogoff B, Paradise R, Arauz RM, Correa-Chávez M, Angelillo C. Firsthand learning through intent participation. Annu Rev Psychol. 2003;54(1):175–203.

Bjørnåvold J. Making learning visible: identification, assessment and recognition of non-formal learning in Europe. Thessaloniki: Cedefop; 2000.

Council of Europe. Linguistic Integration of Adult Migrants (LIAM). Formal, non-formal and informal learning. 2022. https://www.coe.int/en/web/lang-migrants/formal-non-formal-and-informal-learning . Accessed 23 Mar 2022.

South Australian Certificate of Education. Revitalising the Research Project: South Australian Certificate of Education. 2023. https://www.sace.sa.edu.au/innovating/revitalising-rp . Accessed 23 Oct 2023.

Scottish Government. It’s our future: independent review of qualifications and assessment. 2023. http://www.gov.scot/publications/future-report-independent-review-qualifications-assessment/pages/5/ . Accessd 23 Oct 2023.

Johnson M, Majewska D. Formal, non-formal, and informal learning: What are they, and how can we research them? Cambridge: Cambridge University Press & Assessment; 2022. https://www.cambridgeassessment.org.uk/Images/665425-formal-non-formal-and-informal-learning-what-are-they-and-how-can-we-research-them-.pdf

Endeley MN, Zama MMA. Perspectives in curriculum studies. Denver: Spears Books; 2021. p. 337.

Book   Google Scholar  

VERBI Software. MAXQDA 2022. Berlin, Germany; 2021. https://maxqda.com

Eraut M. Non-formal learning and tacit knowledge in professional work. Br J Educ Psychol. 2000;70(1):113–36.

Norqvist L, Leffler E. Learning in non-formal education: Is it “Youthful” for youth in action? Int Rev Educ. 2017;63(2):235–56.

Colley H, Hodkinson P, Malcolm J. Learning and skills research centre. Informality and formality in learning. Leeds: Learning and Skills Research Centre; 2003.

Coombs PH, Ahmed M. Attacking rural poverty: How nonformal education can help. A research report for the world bank prepared by the international council for educational development. Essex, Conn. USA: International Council for Educational Development; 1974.

OECD. Recognition of Non-formal and Informal Learning. 2010. https://www.oecd.org/education/skills-beyond-school/recognitionofnon-formalandinformallearning-home.htm . Accessed 23 Jan 2024

Allaste AA, Beilmann M, Pirk R. Non-formal and informal learning as citizenship education: the views of young people and youth policymakers. J Appl Youth Stud. 2021. https://doi.org/10.1007/s43151-021-00059-z .

Garner N, Siol A, Eilks I. The potential of non-formal laboratory environments for innovating the chemistry curriculum and promoting secondary school level students education for sustainability. Sustainability. 2015;7(2):1798–818.

Filippoupoliti A, Koliopoulos D. Informal and non-formal education: an outline of history of science in museums. Sci Educ. 2014;23(4):781–91.

Eshach H. Bridging in-school and out-of-school learning: formal, non-formal, and informal education. J Sci Educ Technol. 2007;16(2):171–90.

Ionescu F. Non-formal education approaches in school organization: methodological aspects and case study. J Educ Soc Multicult. 2020;1(2):1–14.

Souto-Otero M. Young people’s views of the outcomes of non-formal education in youth organisations: its effects on human, social and psychological capital, employability and employment. J Youth Stud. 2016;19(7):938–56.

Goodwin C. Professional vision. Am Anthropol. 1994;96(3):606–33.

Xenofontos C, Hizli AS. Prospective primary teachers’ professional noticing in non-formal learning environments: the case of a mathematics fair. Educ Sci. 2022;12(1):55.

Ghaye T. Teaching and learning through reflective practice: a practical guide for positive action. 2nd ed. London: Routledge; 2010. p. xii+208.

Johnson M, Mercer N. Using sociocultural discourse analysis to analyse professional discourse. Learn Cult Soc Interact. 2019;1(21):267–77.

Johnson M. Learning to think alike: a study of professional Examiners’ feedback interactions in a UK qualification awarding organisation. In University of Geneva, Switzerland; 2018. https://www.cambridgeassessment.org.uk/Images/505394-learning-to-think-alike-a-study-of-professional-examiners-feedback-interactions-in-a-uk-qualification-awarding-organisation.pdf

Sydorenko V, Shorobura I, Ponomarenko A, Dei M, Dzhus O. Application of technologies of formal and non-formal education for continuous professional development of the modern specialist. In: Revista Tempos e Espaços em Educação [Internet]. 2020, p. 1–24. https://seer.ufs.br/index.php/revtee/article/view/14729 . Accessed 22 Jan 2024.

Cain T, Chapman A. Dysfunctional dichotomies? Deflating bipolar constructions of curriculum and pedagogy through case studies from music and history. Curric J. 2014;25(1):111–29.

Jung H, Choi E. The importance of indirect teaching behaviour and its educational effects in physical education. Phys Educ Sport Pedagogy. 2016;21(2):121–36.

Alnajjar EAM. The impact of a proposed science informal curriculum on students’ achievement and attitudes during the Covid-19. Int J Early Child Spec Educ INT-JECSE. 2021;13(2):882–96.

Sadler DR. Formative assessment and the design of instructional systems. Instr Sci. 1989;18(2):119–44.

Bjørnåvold J. Assessment of non-formal learning: the quality and limitations of methodologies. Vocat Train Eur J. 1997;12:52–67.

Bjørnåvold J. Identification, assessment and recognition of non-formal learning: European tendencies. In: Identification, evaluation and recognition of non-formal learning. Luxembourg: Office for Official Publications of the European Communities; 2002. pp. 9–32. (CEDEFOP panorama series).

Colardyn D, Bjørnåvold J. Validation of formal, non-formal and informal learning: policy and practices in EU member states. Eur J Educ. 2004;39(1):69–89.

Badger J. Learning in non-formal settings: investigating cemetery guides’ talk during school visits. Int J Educ Res. 2021;1(109):101852.

Harris MSG. Fulfilling a European vision through flexible learning and choice. Eur J Educ. 2012;47(3):424–34.

Lee AN. Development of a parent’s guide for the Singapore primary science curriculum: empowering parents as facilitators of their children’s science learning outside the formal classrooms. Asia-Pac Forum Sci Learn Teach. 2012;13(2):1–27.

Affeldt F, Meinhart D, Eilks I. The use of comics in experimental instructions in a non-formal chemistry learning context. Int J Educ Math Sci Technol. 2018;6(1):93–104.

Pedler M, Hudson S, Yeigh T. The teachers’ role in student engagement: a review. Aust J Teach Educ Online. 2020;45(3):48–62.

Andersson DL. Promoting student agency in the classroom. Teacher Magazine. 2021. https://www.teachermagazine.com/au_en/articles/promoting-student-agency-in-the-classroom . Accessed 30 Jan 2024.

South Australia Department for Education. Episode 11: embracing student agency. https://www.education.sa.gov.au/podcast/podcasts/episode-11-embracing-student-agency . Accessed 30 Jan 2024.

Wegener C. Lessons from an Australian School: 4 Key Strategies for Promoting Student Agency [Internet]. Aurora Institute. 2022. https://aurora-institute.org/blog/lessons-from-an-australian-school-4-key-strategies-for-promoting-student-agency/ . Accessed 30 Jan 2024.

Download references

Acknowledgements

We thank the students and teachers who participated in this study, for their time and commitment.

Author information

Authors and affiliations.

Research Division, Cambridge University Press and Assessment, Shaftesbury Road, Cambridge, CB2 8EA, UK

Martin Johnson & Dominika Majewska

You can also search for this author in PubMed   Google Scholar

Contributions

Both authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Martin Johnson and Dominika Majewska. The first draft of the manuscript was written by Martin Johnson and Dominika Majewska.

Corresponding author

Correspondence to Martin Johnson .

Ethics declarations

Informed consent.

Informed consent was obtained from all individual participants included in the study.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Johnson, M., Majewska, D. What is non-formal learning (and how do we know it when we see it)? A pilot study report. Discov Educ 3 , 148 (2024). https://doi.org/10.1007/s44217-024-00255-y

Download citation

Received : 10 April 2024

Accepted : 05 September 2024

Published : 12 September 2024

DOI : https://doi.org/10.1007/s44217-024-00255-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Non-formal learning
• Formal learning
• Informal learning
• Curriculum organisation
• Voluntarism
• Recognition
• Observation methods
• Interview methods
• Find a journal
• Publish with us
• Track your research