literature review of quality management

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Int J Environ Res Public Health

A Systematic Literature Review of Quality Management Initiatives in Dental Clinics

Emil lucian crisan.

1 Faculty of Economics and Business Administration, Department of Management, Babes-Bolyai University, 400591 Cluj-Napoca, Romania; [email protected]

Bogdan Florin Covaliu

2 Faculty of Medicine, Department of Community Medicine, Public Health and Management, Iuliu Hatieganu University of Medicine and Pharmacy Cluj-Napoca, 400337 Cluj-Napoca, Romania

Diana Maria Chis

3 Faculty of Economics and Business Administration, Department of Finance, Babes-Bolyai University, 400591 Cluj-Napoca, Romania; [email protected]

By considering the recently proposed definitions and metrics, oral healthcare quality management (OHQM) emerges as a distinct field in the wider healthcare area. The goal of this paper is to systematically review quality management initiatives (QMIs) implementation by dental clinics. The research methodology approach is a review of 72 sources that have been analyzed using the Context–Intervention–Mechanism–Outcome Framework (CIMO). The analysis identifies five mechanisms that explain how quality management initiatives are implemented by dental clinics. The simplest QMIs implementations are related to (1) overall quality. The next ones, in terms of complexity, are related to (2) patient satisfaction, (3) service quality, (4) internal processes improvement, and (5) business outcomes. This paper is the first attempt to provide a critical review of this topic and represents an important advancement by providing a theoretical framework that explains how quality management is implemented by practitioners in this field. The results can be used by scholars for advancing their studies related to this emerging research area and by healthcare managers in order to better implement their quality management initiatives.

1. Introduction

This paper has been developed considering the emergence of oral healthcare quality management (OHQM) as a distinct field of research in the wider medicine quality management area. In the next paragraphs, the theoretical background of this paper is presented, including more topics as the adoption of quality management initiatives (QMIs) in healthcare, the particularities of oral healthcare and QMIs’ implementation in this field, and the main research streams concerning OHQM. We also present our research question within the final paragraph of this section.

The implementation of quality management initiatives in general healthcare organizations has been analyzed in various papers since quality has surpassed in importance the costs of the service [ 1 ]. While initially, QMIs were observed in healthcare by considering a more general approach [ 1 ], lately, this field has diversified, and more narrow research areas have emerged. The adoption of quality management models, such as the Malcolm Baldrige Quality Award criteria, the European Foundation Quality Management Excellence Model, and the chronic care model, has initially been an important approach, especially for hospitals [ 2 ]. In these cases, adoption efforts and QMIs were extensive, and although they affected the whole system, the results of these interventions were limited [ 2 ]. Later, the adoption of total quality management (TQM) has been observed as challenging [ 3 ], while the six sigma model led to good results related to costs, satisfaction, and resource utilization [ 4 ]. In a comparison of the use and the effectiveness of quality management methodologies in surgical healthcare [ 5 ], it was revealed that the most used ones are: continuous quality improvement, six sigma, TQM, plan–do–study–act or plan–do–check–act, statistical process or quality control, lean, and lean six sigma. Additionally, in a detailed analysis of lean and six sigma adoption in healthcare, it was presented that six sigma (a detailed and consistent continuous improvement system) has been reported earlier in literature, while lean techniques have been more often found in literature (74,63%), in comparison to lean six sigma (22%), and six sigma (18,15%) [ 6 ]. Operations management techniques used in the healthcare industry, such as VSM (visual stream mapping) and standardization of work and visual management, are also recognized as widely used techniques [ 6 ]. More narrow analyses deal with the adoption of specific quality management tools in the healthcare industry. A simple Kano model is recognized as very hard to be used in healthcare since there are many variations regarding customer needs and preferences concerning different types of care provided by healthcare providers [ 7 ]. The use of SERVQUAL in healthcare services for assessing their quality has also been tested, revealing the importance of promptness of response received by patients, cleanliness and hygiene, and empathy of doctors and employees as main areas of quality perceived by patients [ 8 ].

Oral health is recognized as an important determinant for overall health and well-being, and from a statistical point of view, it can be associated with physical, mental, and general health, energy levels, work limitation, depression, and appetite [ 9 ]. It is estimated that dental diseases accounted worldwide in 2015 for USD 356.80 billion as direct costs (dental expenditure), while indirect costs associated with these diseases were estimated at USD 187.61 billion (productivity losses) [ 10 ]. Oral healthcare is different from general care, being characterized by: regular and asymptomatically participation of patients, primarily surgical nature, associations with pain and anxiety, and primarily cosmetic and secondarily disease treatment nature [ 11 ]. Moreover, dental practitioners pay their own wages by the number of patients and interventions they make and are involved in commercial activities, with the dental patient adopting customer rather than patient attitudes [ 11 ]. Though these obvious differences exist, OHQM has adopted in time practices previously used in general medicine quality management. The use of Donabedian’s structure, process, outcome system approach on quality management [ 11 , 12 , 13 , 14 , 15 ], and the use of the quality dimensions proposed by the Institute of Medicine (IoM) (safety, effectiveness, timeliness, patient-centeredness, efficiency, and equity quality dimensions) [ 12 , 13 , 16 ], similar to the dimensions proposed by Donabedian [ 17 , 18 ] and used by Campbell and Tickle [ 11 ], have also been identified for OHQM.

Much of the recent research concerning OHQM is focused on the development of concepts and defining quality. Much of the research is associated with defining quality in this field. It is recognized that quality in oral healthcare is poorly defined in comparison to quality in general medicine [ 11 , 12 , 13 , 14 ], and that the lack of a generally accepted definition and measurement of oral healthcare quality blocks its development [ 11 , 15 , 16 ]. A working definition for quality of oral healthcare has been proposed, including seven domains (patient safety, timeliness, patient-centeredness, equitability, efficiency, effectiveness, and accessibility) and 30 items [ 19 ]. Other conceptual contributions for OHQM target the introduction of adequate sets of measures [ 12 , 13 , 15 ], as well as the establishment of specific goals relevant only for OHQM (which should be generally accepted by practicians, thus providing a unified definition of quality management) [ 11 , 12 , 13 ]. A systematic literature review concerning the metrics used in OHQM reveals that they are mainly related to patients’ satisfaction, 9 out of 11 studies presenting evidence for this patient-centered quality management approach, while the rest are related to self-assessment of practice made by a dentist or a manager [ 12 ]. Efficiency (costs related aspects), and equity are poorly considered.

Secondly, besides these conceptual papers, there are papers and sources that bring evidence that quality management in oral healthcare is transposed into regional and national standards of initial education and continuing professional development of dental professionals [ 20 ]. OHQM is also presented as an activity governed by the state, multinational bodies such as the European Union, or professional associations [ 14 ], which establish policies such as the Quality in Dentistry policy proposed by the FDI World Dental Federation [ 21 ].

Finally, there are papers that analyze the context and the results of quality management practices adoption in healthcare. These results are contradictory. Though these quality management practices have been proven to positively influence healthcare organization performance [ 22 ], it is found that leaders/managers of healthcare organizations are not necessarily well trained or even the right persons for launching such quality management initiatives [ 23 ], and the adoption itself has failed in many organizations [ 24 ]. Contextual factors such as leadership, organizational culture, data infrastructure and information systems, high experience in QMI implementation [ 25 ], but also human resources involvement and their knowledge [ 26 ] are recognized as important factors affecting the success of quality management implementation in healthcare. The lack of a systemic approach and the adoption of rather microsystemic improvements have been regarded as sources for the lack of success in the case of QMIs in healthcare [ 27 ].

After considering the existing OHQM data available in the scientific literature, it is obvious that research in this field is at its beginnings, being mostly concerned with defining quality and establishing metrics. We take this one step further with this systematic literature review and answer the following question: how do dental clinics implement QMIs, as reported by the literature? In order to answer this question, we analyze empirical papers that present QMIs in dental clinics by considering a system design approach—the CIMO Framework proposed by Denyer et al. [ 28 ]. This approach is capable of explaining when and why (context—C), how (intervention—I), and with what results (outcome—O) dental clinics implement specific QMIs. Moreover, this framework targets the identification of explanatory mechanisms regarding how different dental clinics combine C, I, and O. The main result of such an analysis is a theoretical framework that aggregates and explains the QMIs already implemented in practice, this framework being an important input for further advancements in the field.

2. Research Design

The literature review follows the methodology proposed by Tranfield et al. [ 29 ], this methodology being appreciated by medical and quality management researchers due to its transparent and replicative nature [ 6 , 7 , 30 ]. This methodology is commonly used within management literature, being similar to the one detailed within the PRISMA declaration for medical research [ 31 ], since it has been developed considering previous methodologies developed in medical science [ 29 ]. Figure 1 describes the procedure we have followed, including the activities we undertook during each stage.

Research process.

2.1. Planning the Review

The first step was to establish the goal of the review, and that all articles and case studies covering dental clinics QMIs, published at all times, should be included in the review. Later, a pilot search that led to the identification of 12 articles was conducted, this search being used to establish the search strategy and to identify the search terms.

2.2. Performing the Review

Two extended searches were performed in July 2019 (in Web of Science—WoS database) and August 2020 (PubMed database). The initial WoS search included multiple terms in TOPIC (title, abstract, keywords), generating the following number of articles: “dentistry” “customer satisfaction”—7, “dentistry” “quality assessment”—112, ”dentistry” “quality improvement”—43, “dentistry” “quality assurance”—82, “dentistry” “quality management”—11, “dental” “customer satisfaction”—20, “dental” “quality improvement”—142, “dental” “quality assessment”—359, “dental” “quality assurance”—273, “dental” “quality management”—43 articles. This search led to 1092 sources, out of which 223 duplicates were removed, bringing the total to 869 unique sources. A similar PubMed search was performed, all our search terms, with the exception of “customer satisfaction”, being registered as medical subject headings terms. This search led to the identification of 1316 sources, out of which 186 duplicates were removed, bringing the total to 1130 unique sources. These sources were later compared with those from WoS sources, and after excluding duplicates, 183 unique studies were added to the WoS ones. Given the nature of the topic, which belongs both to the management and the medicine field, similar to other reviews, we have considered that the two searches in WoS and PubMed will provide the main relevant sources for our analysis. WoS was chosen due to its comprehensiveness, as it includes a wide range of academic sources in the management field [ 32 ], while PubMed is the mostly used database for medical literature [ 33 ]. Moreover, relevant articles referenced in the previously selected articles have been included in our analysis in order to also cover grey literature, as further described.

The screening of the 1052 sources’ titles and abstracts was conducted using three inclusion criteria: (I1) the articles should present cases related to dental clinics; (I2) the topics addressed in these articles are related to QMIs; (I3) all the papers are written in English. Furthermore, three exclusion criteria were considered: (E1) papers that focus only on a specific part (pathology) of dental treatment (e.g., implant); (E2) papers that focus only on a specific protocol or method related to dental treatment (e.g., radiology); (E3) papers related to national strategies. Each source was analyzed by two of the authors and, in case of doubt, the article was fully read and discussed until a common agreement was reached. The result of screening was 129 eligible sources from both searches. Three reviewers examined the content of these sources, with two reviewers evaluating each source. The same inclusion and exclusion criteria were used, 32 sources related to the WoS search and 20 related to the PubMed search being validated. After extracting the data from sources, 20 new articles, cited by initial sources, were also validated. Therefore, 72 sources were used in this systematic literature review (see Table 1 ).

Sources included in the review and main CIMO details.

All researchers were involved in extracting information from all eligible sources using an online data collection template. This template included metadata fields such as: authors, year of publication, title, publication title, item type (journal article, book chapter, conference paper, report), methodology type (qualitative, quantitative), methodology research methods, OHQM focus (primary or secondary), QMI geographical area, and relevant cited sources.

Data extraction and analysis has been performed following the CIMO framework, which typically produces design propositions that encapsulate patterns of context, interventions, and outcomes to describe the examined phenomenon—in this case, QMIs’ implemented by dental clinics. In management science, it has been previously used to capture how organizational and inter-organizational phenomena occur [ 105 , 106 ]. In comparison to the PICOS criteria which are used to identify components of clinical evidence for systematic reviews in evidence based medicine [ 107 ], CIMO is mostly used for organizational design-oriented research synthesis [ 28 ]. CIMO structure provided the theoretical framework for our approach to coding, which was mostly deductive: we identified a list of codes to reflect the contexts, interventions, and outcomes of QMIs’ implementation and used these existing constructs to make sense of our data by identifying the explanatory mechanisms. For QMIs’ context, two fields have been considered: one for the nature (research/clinical purpose) of the QMI and one for the QMIs’ triggers/expected benefits. Interventions were extracted in a specific field, while for outcomes, four relevant fields have been used: one for the outcomes (similar to triggers/expected benefits identified for context), and two for the nature of the outcome (one for real outcomes versus ideas or recommendations and one for qualitative versus quantitative outcomes).

3.1. Descriptive Analysis

The 72 selected sources were published between 1974 and 2020. Almost 50% entered the literature after 2010 ( n = 35), with a peak of six articles in 2017. Furthermore, 69 of the sources are journal articles, 2 are conference papers, and 1 is a report. Concerning the country of origin, a large number of sources examine dental facilities in the USA ( n = 28), and a significant number of sources ( n = 11) looked at the UK. The majority ( n = 57) of studies refer to dental clinics in Europe and North America. Additionally, 65 sources have quality management as their primary goal, the others having quality management as a secondary one.

The most common research methods were quantitative methods, observed in 64 of sources, and implied statistical analysis of questionnaires results (45 sources) and quantitative observations from patients’ applications, dental care files, integrated electronic health records, list of patient complaints, informed consents, etc. (21 sources). The least commonly used methods were the qualitative methods (15 sources), such as summarizing, categorizing, or interpreting interviews responses (five sources), focus group discussions (three sources), and other qualitative methodologies (eight sources).

3.2. CIMO Results

3.2.1. context.

As stated by Denyer et al. [ 28 ], the context is related to the external and internal environment factors that impact behavioral change. Considering the nature of the intervention, QMIs’ context has been divided into research driven (29 studies), where the intervention was associated with a research project/initiative, and clinical driven (43 studies), where the intervention was initiated internally by the clinic without an exclusive research goal.

Five main categories regarding QMIs adoption triggers/expected benefits have emerged (see Table 2 ). The most common category of triggers of QMIs adoption is patient satisfaction, also mentioned as a more patient-centered oral healthcare, or the improvement of the patient–provider relationship (41 studies). This result confirms the increasing importance placed by the patient [ 70 ], and that patient satisfaction generally has been accepted as an important element of OHQM [ 34 ]. The second most common category of expected benefits refers to the improvement of professional practice and organizational activities (33 studies), including the improvement of professional practice and the overall practice of clinics, reshaping dentist practice patterns, increasing awareness and the level of knowledge among dentists, improving the quality of the work environment, minimizing the time-consuming and stressful patient-search process, managing the quality of products and services delivered to the customer, confidentiality and security, increased opportunities for reusing electronic data for quality assurance and research, etc. [ 52 , 84 , 88 , 96 ]. The third category, recognized in 32 studies, refers to more general triggers for dental organizations: improving the overall quality of care and health outcomes, improving access and reducing disparities in oral healthcare, etc. [ 39 , 53 , 90 ]. The fourth category of dental clinics expected benefits related to QMIs adoption, mentioned in 23 studies, is enhancing service quality of dental care, involving the improvement of the quality of delivered healthcare services, increasing the overall supply of dental services available, and encouraging utilization of dental health services, commitment to provide a high-quality service, achieving and ensuring good service quality to meet or exceed, delivering a more effective oral health service to residents, improving the service quality of care for homeless and vulnerably housed people, etc. [ 36 , 44 , 60 , 81 ]. Finally, the least mentioned category of expected benefits, appearing in only 13 studies, is related to the improvement of dental clinics business outcomes: clinic efficiency, revenue, profit, attracting new patients, enhancing dental care service providers performance and gaining customer preferences, cost containment/savings, financial stability, cost-effectiveness of the new service delivery model, etc. [ 37 , 53 , 103 ].

Context, intervention and outcomes summary.

3.2.2. Intervention

The most common category of interventions (see Table 2 ), applied in the majority of the analyzed cases (37 studies), is the evaluation of patients’ opinions/perceptions (satisfaction, complaints, factors influencing the access to oral healthcare services, criteria they use to choose a dentist, the communication they have with dentists, etc.), followed by dental care providers and other staff opinions (satisfaction, elements regarding the organizational environment, the use of information systems, etc.). The evaluated studies use various instruments (questionnaires, indicators, etc.), mainly in order to improve patient satisfaction and the delivery of high-quality dental services [ 44 , 67 , 83 ]. Another important and very common category of interventions, mentioned by 35 studies, are the ones that propose the implementation and/or the assessment of quality improvement programs, such as: implementation of a pay-for-performance incentive program for medical personnel; developing a new model of dental record; examining the effectiveness of a quality improvement and management program consisting of a set of quality indicators for external and internal dimensions; using quality improvement methods to implement an early childhood oral health program; a program to reduce the number of patients’ failed appointments; the development and implementation of an integrated model of care using oral health practitioners and tele-dentistry; assessing the implementation of an educational program related to dental care, etc. [ 14 , 53 , 97 , 102 ]. In contrast, the least common category of interventions is the adoption of technology and digitization instruments. Considered only by eight studies, this type of intervention included: the development of a prioritization system; a screening website that improves access to care for patients and assists in the matching of patients and students; use of electronic health records; evaluating the effectiveness of a pre-play communication instrument; shifting from a time-based to an item-based fee-paying system in order to improve patient satisfaction; introduction of an automated confirmation system of appointments, etc. [ 38 , 48 , 51 , 96 ].

3.2.3. Outcomes

While reviewing the outcomes for each case, we have observed that the nature of the outcomes varies from real outcomes (24/72 of sources). While real outcomes present themselves in the form of real improvements/changes related to QMIs, the nature of most outcomes are exposed in the form of proposals, including ideas and/or recommendations as a result of QMIs’ adoption. The majority of studies had presented only proposals as a result of their intervention (48/72), while seven studies (7/72) present both real results and propositions.

By considering the qualitative (e.g., improved documentation, better care) and quantitative (results which could be measured, e.g., a two-fold increase in diagnostic and treatment services capacity) nature of the presented outcomes and the research-driven or clinical-driven nature of the QMIs, we observed two patterns. Research-driven studies had qualitative outcomes (25/29 cases), and their results were mostly in the form of ideas and recommendations (27/29), while clinical-driven sources had more quantitative outcomes (17/43 compared with 5/29), and the proportion of these studies that had real outcomes was larger than the ones that had research as their main purpose (21/43 compared to 3/29).

Another analysis we made regarding the outcomes focused on the dimension to which they refer. In this analysis, the previously identified categories for triggers/expected benefits in the context section were used, and each outcome was matched with a corresponding trigger. For most studies included in our analysis, outcomes are part of multiple categories (only 10/72 studies have reported outcomes included into a single category). Overall quality and access to oral healthcare related outcomes (proposals and real outcomes) were reported in 47 cases. Outcomes related to patient satisfaction, patient-centered oral healthcare, and patient–provider relationship are present in 40/72 sources. Outcomes concerning oral healthcare service quality were revealed in 29/72 sources. Outcomes related to the improved professional practice and organizational activities have been identified in 35/72 sources. Finally, business outcomes were present in only 12/72 sources.

4. Mechanisms and Discussion

The main contribution provided by our systematic literature review, performed through the use of CIMO framework, is the highlight of the explanatory mechanisms for the phenomenon of QMIs. Regarding QMIs’ adoption (as presented by the 72 analyzed sources), these can be explained considering two perspectives: one related to the triggers of the interventions (in this case, two mechanisms being observed: research-driven and clinical-driven QMIs) and another that focuses on the nature of the intervention (in this case, five design propositions or mechanisms being observed, as further described).

4.1. Mechanisms

By considering QMIs’ nature, we have identified QMIs that cover different areas regarding quality. These areas are similar to the five-stages framework, which explains small and medium enterprises’ approaches of quality management proposed by Yang [ 108 ]: product quality (product related quality control and process inspection practices), process quality (process standardization practices), system quality (quality management system such as ISO practices), total quality (much emphasis is given to customer focus and a quality culture across the organization), and business quality (quality becomes a matter of business strategy, being related to strategic management, human resource management, or business performance). In our case, the mechanisms cover five areas: internal processes, patient satisfaction, service quality, overall quality, and business outcomes.

Internal processes improvement is the second most frequently encountered mechanism for QMIs (24 cases). The context ranges from the desire to improve the professional practice and the overall practice of dental providers, to reshape dentist practice patterns, and improve the level of knowledge among dentists. The most encountered category of interventions was the implementation and assessment of quality improvement projects, programs, and methods, due to the fact that this mechanism is more focused and specialized on clinical activities and management practices, and it included specific and unique initiatives that improve their practices [ 14 , 45 , 64 ]. The majority of outcomes in this category are qualitative and imply ideas and recommendations for improving dental quality management and professional practices. An important part of outcomes is represented by qualitative and quantitative real outcomes, such as improved documentation [ 35 ], reduced waiting list [ 48 ], improved work environment [ 89 ], enhancement of interdisciplinary collaboration [ 66 ], and reduction in number of missed appointments [ 97 ].

Patient satisfaction is the most frequently encountered mechanism (25 cases). In this case, the context categories involve the need to increase patient satisfaction, more patient-centered oral health care, and the improvement of the patient–provider relationship. The interventions involve three categories of initiatives, and some clinics propose a combination of more initiatives [ 82 , 86 ]: evaluation of patients’ satisfaction using various instruments (questionnaires, indicators, etc.), the use of quality improvement programs, and technology and digitization instruments. In this case, the majority of outcomes are qualitative, patient–customer-focused proposals, mainly emphasizing means to improve patient satisfaction and patient–provider relationship. Nevertheless, some studies present real quantitative and qualitative patient satisfaction improvements [ 63 , 65 ].

Service quality mechanism is encountered in 11 cases. The context, in this case, refers to some particular triggers: the improvement in the quality of delivered healthcare services, the increase in the overall supply of available dental services, and increasing and encouraging utilization of dental health services. The main intervention here is the evaluation of patients’ opinions, using questionnaires and other instruments, considering the importance of patients’ expectations in achieving and ensuring good service quality. The majority of outcomes are represented by qualitative outcomes in the form of proposals to enhance dental service quality. Additionally, there are some qualitative and quantitative real outcomes focused on services improvements [ 51 ].

Overall quality is the third most frequently encountered mechanism (13 cases). The context refers to some expected benefits/triggers that focus on the improvement of the overall quality of care and health outcomes, the improved access to oral healthcare, and the reduction in disparities in oral healthcare. The main intervention in this category implies quality improvement programs [ 55 , 93 , 103 ], and only a few cases refer to the evaluation of patients and dental care providers’ opinions using various instruments (questionnaires, indicators, etc.) regarding the quality of care [ 70 , 71 ]. As opposed to the above mechanisms, in this category (overall quality) the majority of outcomes are quantitative real outcomes, such as oral healthcare quality and patients’ care improvements [ 50 , 102 ].

Finally, the business outcomes represent the least frequent mechanism, recognized in nine cases. The context, in terms of the business expected benefits, vary from the desire to improve clinical efficiency, to increased revenues and profits, cost savings, financial stability, and increased number of patients. Although the most encountered type of intervention is represented by unique and specific quality improvement programs, an important role is also attributed to technology and digitization tools such as introducing an automated appointment confirmation system [ 38 ]. Similar to the previous mechanism, the most encountered category of outcomes is quantitative real outcomes, mainly increased number of patients [ 37 ], increased efficiency due to broken appointments rates reduction [ 38 , 97 ], costs savings [ 97 ], financial stability, and increased revenues [ 103 ].

4.2. Discussion

QMIs adoption by dental clinics is performed in the context of research projects or is clinically driven. By considering the 72 sources included in the current review, the existence of these two approaches, with research-driven interventions in 29 studies and clinical-driven interventions in 43 studies, explains the variation observed especially when we consider real outcomes and proposed outcomes that derive from these interventions. However, the proposed mechanisms that explain QMIs’ adoption are similar to maturity models presented by the literature [ 108 ]. Based on our analysis, it can be observed that QMIs can have narrow internal focus, such as improvement of processes, but are mainly externally driven (patient satisfaction and service improvements). Larger focuses, such as overall quality and business outcome mechanisms, have been also identified. The five mechanisms explain the evolutionary nature of quality adoption in any organization, which usually starts from simple internal processes improvements and later develops a customer focus (patient satisfaction and service quality in our case), followed by a quality management system focus, and finally, a business impact focus, similar to the self-assessment tool for SMEs created by Sturkenboom et al. [ 109 ]. The main reasons for not passing to the more evolved stages are the lack of knowledge and resources [ 110 ], or in the case of dental clinics, the lack of a generally accepted definition and measurement tools for oral healthcare quality management [ 11 , 15 , 16 ].

While comparing the initial triggers and expected benefits when adopting QMIs, it can be observed that the number of outcomes exceeds the number of triggers, especially while considering the proposals associated with QMIs. Supplementary outcomes were identified in relation to the initial proposed context (based on our counting a 14% increase was observed), which can suggest either insufficient planning of the intervention, or external factors driving to other results related to specific QMIs.

Additionally, considering the two areas—research and clinical-driven cases—the most encountered category of clinical-driven studies outcomes is qualitative and involves proposals for future overall quality of dental care improvements, while the research-driven cases mainly provide proposals for patient satisfaction and patient–provider relationship improvements, as well as proposals for improving the overall quality of dental care. Moreover, besides the service quality mechanism, the rest of our mechanisms were observed mostly in clinical-driven studies, probably because the clinical real interests are mainly related to internal processes, patient satisfaction, overall quality improvements, and obtaining better business outcomes. Almost a third of interventions had a research purpose only, without impacting the actual quality of the system. However, the majority of studies were initiated internally based on real needs.

5. Conclusions

We have reviewed 72 papers in order to observe how quality management initiatives are implemented by dental clinics. Five design propositions or mechanisms were observed, ranging from overall quality to patient satisfaction, service quality, internal processes improvement, and business outcomes. The main focus of quality management in this field is related to patients’ satisfaction, followed by process improvements, a balance between internal and external-driven quality management initiatives being observed. It is obvious that more systemic approaches are required [ 14 ], business outcomes being targeted at a low level. Dental clinics’ organizational capabilities, as quality management is also defined, should be considered towards the technical capabilities as an important area of oral healthcare. Although about twenty years have passed, the four core properties of successful quality-improvement work proposed by Ferlie and Shortell [ 1 ] (leadership, culture, teams, and technologies) are still insufficiently implemented in the field of dental care.

Considering the five mechanisms identified in our study, it has become clear that dental clinics managers should perform a detailed analysis on the fitness of a specific QMI for their organizational context. Depending on internal needs, proper QMIs should be selected. Similar to the initiatives implemented in healthcare in general (ex. [ 24 ]), the implementation of different quality management initiatives should be well planned and communicated across the organization, otherwise the results of the initiatives could be different than the ones initially targeted. The development of more complex quality management initiatives that have, as a goal, the improvement of business outcomes should also be considered by dental clinic managers, since patient satisfaction and process improvements are important as long as they are linked to more customers and increased financial benefits.

Acknowledgments

This work was supported by the grant Partnership for the transfer of knowledge in biogenomics applications in oncology and related fields–BIOGENONCO, Project co-financed by FEDR through Competitiveness Operational Programme 2014–2020, contract No. 10/01.09.2016.

Author Contributions

Conceptualization, E.L.C.; methodology, E.L.C. and D.M.C.; validation, E.L.C., B.F.C., and D.M.C.; formal analysis, E.L.C. and D.M.C.; investigation, E.L.C., B.F.C., and D.M.C.; resources, E.L.C., B.F.C., and D.M.C.; data curation, E.L.C.; writing—original draft preparation, E.L.C.; writing—review and editing, E.L.C., B.F.C., and D.M.C.; visualization, D.M.C.; supervision, E.L.C.; project administration, E.L.C.; funding acquisition, E.L.C., B.F.C., and D.M.C. All authors have read and agreed to the published version of the manuscript.

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Conflicts of interest.

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

A systematic literature review of empirical research on quality requirements

• Original Article
• Open access
• Published: 08 February 2022
• Volume 27 , pages 249–271, ( 2022 )

Cite this article

You have full access to this open access article

• Thomas Olsson   ORCID: orcid.org/0000-0002-2933-1925 1 ,
• Séverine Sentilles   ORCID: orcid.org/0000-0003-0165-3743 2 &
• Efi Papatheocharous   ORCID: orcid.org/0000-0002-5157-8131 1  

6236 Accesses

12 Citations

3 Altmetric

Explore all metrics

Quality requirements deal with how well a product should perform the intended functionality, such as start-up time and learnability. Researchers argue they are important and at the same time studies indicate there are deficiencies in practice. Our goal is to review the state of evidence for quality requirements. We want to understand the empirical research on quality requirements topics as well as evaluations of quality requirements solutions. We used a hybrid method for our systematic literature review. We defined a start set based on two literature reviews combined with a keyword-based search from selected publication venues. We snowballed based on the start set. We screened 530 papers and included 84 papers in our review. Case study method is the most common (43), followed by surveys (15) and tests (13). We found no replication studies. The two most commonly studied themes are (1) differentiating characteristics of quality requirements compared to other types of requirements, (2) the importance and prevalence of quality requirements. Quality models, QUPER, and the NFR method are evaluated in several studies, with positive indications. Goal modeling is the only modeling approach evaluated. However, all studies are small scale and long-term costs and impact are not studied. We conclude that more research is needed as empirical research on quality requirements is not increasing at the same rate as software engineering research in general. We see a gap between research and practice. The solutions proposed are usually evaluated in an academic context and surveys on quality requirements in industry indicate unsystematic handling of quality requirements.

Similar content being viewed by others

Requirements quality research: a harmonized theory, evaluation, and roadmap

Empirical research on requirements quality: a systematic mapping study

Quality in model-driven engineering: a tertiary study, explore related subjects.

• Artificial Intelligence

Avoid common mistakes on your manuscript.

1 Introduction

Quality requirements—also known as non-functional requirements—are requirements related to how well a product or service is supposed to perform the intended functionality [ 48 ]. Examples are start-up time, access control, and learnability [ 56 ]. Researchers have long argued the importance of quality requirements [ 39 , 68 , 82 ]. However, to what extent have problems and challenges with quality requirements been studied empirically? A recent systematic mapping study identified quality requirements as one of the emergent areas of empirical research [ 7 ]. There are several proposals over the years for how to deal with quality requirements, e.g., the NFR method [ 78 ], QUPER [ 87 ], quality models [ 37 ], and i* [ 107 ]. However, to what extent have they been empirically validated? We present a systematic literature review of empirical studies on problems and challenges as well as validated techniques and methods for quality requirements engineering.

Ambreen et al. conducted a systematic mapping study on empirical research in requirements engineering [ 7 ], published in 2018. They found 270 primary studies where 36 papers were categorized as research on quality requirements. They concluded that empirical research on quality requirements is an emerging area within requirements engineering. Berntsson Svensson et al. carried out a systematic mapping study on empirical studies on quality requirements [ 15 ], published in 2010. They found 18 primary empirical studies on quality requirements. They concluded that there is a lack of unified view and reliable empirical evidence, for example, through replications and that there is a lack of empirical work on prioritization in particular. In our study, we follow up on the systematic mapping study om Ambreen et al. [ 7 ] by performing a systematic literature review in one of the highlighted areas. Our study complements Berntsson Svensson et al. study from 2010 by performing a similar systematic literature review 10 years later and by methodologically also using a snowball approach.

There exist several definitions of quality requirements as well as names [ 48 ]. Glinz defines a non-functional requirement as an attribute (such as performance or security) or a constraint on the system. The two prevalent terms are quality requirements and non-functional requirements. Both are used roughly as much and usually mean approximately the same thing. The ISO25010 defines quality in use as to whether the solution fulfills the goals with effectiveness, efficiency, freedom from risk, and satisfaction [ 56 ]. Eckhardt et al. analyzed 530 quality requirements and found that they described a behavior—essentially a function [ 40 ]. Hence, the term non-functional might be counter-intuitive. We use the term quality requirements in this paper. In layman’s terms, we mean a quality requirement expresses how well a solution should execute an intended function, as opposed to functional requirements which express what the solution should perform. Furthermore, conceptually, we use the definition from Glinz [ 48 ] and the sub-characteristics of ISO25010 as the main refinement of quality requirements [ 56 ].

We want to understand from primary studies (1) what are the problems and challenges with quality requirements as identified through empirical studies, and (2) which quality requirements solutions have been empirically validated. We are motivated by addressing problems with quality requirements in practice and understanding why quality requirements is still, after decades of research, often reported as a troublesome area of software engineering in practice. Hence, we study which are the direct observations and experience with quality requirements. We define the following research questions for our systematic literature review:

Which empirical methods are used to study quality requirements?

What are the problems and challenges for quality requirements identified by empirical studies?

Which quality requirements solution proposals have been empirically validated?

We study quality requirements in general and therefore exclude papers focusing on specific aspects, e.g., on safety or user experience.

We summarize the related literature reviews in Sect.  2 . We describe the hybrid method we used for our systematic literature review in Sect.  3 . Section  4 elaborates on the findings from screening of 530 papers to finally include 84 papers from the years 1995 to 2019. We discuss the results and threats to validity in Sect.  5 ; empirical studies on quality requirements are—in relative terms—less common than other types of requirements engineering papers, there is a lack of longitudinal studies of quality requirements topics, we found very few replications. We conclude the paper in Sect.  6 with a reflection that there seems to be a divide between solutions proposed in an academic setting and the challenges and needs of practitioners.

2 Related work

A recent systematic mapping study on empirical studies on requirements engineering states that quality requirements are “by far the most active among these emerging research areas” [ 7 ]. They classified 36 papers of the 270 they included as papers in the quality requirements area. In their mapping, they identify security and usability as the most common topics. These results are similar to that of Ouhbi et al. systematic mapping study from 2013 [ 81 ]. However, they had slightly different keywords in their search, including also studies on quality in the requirements engineering area, which is not necessarily the same as quality requirements. A systematic mapping study is suggested for a broader area whereas a systematic literature review for a narrower area which is studies in more depth [ 63 ]. To our knowledge, there are no recent systematic literature reviews on quality requirements.

Berntsson Svensson et al. performed a systematic literature review on empirical studies on managing quality requirements in 2010 [ 15 ]. They identified 18 primary studies. They classified 12 out of the 18 primary studies as case studies, three as experiments, two as surveys, and one as a mix of survey and experiment. They classified only four of the 18 studies as properly handling validity threats systematically. Their results indicate that there is a lack of replications and multiple studies on the same or similar phenomena. However, they identify a dichotomy between two views; those who argue that quality requirements need special treatment and others who argue quality requirements need to be handled at the same time as other requirements. Furthermore, they identify a lack of studies on prioritization of quality requirements. Berntsson Svensson et al. limited their systematic literature review to studies containing the keyword “software,” whereas we did not in our study. Furthermore, Berntsson Svensson et al. performed a keyword-based literature search with a number of keywords required to be present in the search set. We used a hybrid approach and relied on snowballing instead of strict keywords. Lastly, we used Ivarsson and Gorschek [ 58 ] for rigor, which entailed stricter inclusion criteria, i.e., as a result we did not include all studies from Berntsson Svensson et al. This, in combination with performing the study 10 years afterward, means we complement Berntsson Svensson both in terms of the method as well as studied period.

Alsaqaf et al. could not find any empirical studies on quality requirements in their 2017 systematic literature review on quality requirements in large-scale agile projects [ 5 ]. They included studies on agile practices and requirements in general. Hence, their scope does not overlap significantly with ours. They found, however, 12 challenges to quality requirements in an agile context. For example, a focus on delivering functionality at the expense of architecture flexibility, difficulties in documenting quality requirements in user stories, and late validation of quality requirements. We do not explicitly focus on agile practices. Hence, there is a small overlap between their study and ours.

We designed a systematic literature review using a hybrid method [ 77 ]. The hybrid method combines a keyword-based search, typical of a systematic literature review [ 63 ], to define a start set and a snowball method [ 105 ] to systematically find relevant papers. We base our study on two literature reviews [ 7 , 15 ], which we complement in a systematic way. The overall process is found in Fig.  1 .

3.1 Approach

We decided to use a hybrid approach for our literature review [ 77 ]. A standard keyword-based systematic literature review [ 63 ] can result in a very large set of papers to review if keywords are not restrictive. On the other hand, having too restrictive keywords can result in a too-small set of papers. A snowball approach [ 105 ], on the other hand, is sensitive to the start set. If the studies are published in different communities not referencing each other, there is a risk of not finding relevant papers if the start set is limited to one community. Hence, we used a hybrid method where we combine the results from a systematic mapping study and a systematic literature review to give us one start set with a keyword-based search in the publication venues of the papers from the two review papers.

We used two different approaches to create the start sets: Start set I is based on two other literature reviews, and Start set II is created through a keyword-based search in relevant publication venues. The two start sets are combined and snowballed on to arrive at the final set of included papers. The numbers between the steps in each set are the number of references within that set. The numbers between the sets are the total number of references included in the final set

Start set I

We defined Start set I for our systematic literature review by using a systematic mapping study on empirical evidence for requirements engineering in general [ 7 ] from 2018 and a systematic literature review from 2010 [ 15 ] with similar research questions as in our paper.

The systematic literature review from 2010 by Berntsson Svensson et al. includes 18 primary studies [ 15 ]. However, we have different inclusion criteria (see Sect.  3.2 ). Hence, not all the references are included. In our final set, we included 10 of the 18 studies.

The systematic mapping by Ambreem et al. from 2018 looks at empirical evidence in general for requirements engineering [ 7 ]. They included 270 primary studies. However, there are some duplicates in their list. They classified 36 papers to be in the quality requirements area. However, there is an overlap with the Berntsson Svensson et al. review [ 15 ]. When we remove the already included papers from Berntsson Svensson et al., we reviewed 24 from Ambreem et al. and in the end included 4 of them.

Start set II

To complement start set I, we also performed a keyword-based search. We have slightly different research questions than the two papers in the Start set I. Therefore, our search string is slightly different than that of Ambreen et al. and Berntsson Svensson et al. Also, the most recent references in Start set I are from 2014, i.e., five years before we performed our search. Hence, we also fill the gap of the papers published since 2014. We include all studies, not just studies from 2014 and onward, as our method and research questions are slightly different.

We used the most frequent and highly ranked publication venues from Start set I to limit our search but still have a relevant scope. Table  1 summarizes the included conferences and journals. Even though the publication venues included are not an exhaustive list of applicable venues, we believe they are representative venues that are likely to include most communities and thereby reducing the risk with a snowball approach of missing relevant publications, as intended with the hybrid method.

We used Scopus to search. The search was performed in September 2019. Table  2 outlines the components of the search string. The title and abstract were included in the search and only papers that include both the keyword for quality requirements and the keywords for empirical research.

Snowballing

The last step in our hybrid systematic review [ 77 ] is the snowballing of the start set papers. We snowballed on the extended start set—the combination of Start set I and Start set II—to get to our final set of papers (cf. Fig.  1 ). In a snowball approach, both references in the paper (backward references) and papers referring to the paper (forward references) were screened [ 105 ]. We used Google Scholar to find forward references.

3.2 Planning

We arrived at the following inclusion criteria, in a discussion among the researchers and based on related work:

The paper should be on quality requirements or have quality requirements as a central result.

There should be empirical results with a well-defined method and validity section, not just an example or anecdotal experience.

Papers should be written in English.

The papers should be peer-reviewed.

The papers should be primary studies.

Conference or journal should be listed in reference ranking such as SJR.

Similarly, we defined our exclusion criteria as:

Literature reviews, meta-studies, etc.,—secondary studies—are excluded.

If a conference paper is extended into a journal version, we only include the journal version.

We include papers only once, i.e., duplicates are removed throughout the process.

Papers focusing on only specific aspect(s) (security, sustainability, etc.) are excluded.

All researchers were involved in the screening and classification process, even though the primary researcher performed the bulk of the work. The screening and classification were performed as follows:

Screen based on title and/or abstract.

We performed a full read when at least one researcher wanted to include the paper from the screening step.

Papers were classified according to the review protocol, see Sect.  3.3 . This was performed by the primary researcher and validated by another researcher.

To ensure reliability in the inclusion of papers and coding, the process was performed iteratively according to the sets.

For Start set I, all references from the systematic literature review [ 15 ] and systematic mapping study [ 7 ] were screened by two or three researchers. We only used the title in the screening step for Start set I. Full read and classification were performed by two or three researchers.

For Start set II, the screening was primarily performed by the primary researcher but with frequent alignment with at least one more researcher to ensure consistent screening—both on title and abstract. Similarly for the full read and classification of the papers. Specifically, we paid extra attention to which papers to exclude to ensure we did not exclude relevant papers.

The primary researcher performed the snowballing. We screened on title only for backward and forward snowballing. We included borderline cases, to ensure we did not miss any relevant references.

The full read and classification were primarily performed by the primary researcher for Start set II and the Snowballing set. A sample of the papers was read by another researcher to improve validity in addition to those cases already reviewed by more than one researcher.

The combined number of primary studies from the systematic literature review [ 15 ] and systematic mapping study [ 7 ] are 288 in Start set I. However, there is an overlap between the two studies and there are some duplicates in the Ambreen et al. paper [ 7 ]. In the end, we had 274 unique papers in Start set I. After the screening, 41 papers remained. After the full read, additional papers were excluded resulting in 14 papers in the Start set I.

Our search in Start Set II resulted in 190 papers. A total of 173 papers remained after removing duplicate papers and papers already included in Start set I. After the screening and full read, the final Start set II was 23. Hence, the extended start set (combining Start set I and Start set II) together resulted in the screening of 447 papers and the inclusion of 37 papers.

The snowball process was repeated until no new papers are found. We iterated 2 times—denoted I1 and I2 in Fig.  1 . In iteration 1, we reviewed 77 papers and included 43. In iteration 2, we reviewed 6 papers and included 4. This resulted in a total of 84 papers included and 530 papers reviewed.

3.3 Classification and review protocol

We developed the review protocol based on the systematic literature review [ 15 ] and systematic mapping study [ 7 ], the methodology papers [ 63 , 105 ], and our research questions. The main items in our protocol are:

Type of empirical study according to Wieringa et al. [ 104 ]. As we are focusing on empirical studies, we use the evaluation type—investigations of quality requirements practices in a real setting—,validation type—investigations of solution proposals before they are implemented in a real setting—,or experience type—studies where the researchers are taking a more part in the study, not just observing.

Method used in the papers. We found the following primary methods used: Experiment, test, case study, survey, and action research.

Analysis of rigor according to Ivarsson and Gorschek [ 58 ].

Thematic analysis of the papers—in an initial analysis based on the author keyword and in later iterations further refined and grouped during the analysis process.

We used a spreadsheet for documentation of the classification and review notes. The classification scheme evolved iteratively (see Sect.  3.2 ) as we included more papers. The initial themes were documented in the review process. In the analysis phase, the initial themes were used for an initial grouping of the papers. The themes were aligned and grouped in the analysis process of the papers, which included a number of meetings and iterative reviews of the results. The final themes which we used for the papers are the results of the iterative analysis process, primarily performed by the first and third researcher.

3.4 Validity

All cases where there were uncertainties whether to include a paper—both in the screening step and the full read step—or on the classification were reviewed by at least two researchers. Furthermore, to ensure consistent use of the inclusion and exclusion criterion as well as the classification we also sampled and reviewed papers that had only been screened or reviewed by only one researcher.

We used Scopus for Start set II. We confirmed that all journals and conferences selected from Start set I were found in Scopus. However, REFSQ was only indexed from 2005 and onward. However, we do not see this as a problem as we are snowballing and the papers that are missing from the Scopus search due to this, should appear in the results through the snowballing process.

We used Google Scholar in the snowballing. This is recommended [ 105 ] and usually gives the most complete results.

A hybrid search strategy can be sensitive to starting conditions, as pointed out by Mourão et al. [ 77 ]. However, their results indicate that the strategy can produce similar results as a standard systematic literature review. We carefully selected the systematic literature review and the systematic mapping study as Start set I and extended it with a keyword-based search for selected forums in Start set II. Hence, we believe the extended start set on which we snowballed is likely to be sufficient to ensure a good result when complemented with the snowball approach.

4 Analysis and results

The screening and reading of the papers in Start set I was performed in August and September 2019. The keyword-based search for Start set II was performed in September 2019. The snowballing was subsequently performed in October and November. In total, 530 papers are screened, of which 194 papers are read in full. This resulted in including 84 papers, from 1995 to 2019—see Fig.  2 .

An overview of papers included—publication type and year of publication

4.1 RQ1 Which empirical methods are used to study quality requirements?

The type of studies performed is found in Table  3 —categorized according to Wieringa et al. [ 104 ]. We differentiate between two types of validations: experiments involving human subjects and tests of algorithms on a data set. For the latter, the authors either report experiment or case study, whereas we call them test. The evaluations we found are either performed as case studies or surveys. Lastly, we found three papers that used action research—categorized as experience in Table  3 . It should be noted that the authors of the action research papers did not themselves explicitly say they performed an action research study. However, when we classified the papers, it is quite clear that, according to Wieringa et al. [ 104 ], they are in the action research category.

Case studies in an industry setting are the most common (35 of 84), followed by surveys in industry (14 of 84) and test in academic settings (13 of 84). This indicates that research on quality requirements is applied and evidence is primarily of individual case studies rather than through validation in laboratory settings. Case studies seem to have similar popularity over time, see Fig.  3 . We speculate that since requirements engineering in general as well as quality requirements in particular is a human-intensive activity, there are not so many clear cause–effect relationships to study in a rigorous experiment. Rather, it is more important to study practices in realistic settings. However, there are only three longitudinal studies.

Tests are, in contrast to the case studies, primarily performed in an academic setting, which is not necessarily representative in terms of scale and artifacts. The papers are published from 2010 to 2019—one exception, published in 2007, see Fig.  3 . One explanation might be the developments in computing driving the trend to use large data sets.

We found only one study on open source, see Table  3 , which is also longitudinal. We speculate that requirements engineering is sometimes seen as a business activity where someone other than the developers decides what should be implemented. In open-source projects, there is often a delegated culture where there is no clear product manager or similar deciding what to do, albeit there can be influential individuals such as the originator or core team member. We believe this entails that quality requirements engineering is different in open-source projects than when managed within an organization. It would be interesting to see if this hypothesis holds for requirements engineering in general and not just quality requirements. We believe, however, that by studying forums, issue management, and reviews that open-source projects are an untapped resource for quality requirements research.

An overview of papers included—method and accumulated number of publications per year

We classify the studies according to rigor, as proposed by Ivarsson and Gorschek [ 58 ]. We assess the design and validity rigor. Table  4 presents our evaluation of design and validity rigor in the papers. An inclusion criterion is that there should be an empirical study, not just examples or anecdotes. Hence, it is not surprising that overall studies score well in our rigor assessment.

High rigor is important for validations studies—to allow for replications—which is also the case for 11 out of 23 studies. The number increases to 13 if we include papers with a rigor score 0.5 for both design and validity and to 19 if we focus solely on the design rigor. Interestingly, we found no replication studies. Furthermore, the number of studies on a single (similar) approach or solution is in general low. We speculate that the originators of a solution have an interest in performing empirical studies on their solution. However, it seems unusual that practitioners or empiricists with no connection to the original solution or approach try to apply it. Furthermore, we also speculate that academic quality requirements research is not addressing critical topics for industry as there seems not to be an interest in applying and learning more about them. This implies that the research on quality requirements might need to better understand what are the real issues facing software developing organizations in terms of quality requirements.

The validity part of rigor is also important for evaluations and experience papers. Strict replications are typically not possible. However, understanding the contextual factors and validity are key in interpreting the results and assessing their applicability in other cases and contexts. 22 of the 58 evaluation and 1 of the 3 experience papers do not have a well-described validity section (rigor score 0), and 10 evaluation and 1 experience paper have a low score (rigor score 0.5). Hence, we conclude that the overall strength of evidence is weak.

4.1.1 Validations

Experiments are, in general, the most rigorous type of empirical study with the most control. However, it is difficult to scale to a realistic scenario. We found four experiments validating quality requirements with human subjects, see Table  5 .

We note that all experiments are performed with students—at varying academic levels. This might very well be appropriate for experiments [ 54 ]. We notice that there are only four experiments, which might be justified by: (1) Experiments as a method is not well accepted nor understood in the community. (2) Scale and context are key factors for applied fields such as requirements engineering, making it more challenging to design relevant experiments.

Several empirical studies study methods or tools by applying them to a data set or document set. We categorize those as tests, see Table  6 . We found three themes for tests.

Automatic analysis—The aim is to evaluate an algorithm or statistical method to automatically analyze a text, usually a requirements document.

Tool—The aim is to evaluate a tool specifically.

Runtime analysis—Evaluating the degree of satisfaction of quality requirements during runtime.

Tests are also fairly rigorous in that it is possible to control the study parameters well. It can also be possible to perform realistic studies, representative of real scenarios. The challenge is often to attain data that is representative. The data set is described in Table  7 .

The most commonly used data set is the DePaul07 data set [ 27 ]. It consists of 15 annotated specifications from student projects at DePaul University from 2007. This data set consists of requirements specification—annotated to functional and quality requirements as well as the type of quality requirement—from student projects.

There are few examples where data from commercial projects have been used. The data do not seem to be available for use by other researchers. There are examples where data from public organizations—such as government agencies—are available and used, e.g., the EU procurement specification, see Table  7 .

The most common type of data is a traditional requirements document, written in structured text. There are also a couple of instances where use case documents are used. For non-requirements specific artifacts, manuals, data use agreements, request for proposals (RFPs), and app reviews are used. From the papers in this systematic literature view, artifacts such as backlogs, feature lists, roadmaps, pull requests, or test documents do not seem to have been included.

4.1.2 Evaluations

It is usually not possible to have the same rigorous control of all study parameters in case studies [ 38 ]. However, it is often easier to have realistic scenarios, more relevant for a practical setting. We found both case studies performed in an industry context with practitioners as well as in an academic context with primarily students at different academic levels. We found 43 papers presenting case study reports on quality requirements, see Fig.  4 (all details can be found in Table  10 ). We separate case studies that explicitly evaluate a specific tool, method, technique, framework, etc., and exploratory case studies aiming to understand a specific context rather than evaluating something specific.

Case studies sometimes study a specific object, e.g., a tool or method, see Table  10 . We found 25 case studies explicitly studying a particular object. Two objects are evaluated more than once, otherwise just one case study per object. We found no longitudinal cases; hence, the case studies are executed at one point in time and not followed up at a later time. The QUPER method is studied in several case studies in several different contexts (see Table  10 ). There are several case studies for the NFR method; however, it seems the context is similar or the same in most of the cases (row 2).

Overview of the case studies on quality requirements—43 papers of the 84 included in this literature review. Scale refers to the context of the case study—small: sampling parts of the context (e.g., one part of a company) or is overall a smaller context (e.g., example system), medium: sampling a significant part of the context or a larger example system, large: sampling all significant parts of a context of an actual system (not example or made up). The context is also classified according to where the case studies are executed. Academic means primarily by students (at some academic level). Mixed means the case studies are executed in both an academic and industry context. For details, please see Table  10

We found 18 exploratory case studies on quality requirements where a specific object wasn’t the focus, see Table  10 . Rather, the goal is to understand a particular theme of quality requirements. Eight case studies want to understand details of quality requirements, e.g., the prevalence of a specific type of quality requirement or what happens in the lifecycle of a project. Five case studies have studied the process around quality requirements, two studies on sources of quality requirements (in particular app reviews), two studies in particular on developers’ view on quality requirements (specifically using StackOverflow), and lastly one study on metric related to quality requirements. We found two longitudinal case studies.

The goal of a survey is to understand a broader context without performing any intervention [ 38 ]. Surveys can be used either very early in the research process before there is a theory to find interesting hypotheses or late in the process to understand the prevalence of a theme from a theory in a certain population. We found 15 surveys, see Table  8 ; 5 interviews, 9 questionnaires, one both. The goals of the surveys are a mix of understanding practices around the engineering of quality requirements and understanding actual quality requirements as such.

Overall, the surveys we found are small in terms of the sample of the population. In most cases, they do not report from which population they sample. The most common theme is the importance of quality requirements and specific sub-characteristics—typically according to ISO9126 [ 57 ] or ISO25010 [ 56 ]. However, we cannot draw any conclusions as sampling is not systematic and the population unclear. We believe it is not realistic to systematically sample any population and achieve a statistically significant result on how important quality requirements are nor which sub-characteristics are more or less important. We speculate that, besides the sampling challenge, the variance among organizations and point in time will likely be large, making the practical implications of such studies of questionable value.

4.1.3 Experience

In action research, the researchers are more active and part of the work than, e.g., in a case study [ 38 ]. Whereas a case study does not necessarily evaluate a specific object, action research typically reports some kind of intervention where something is changed or performed. We found 3 papers we classify as action research types of experience papers [ 104 ], see Table  9 .

Two of the studies are performed at one point in time [ 2 , 71 ]. One study is longitudinal, describing the changes to the processes and practices around quality requirements over several years [ 75 ]. Interestingly, all three studies directly refer to ISO9126 [ 57 ] or ISO25010 [ 56 ].

4.2 RQ2 What are the problems and challenges for quality requirements identified by empirical studies?

We have grouped the studies on quality requirements themes thematically to analyze the problems and challenges that are identified in the empirical studies. The groups are developed iteratively among the researchers, initially from the author keywords in the included papers and then iteratively refined.

4.2.1 Quality requirements and other requirements

There is an academic debate on what quality requirements are and what they should be called [ 48 ]. We found a study indicating that quality requirements—sometimes called non-functional requirements—are functional or behavioral [ 41 ]. This is in line with other studies that report that a mix of requirements type is common [ 14 ]. Two studies find that architects address quality requirements the same way as other requirements [ 33 , 84 ], also confirmed in other surveys [ 83 ]. However, there are also research studies indicating a varying prevalence and explicitness than other requirements [ 14 , 41 , 49 , 80 , 90 ]. We interpret the current state of evidence to be unclear on the handling of quality requirements. We speculate that the answers to opinion surveys might be biased towards the expected “right” answer—as expected by the researcher—rather than the actual viewpoint of the respondent.

4.2.2 Importance and prevalence of quality requirements

Many papers present results related to the importance and prevalence of quality requirements—or sub-characteristics of quality requirements. Four papers present results from artifact analysis [ 14 , 22 , 80 , 90 ]. We found eight personal opinion survey papers  [ 8 , 9 , 12 , 23 , 34 , 35 , 47 , 97 ]. Similarly, a list of quality requirements types is developed through a survey for service-oriented applications [ 11 ]. Furthermore, we found three papers analyzing app store reviews [ 50 , 60 , 72 ] and two papers developer’s discussions on StackOverflow [ 1 , 110 ] and one paper studying 8 open-source projects communication [ 43 ]. The individual papers do not present statistical tests or variance measures. Furthermore, we found no papers elaborating on a rationale for why the distribution of sub-characteristics of quality requirements are more or less prevalent or seems as important by the subjects. We hypothesize that the importance of different quality requirements types varies over time, domain, and with personal opinion. This implies that there is no general answer to the importance of different quality requirements types. Rather, we believe it is important to adapt the quality requirements activities—such as planning and prioritization—to the specific context rather than to use predefined lists.

4.2.3 Specification of quality requirements

We found three case study papers reporting on artifact analysis of realistic requirements documents [ 14 , 41 , 90 ]. The practice seems to vary in how quality requirements are written; quantification, style, etc. One paper reporting on a scope decision database analysis [ 80 ]. The prevalence of quality requirements features is low and varies over time. Two interview surveys, furthermore, find quantification varies for the different cases as well as for the different quality requirements types [ 33 , 97 ]. Four surveys indicate that quality requirements are often poorly documented and without template [ 9 , 20 , 23 , 108 ]. Overall, the studies mostly report the usage of informal specification techniques (structured text) rather than specific modeling notations.

4.2.4 Roles perspective

Different roles—for example, project manager, architect, product manager—view and work with quality requirements differently. Two interview surveys report that architects are often involved in the elicitation and definition of quality requirements [ 9 , 33 ]. Furthermore, the clients or customers—in a bespoken context—are not explicit nor active in the elicitation and definition of quality requirements [ 33 ]. We found six papers collecting opinion data on the priority of quality requirements types from a role perspective [ 8 , 9 , 35 , 51 , 61 , 97 ]. We did not find any particular trend nor general view for different roles, except that when asked subjects tend to answer that quality requirements as a topic is important and explicitly handled—albeit that there are improvement potentials. Hence, it seems to us that, again, there might not be a general answer to the importance of different quality requirements types.

One study found that architects—despite being one source of quality requirements—are not involved in the scoping [ 34 ]. Another study found that relying on external stakeholders might lead to long lead-times and incomplete quality requirements [ 80 ]. We found one study on quality requirements engineering in an agile context. They report that communication and unstated assumptions are major challenges for quality requirements [ 4 ]. Even though opinion surveys indicate that subjects—independent of roles—claim to prioritize and explicitly work with quality requirements, there are indications that implicit quality requirements engineering is common and this leads to misalignment.

We find evidence of how different roles perceive and handle quality requirements to be insufficient to draw any particular conclusions.

4.2.5 Lifecycle perspective

We found two papers presenting results related to changes over time for the prevalence of different quality requirements types. Ernst and Mylopoulos study 8 open-source project [ 43 ] and Olsson et al. study scope decisions from a company [ 80 ]. Ernst and Mylopoulos did not find any specific pattern across the 8 open-source project in terms of prioritization or scoping of quality requirements. Olsson et al. concluded that there was an increase in the number of quality-oriented features and the acceptance of quality requirements in the scope decision process later in the product lifecycle compared to early in the product lifecycle. We found one student experiment on the stability of prioritization within the release of a smaller project [ 29 ]. They conclude that interoperability and reliability are more stable in terms of project priority whereas usability and security changed priority more in the release cycle. Lastly, we found a paper presenting a study on the presence of “Not a Problem” issue reports in the defect flow compared to how precise quality requirements are written [ 53 ]. The main result is that the more precise quality requirements are written, the lower the amount of “Not a Problem” issue reports.

The number of studies is small, which makes it difficult to draw any conclusions. However, we speculate that what happens over time is also likely to vary and be context specific. We hypothesize that there might be general patterns that, for example, products early in the lifecycle tend to overlook quality requirements, whereas products later in the lifecycle tend to focus more on quality requirements. Furthermore, it might also be differences in the handling of quality requirements depending on how close to release the project is. We see these topics as relevant to study in more detail. Longitudinal studies, involving different artifacts and sources information, e.g., issue report systems, can be an interesting way forward.

4.2.6 Prioritization

We found two case studies on quality requirements prioritization [ 33 , 96 ]. Berntsson Svensson et al. conducted an interview study with product managers and project leaders [ 96 ]. They found that ad hoc prioritization and priority grouping of quality requirements are the most common. Furthermore, they found that project leaders are more systematic (55% prioritize ad hoc) compared to the product managers (73% prioritize ad hoc). Daneva et al. found in their interview study that architects are commonly involved in prioritization of quality requirements [ 33 ]. They identified ad hoc and priority grouping as the most common approach to prioritization. Daneva et al., furthermore, found that 7 out of the 20 architects they interviewed considered themselves the role that sets the priority for quality requirements.

In summary, we find there is overall a lack of understanding of quality requirements prioritization. The studies indicate the involvement of different roles, which we believe warrants further research. Furthermore, the lack of systematic prioritization seems to be in line with requirements in general and not just for quality requirements.

4.2.7 Sources of quality requirements

There can be several sources of requirements, both roles as well as artifacts. As reported before, architects are sometimes involved in the elicitation and definition of quality requirements [ 9 , 33 ]. Three studies have identified user reviews on mobile app markets as a potential source of quality requirements [ 50 , 60 , 103 ]. One study found that users are not sufficiently involved in the elicitation [ 49 ]. However, we did not find studies on, for example, usage data or customer services data as a means to elicit and analyze quality requirements.

4.3 RQ3 Which quality requirements solution proposals have been empirically validated?

Several techniques, tools, etc., have been proposed to address problem and challenges with quality requirements. The results of the evaluations or validation of different quality requirements solutions are grouped after similarity. The sections are ordered according to the number of studies.

4.3.1 Automatic analysis

One research direction which has gained popularity is different forms of automatic analysis. The idea is that a tool can be developed to support human engineers in different aspects of quality requirements engineering. All studies we found reported positive results.

We found a number of papers investigating automatic identification and classification of quality requirements from different types of sources [ 6 , 10 , 24 , 28 , 66 , 70 , 74 , 86 , 91 , 94 , 95 , 100 , 106 , 109 ]. The different papers test different algorithms and approaches on different data sets. The most commonly used data set is from a project course at DePaul University from 2007. That data set has annotated requirements (functional or quality requirements) as well as quality requirements types, see Tables  6 and 7 . Overall, the studies are executed in an academic context (11 out of 14) and all at a small scale which might not be representative for realistic commercial cases. Furthermore, it is often assumed the presence of requirements documents, which might not be the case for agile contexts.

We found two papers presenting studies of user reviews in app stores, e.g., Apple app store or Google play [ 60 , 72 ], both rigorous. Similar to other work on automatic classification, the two studies evaluated different algorithms to identify and classify quality requirements in app reviews.

We found one paper on early aspect mining [ 85 ]. The study evaluated a tool to detect quality requirements aspects in a requirements document. Based on the detection, quality requirements are suggested to the requirements engineer. Another study evaluated a use case tool with explicit quality requirements for an agile context [ 44 ]. Both studies imply feasibility but cost or amount of effort of using them in large-scale realistic cases are not studied.

We found one study on runtime adaptations of quality requirements for self-managing systems [ 42 ]. Rather than defining fixed values for trade-offs among quality requirements, quality requirements are defined as intervals that can be optimized in runtime, depending on the specific operational conditions. They test their approach on two example systems, which show better compliance when using their approach than not.

We summarize that, while there are many tests and experiments, there are few studies of realistic scale and with realistic artifacts on automatic analysis in an quality requirements context. We also find that there is a lack of costs of running the automatic analysis, such as preparation of data, needs in terms of hardware and software, and knowledge needed by an analyst. We conclude that automatic analysis shows promise in an academic setting but has yet to be studied in a realistic scale case study or action research.

4.3.2 Goal modeling

We found two experiments on different extensions of i*, validating usefulness and correctness of the extensions compared to the original i* approach [ 99 , 111 ]. The experiments are conducted in an academic setting. Both experiments conclude that the extensions are better. Based on these experiments, we cannot say anything in general about modeling of quality requirements and usefulness of i* in general.

Researchers have performed several case studies [ 26 , 30 , 31 , 32 ]. The researchers and case context are similar and all present how the NFR method and goal modeling can work in different situations. One case study evaluated a process where business models and a quality requirements catalog are used to finally build a goal model for relevant quality requirements [ 19 ]. We found one case study using goal modeling to support product family quality requirements using goal modeling [ 79 ]. All of these case studies are of low rigor both in terms of design and validity. We have found one paper describing a test of generating goal graphs from textual requirements documents [ 86 ], and another paper testing a tool for goal modeling in an agile context [ 44 ]. Both papers indicate feasibility, i.e., the techniques seem to work in their respective context.

Overall, the evidence point to that goal modeling—in various forms—can be used and does add benefits in terms of visualization and systematic reasoning. However, we have not found any realistic scale case studies on quality requirements, nor any data on effort or impact on other parts of the development. We have not found any surveys on modeling techniques used for quality requirements. Hence, we have not found evidence of the use of goal modeling in industry specifically for quality requirements. We judge the collected evidence that goal modeling does have potential benefits but they have not been evaluated in realistic scale projects with a systematic evaluation of the whole quality requirements process.

4.3.3 Quality models and ISO9126 / ISO25010

ISO9126 [ 57 ]—and the updated version in ISO25010 [ 56 ]—is used in many of the papers we found. Al-Kilidar et al. validated the usefulness of ISO9126 in an experiment with students [ 3 ]. They conclude that the standard is difficult to interpret and too general to be useful. However, the experiment has a low rigor score—0.5 for design and 0 for validity—and lacks relevant validity description.

ADEG-NFR use ISO25010 [ 56 ] as catalogue of quality requirements types [ 93 ]. The IESE NFR method also uses the ISO standard as the basis for creating checklists and quality models [ 36 ]. Sibisi and van Waveren proposes a similar approach as the IESE NFR method, using the ISO standard as a starting point in customizing checklists and quality model [ 92 ]. Two papers present evaluations of two approaches combining ISO9126 quality models with goal modeling [ 2 , 19 ]. Another paper evaluated an approach where a checklist was derived from ISO9126 to guide the elicitation [ 67 ]. Similarly, two papers evaluate workshop and brainstorming approaches to elicitation and analysis based on ISO9126, which they propose to complement with multi-stakeholder workshops [ 65 , 101 ]. Lastly, the Quamoco approach suggests connecting the abstract quality model in the ISO standard with concrete measurements [ 102 ].

We found one paper defining a catalog of quality requirements for service-oriented applications [ 11 ]. They proposed an initial list of quality requirements which was evaluated with practitioners using a questionnaire-based approach. Mohagheghi and Aparicio conducted a three-year-long project at the Norwegian Labour and Welfare Administration [ 75 ]. The aim was to improve the quality requirements. Lochmann et al. conducted a study at Siemens, Germany [ 71 ]. The business unit in question develops a traffic control system. They introduced a quality model approach to the requirements process for quality requirements. The approach is based on ISO9126 [ 57 ].

All of the approaches suggest incorporating the quality requirements specific parts into the overall requirements process as well as tailoring to the needs of the specific organization. The different approaches seem to be recognized as useful in realistic settings, leading to a more complete understanding of the quality requirements scope with a reasonable effort. It further seems as if the tailoring part is important to gain relevance and acceptance from the development organizations—especially when considering the experiment from Al-Kilidar et al. [ 3 ].

4.3.4 Prioritization and release planning

We found one method—QUPER—focused explicitly on prioritization and release planning [ 88 ]. The researchers behind the method have performed several case studies to evaluate it [ 13 , 16 , 17 , 18 ]. We also found a prototype tool evaluation [ 17 ]. This is the single most evaluated approach. QUPER is the only approach we found which is explicitly focused on prioritization and release planning.

We found one paper proposing and evaluating an approach to handle interaction and potentially conflicting priorities among quality requirements [ 46 ]. This is similar to QUARCC, which is tested in a tool evaluation [ 55 ].

We summarize that QUPER has been evaluated both with academics and with practitioners in realistic settings. However, the long-term impact of using QUPER seems not to have been studied. However, other than QUPER, we conclude that there is no strong evidence for other solutions for prioritization and release planning of quality requirements.

4.3.5 Metrics and quality models

We found two papers evaluating the connection between key metrics to measure quality requirements types and user satisfaction [ 62 , 69 ]. The results imply that quality requirements metrics—measuring the presence of quality requirements types according to ISO9126 [ 57 ] in the specifications—is correlated with user satisfaction. Hence, even though this is not a longitudinal study, there are implications that good quality requirements engineering practices might increase user satisfaction. Both studies are personal opinion surveys, which makes it difficult to evaluate causality and root-cause. Furthermore, they measured at one point in time.

We found one paper proposing and evaluating an approach to create metrics to evaluate the responses to a request for proposal (RFP) [ 89 ]. The metrics are based on recommendations from authorities and focused on process metrics rather than product metrics. They report in their case study that they could identify specifications with deficiencies in quality requirements.

We summarize that there is not a lot of evidence on the usage of metrics in connection to quality requirements. We find that the studies have identified interesting hypotheses that can be evaluated both in an academic setting through experiments or case studies as well as in real settings through case studies or action research.

4.3.6 Knowledge management

We found three papers on different aspects of knowledge management to address quality requirements engineering. Balushi et al. report on a study at the University of Manchester [ 2 ]. They applied the ElicitO framework on a project to enhance the university website. The ontology in ElicitO implements ISO9126 [ 57 ]. The MERliNN framework suggests procedures to identify and manage knowledge flows in the elicitation and analysis process [ 21 ]. One paper evaluating a tool for QUARCC and S-Cost—knowledge-based tools for handling inter-relationships among quality requirements and stakeholders [ 55 ]. All three approaches report improved completeness of quality requirements and aligned terminology among stakeholders.

We summarize that knowledge management solutions are not well studied in the quality requirements context. We also note that there are similarities between knowledge management and quality models—which is also evident as one of the studies used ISO 9126 [ 57 ].

4.3.7 Others

We found one paper evaluating MOQARE, a misuse oriented approach to find adversarial quality requirements [ 52 ]. Another paper found that creating a clearer template and instructions for how to write a specification improved not only the quality requirements but also the attitude towards quality requirements [ 59 ]. One paper evaluates a method for how to select an appropriate technique depending on the relevant quality requirements and context factors such as lifecycle phase, etc [ 25 ]. Kopczyńska et al. experimented on a template approach for eliciting quality requirements [ 64 ]. They define a template as a regular expression. The experiment is performed with students in the third year at university. They find that using templates improved completeness and overall quality of the quality requirements. However, using templates did not speed up the elicitation process. As this is a small-scale validation, more research is needed to understand how it performs in a realistic setting.

5 Discussion

The results of our systematic literature review indicate that there are many quality requirements engineering aspects that warrant further research. The small number of studies found—84 papers over 30 years—point to a lack of studies. Furthermore, it seems to us that there is a divide between academically proposed solutions and needs of practitioners.

5.1 RQ1 Which empirical methods are used to study quality requirements?

A central research question when performing a systematic literature review is to try and answer a specific question through empirical evidence from several studies [ 63 ]. There is a tendency towards more empirical studies on quality requirements in Fig.  2 . We found 1–5 papers per year in the 1990s and 4–11 papers per year in the 2010s. However, considering that the scientific community is producing more and more papers every year, the tendency to more empirical studies in quality requirements might be smaller than that of the empirical software engineering community as a whole. A naive search on Scopus for “empirical software engineering” resulted in 50–100 papers per year during the 1990s and 450–800 papers per year in the 2010s. When we further limit the result to “requirements,” we end up with 1–20 in the 1990s and 100–250 papers per year in the 2010s. Ambreen et al. argued that quality requirements research is one of the emerging areas in their mapping study [ 7 ]. In their mapping study, they found 1–7 papers per year in the 1990s and 17–32 papers 2005–2012 Footnote 1 . In a recent paper in IEEE Software, quality requirements or non-functional requirements occurred in less than 1% as a keyword in papers in the Requirements Engineering Journal and at the REFSQ conference and not at all in the top-ten list for the Requirements Engineering conference [ 98 ]. This is an indication that the statement that quality requirements being an emerging area of research needs to be nuanced. We believe there is a need for further research on quality requirements. However, with the results we got from our study, it seems that the research on quality requirements might be less directed towards the practical challenges facing industry. This, however, is something that needs more research to be confirmed.

It should be noted that we have not included studies where specific sub-characteristics of quality requirements, such as security or usability, are the focus. Ambreen et al., however, included also those [ 7 ]. Hence, the figures we presented might be on the lower side regarding the number of empirical studies. At the same time, we included 84 papers, whereas Ambreen et al. found 36 papers. If we limit our results to papers before 2013—i.e., to the same period as Ambreen et al.—we found 43 papers.

In terms of the type of research performed, we see a similar distribution among validation, evaluation, and experience studies as Ambreen et al. [ 7 ]—see Table  3 . Evaluation research is the most common and case study in industry the largest category. As noted in Sect.  4.1 , however, the rigor overall is weak. Furthermore, replications, longitudinal, and evaluation studies of a particular solution are rare. Similarly, we found only 4 experiments that can infer that the field could benefit from larger research initiatives planning several studies over many years. This would enable researchers to plan multiple studies, combining different research methods and larger sampling of the relevant population.

5.2 RQ2 What are the problems and challenges for quality requirements identified by empirical studies?

We found studies claiming quality requirements are treated the same way as other types of requirements. We also found studies claiming companies do prioritize quality requirements and other studies claiming quality requirements are not handled properly. Furthermore, we found several studies attempting to discern which of the sub-characteristics—such as security or performance—are more important than others in a certain context. The studies we found are conducted in different contexts, domains, and different research methods. We interpret the empirical data that, on the one hand, different sub-characteristics might warrant individual attention, on the other, different research questions and methods are needed to understand and address industry-relevant challenges. We hypothesize that the requirements engineering community has not yet found a good way to analyze the quality requirements practices and challenges. We believe longitudinal studies is one way forward to deepen the understanding of quality requirements over the lifecycle of a product family rather than individual products or even just one point in time. This, of course, is not isolated to quality requirements and would entail changes to how projects are funded to allow for bigger projects.

Overall, we summarize that quality requirements are written informally without a specific notation or modeling approach, there is no clear industry practice, and documentation of quality requirements seems to be performed in the same way as other requirements. Firstly, we speculate that there is a lack of understanding from practitioners of the specific needs for quality requirements, as they do not seem to prioritize separate handling. We believe the key to improving the understanding of the importance of quality requirements is to better understand the consequences and implications of quality requirements. Secondly, we believe the results imply that quality requirements engineering need to align well with other requirements topics as the cost of separate handling might deter usage.

We did not find any studies connecting quality requirements to business value nor success criteria such as timely delivery or increased sales. There are some attempts to connect user satisfaction to quality requirements and defect flows to quality requirements. There are also some studies trying to discern perspectives internally—e.g., that of architects. We also found some studies trying to understand app reviews as a source of requirements. However, we did not find any other studies attempting to identify other sources or ways of eliciting or analyzing new requirements. Interestingly, we found only one study explicitly on open-source software. There are some studies on agile methods, but we did not find any studies on DevOps, nor any studies bridging the gap between engineering and business. We propose to study quality requirements in more contexts, especially software ecosystems where commercial organizations cooperate both on development as well as operations. Furthermore, open-source is increasingly common. Hence, we believe these areas warrant more research.

We believe—at least for certain quality requirements sub-characteristics—it is key to understand the actual usage by actual users to discern which quality requirements to address. We propose data-driven approaches as an important trend for quality requirements, seen with different automatic analysis techniques of app stores. Another example of unexploited potential is customer service data—whether through, e.g., issues or reviews. We lack a clear data-driven perspective, where usage data, as one example, is studied with a quality requirements perspective. Furthermore, we believe the community needs to understand and address the connection between quality requirements and external factors such as business value or project success. An often-cited study by Finkelstein et al. claim quality requirements as one source of problem [ 45 ]. We recognize, though, that this type of research is both challenging to design, expensive to perform, and difficult to get rigorous and relevant results. We see a need for future research in quality requirements to study not quality requirements in isolation but as part of larger studies where quality requirements are one of the research questions.

5.3 RQ3 Which quality requirements solution proposals have been empirically validated?

Quality models—as an over-arching principle—is the most common approach to elicitation. The different approaches typically propose tailoring of a generic quality model—often ISO9126 [ 57 ] or ISO25010 [ 56 ]—and a process with workshops to elicit and analyze quality requirements. The studies overwhelmingly report success with such strategies, albeit few have concrete and validated numbers for effort nor lead-time. However, it seems to us that there is no solid evidence of the cost-effectiveness of quality models, a lack of evidence of adherence over time, nor clear data for success factors from a more complete scope when using quality models. We see an opportunity for companies already using—or willing to introduce—quality models which should make it reasonable to conduct longitudinal studies on quality models as future work.

We only found one modeling approach—goal modeling. Goal modeling is—similar to quality models—well researched. There are experiments and case studies. However, we could not find any surveys nor action research. Goal modeling is, just as quality models, integrated often integrated into a process encompassing all aspects of quality requirements. We interpret the lack of quality requirements modeling studies as follows: (1) Modeling of quality requirements cannot be separated from the modeling of other requirements. This is in line with quality requirements artifact studies, which often reports similar or identical specification for quality requirements and other requirements. (2) The modeling solutions proposed by academia are not something practitioners see as applicable or relevant.

We found very few studies on data-driven requirements engineering [ 73 ] in the context of quality requirements. Rather, there seems to be a focus on the requirements specification, e.g., with quality models, goal modeling, and with the automatic analysis studies on mining specifications for quality requirements. Furthermore, we could not find any approaches integrating quality requirements engineering with, for example, DevOps and continuous experimentation. We see a gap for studies on how user feedback (reviews, customer service data, etc.) and usage data (measurements when using a software product or service) can be used for quality requirements prioritization, elicitation, and release planning. Furthermore, we propose that evaluating the trend over time as a means to better understand the connection between quality requirements and user satisfaction. Especially, we see an opportunity with the lead-time from an improvement in the quality requirements—through the implementation—and the lead-time from downward-trending user satisfaction to actions taken—using various measurements—are important to develop relevant early forecasting metrics and improved prioritization mechanisms which consider estimations of the user satisfaction.

We note that the validation studies we found tend to report an improvement and rarely conclude that the proposals as a whole do not work. The explanations can be many, but publication bias can be one, another might be confirmation bias. We see this as an indication that empirical software engineering as an area is still developing and maturing.

5.4 Limitations and threats to validity

5.4.1 construct validity.

Construct validity refers to the decisions on method and tools, and whether they are appropriate for the research questions. We utilized a hybrid search strategy. The risk is if the start set for the snowballing approach is insufficient, all papers will not be found when snowballing. However, Mourao et al. recently published an evaluation that a hybrid strategy is an appropriate alternative [ 76 ]. Hence, the hybrid method is considered to be appropriate for our research questions.

Start set I includes 274 papers between the years 1991 and 2014. Start set II includes 173 papers between 1990 and 2019—of which 70 are from the period 2015–2019. The snowballing iterations included 83 papers from 1976–2019. 447 of the 530 (84%) of papers screened come from Start set I and the Start set II of papers. Furthermore, based on our experience, we believe we have included several key papers on empirical evidence on quality requirements. Hence, we believe that the fact that most of the included papers come from Start set I and the Start set II indicates that we have likely found the majority of all relevant papers. This implies that our method selection is appropriate.

5.4.2 Internal validity

A validity threat is if papers are excluded even though they should have been included or erroneous classification. We ensured that all border-line cases were screened by at least two researchers and a sample of all papers was also screened by at least two researchers to mitigate this, see Sect.  3.4 . We also followed a pre-defined method thoroughly. In the end, the process resulted in:

For Start set I, all papers were screened by at least 2 researchers.

For Start set II, 61% of the papers were screened by at least two researchers.

For the two snowballing iterations, 29% and 60% of the papers respectively were screened by at least two researchers.

66% of the 193 papers included in the full read were read by at least two researchers, including reviewing the classification.

83% of excluded papers in the screening step and 77% of the excluded papers in the full read step were read by at least two researchers.

Excluded papers were reviewed more often by at least two researchers than included papers, mitigating the threat of excluding papers that should be included. We argue that threats to internal validity are low.

There is a risk that we missed relevant papers are we excluded relevant papers as we excluded papers focusing on a specific type of quality requirements, e.g., performance or security. The risk is related to, on the one hand, terminology and, on the other, that the specific empirical study is on a specific type but the method or phenomena applies to quality requirements in general. For the former, we believe that our selection of search terms in the extended step is by far the most prevalent, hence, it should not be a big problem. Furthermore, since we also use a snowballing approach, this threat is further minimized. For the latter, we cannot completely dismiss the threat as some empirical evidence might not be presented on other papers even though the results might be applicable. We excluded 5 papers based on these criteria. Hence, we conclude that even though this is a threat, it is not likely to largely impact the internal from our paper.

5.4.3 Conclusion validity

We followed a systematic process to address threats to the conclusion validity. Furthermore, we report the steps and results in such a way that it should be possible to replicate them. The threat to conclusion is the inclusion/exclusion primarily, which entails a human judgment and thereby susceptible for errors. However, as mentioned for internal validity threats, we used a peer review process among the authors to minimize the threats of human errors.

5.4.4 External validity

External validity concerns the applicability of the results of our study. We believe the systematic hybrid process limits this threat as we do not exclude research communities nor do we exclude studies even if particular keywords are missing. However, we did not analyze different domains in detail as that information was not available in sufficient detail in enough studies. It might be that different domains exhibit different characteristics in terms of quality requirements engineering. Hence, the results should be applied after careful consideration.

6 Conclusion

The results of our systematic literature review indicate that there are many quality requirements engineering aspects that warrant further research. We judge that 84 papers over 30 years point to a lack of studies. However, this is something that should be studied in more detail to be confirmed. Furthermore, it seems to us that there is a divide between academically proposed solutions accepted by practitioners. The proposed solutions are rarely evaluated in realistic settings—and replications are non-existent. Furthermore, practitioners rarely report using any specific approach for quality requirements. At the same time, the existing surveys are small, have an unclear sample and population, and are rarely connected to any theory. We, therefore, hypothesize that overall, there is a lack of clear empirical evidence for what software developing organizations should adopt. This, again, is something that warrants further research to understand the needs of practitioners and their relation to proposed solutions found in the literature.

For practitioners, there are some recommendations of what has worked in realistic contexts. Quality models with the associated processes, QUPER, and the NFR method have been reported as useful in several studies. However, it is not clear what the return of investment is nor the long-term effect. Still, we believe our results indicate those to be a good starting point if an organization should improve their quality requirements practices. Furthermore, goal modeling has been evaluated in academic settings with positive results. However, we could not find any evaluations in a realistic setting specifically for quality requirements. In the context where a document or specification is received, different automatic analysis approaches seem to be able to help in identifying quality requirements. However, we could not find any available tools nor clear integration in the overall software engineering process. Hence, even though these solutions show potential, the effort needed to apply them in practice is unclear.

For researchers, we see a need for longitudinal studies on quality requirements. There are examples of solutions evaluated at one point in time. However, we could not find any studies on the long-term effect and costs of changing how companies work with quality requirements. We believe that the product or portfolio lifecycle is particularly under-researched. Furthermore, we believe there is a lack of understanding of the challenges and needs in realistic settings, as the solutions proposed by researchers seem to fail in getting acceptance from practitioners. This is a rather difficult issue for individuals to address, rather the requirements engineering community should try to establish a new way of performing research where larger and longer studies are viable.

Furthermore, there are only a few studies on sources of quality requirements in general and data-driven alternatives specifically. We believe there is potential in sources such as usage data, customer service data, and continuous experimentation to complement stakeholder analysis, expert input, and focus groups. The former has the potential to take in a breadth of input closer to the actual users while the latter will focus on fewer persons’ opinions or experiences which will be less representative of the actual usage.

We limited our systematic literature review to quality requirements in general and excluding sub-categories such as security or usability. We believe it would be interesting to perform a similar study on the different sub-categories. For one, there might be differences in the sub-categories both regarding the strength of evidence and the types of solutions proposed. On the other, it might be that it does not make sense to have one solution for all types of quality requirements categories.

Their results do not include papers after 2012. Hence, we choose a slightly different interval.

Ahmad A, Feng C, Li K, Asim SM, Sun T (2019) Toward empirically investigating non-functional requirements of ios developers on stack overflow. IEEE Access 7:61145–61169. https://doi.org/10.1109/access.2019.2914429

Article   Google Scholar  

Al Balushi TH, Sampaio PRF, Loucopoulos P (2013) Eliciting and prioritizing quality requirements supported by ontologies: a case study using the e licit o framework and tool. Expert Syst 30(2):129–151

Al-Kilidar H, Cox K, Kitchenham B (2005) The use and usefulness of the iso/iec 9126 quality standard. In: International symposium on empirical software engineering. IEEE, pp 126–132. https://doi.org/10.1109/ISESE.2005.1541821

Alsaqaf W, Daneva M, Wieringa R (2019) Quality requirements challenges in the context of large-scale distributed agile: an empirical study. Inf Softw Technol 110:39–55. https://doi.org/10.1016/j.infsof.2019.01.009

Alsaqaf W, Daneva M, Wieringa R (2017) Quality requirements in large-scale distributed agile projects–a systematic literature review. In: International working conference on requirements engineering: foundation for software quality. Springer, pp 219–234. https://doi.org/10.1007/978-3-319-54045-0_17

Amasaki S, Leelaprute P (2018) The effects of vectorization methods on non-functional requirements classification. In: 2018 44th euromicro conference on software engineering and advanced applications (SEAA). IEEE, pp. 175–182. https://doi.org/10.1109/seaa.2018.00036

Ambreen T, Ikram N, Usman M, Niazi M (2018) Empirical research in requirements engineering: trends and opportunities. Requir Eng 23(1):63–95. https://doi.org/10.1007/s00766-016-0258-2

Ameller D, Galster M, Avgeriou P, Franch X (2016) A survey on quality attributes in service-based systems. Softw Qual J 24(2):271–299

Ameller D, Ayala C, Cabot J, Franch X (2012) How do software architects consider non-functional requirements: an exploratory study. In: 2012 20th IEEE international requirements engineering conference (RE). IEEE, pp 41–50. https://doi.org/10.1109/re.2012.6345838

Baker C, Deng L, Chakraborty S, Dehlinger J (2019) Automatic multi-class non-functional software requirements classification using neural networks. In: 2019 IEEE 43rd annual computer software and applications conference (COMPSAC), vol 2. IEEE, pp 610–615. https://doi.org/10.1109/compsac.2019.10275

Becha H, Amyot D (2012) Non-functional properties in service oriented architecture - a consumer’s perspective. J Softw 7(3):575–587. https://doi.org/10.4304/jsw.7.3.575-587

Benslimane Y, Cysneiros LM, Bahli B (2007) Assessing critical functional and non-functional requirements for web-based procurement systems: a comprehensive survey. Requir Eng 12(3):191–198. https://doi.org/10.1007/s00766-007-0050-4

Berntsson Svensson R, Sprockel Y, Regnell B, Brinkkemper S (2012) Setting quality targets for coming releases with quper: an industrial case study. Requir Eng 17(4):283–298. https://doi.org/10.1007/s00766-011-0125-0

Berntsson Svensson R, Olsson T, Regnell B (2013) An investigation of how quality requirements are specified in industrial practice. Inf Softw Technol 55(7):1224–1236. https://doi.org/10.1016/j.infsof.2013.01.006

Berntsson Svensson R, Host M, Regnell B (2010) Managing quality requirements: a systematic review. In: 2010 36th EUROMICRO conference on software engineering and advanced applications. IEEE, pp 261–268. https://doi.org/10.1109/seaa.2010.55

Berntsson Svensson R, Olsson T, Regnell B (2008) Introducing support for release planning of quality requirements-an industrial evaluation of the quper model. In: Second international workshop on software product management. IEEE, pp 18–26

Berntsson Svensson R, Parker PL, Regnell B (2011) A prototype tool for quper to support release planning of quality requirements. In: Fifth international workshop on software product management (IWSPM). IEEE, pp 57–66. https://doi.org/10.1109/iwspm.2011.6046204

Berntsson Svensson R, Regnell B (2015) A case study evaluation of the guideline-supported quper model for elicitation of quality requirements. In: International working conference on requirements engineering: foundation for software quality. Springer, pp 230–246. https://doi.org/10.1007/978-3-319-16101-3_15

Bittencourt RV, De Araujo RM (2011) Using business models to identify expected quality for information systems. Int J Bus Process Integr Manag 5(2):159–172. https://doi.org/10.1504/ijbpim.2011.040207

Borg A, Yong A, Carlshamre P, Sandahl K (2003) The bad conscience of requirements engineering: an investigation in real-world treatment of non-functional requirements. In: Software engineering research and practice in Sweden

Buitr SL, Pino FJ, Flores-Rios BL, Ibarra-Esquer JE, et al. (2017) A model for enhancing tacit knowledge flow in non-functional requirements elicitation. In: 2017 5th international conference in software engineering research and innovation (CONISOFT). IEEE, pp 25–33. https://doi.org/10.1109/conisoft.2017.00011

Calazans TA, Paldês RÁ, Canedo ED, Masson TE, Guimares FdA, Rezende MK, Braosi E, Kosloski RAD (2019) Quality requirements: analysis of utilization in the systems of a financial institution. In: ACM/IEEE international symposium on empirical software engineering and measurement (ESEM). IEEE, pp 1–6. https://doi.org/10.1109/esem.2019.8870151

Caracciolo A, Lungu MF, Nierstrasz O (2014) How do software architects specify and validate quality requirements?. In: European conference on software architecture. Springer, pp 374–389. https://doi.org/10.1007/978-3-319-09970-5_32

Casamayor A, Godoy D, Campo M (2010) Identification of non-functional requirements in textual specifications: a semi-supervised learning approach. Inf Softw Technol 52(4):436–445. https://doi.org/10.1016/j.infsof.2009.10.010

Chiam YK, Staples M, Ye X, Zhu L (2013) Applying a selection method to choose quality attribute techniques. Inf Softw Technol 55(8):1419–1436. https://doi.org/10.1016/j.infsof.2013.02.001

Chung L, Nixon BA (1995) Dealing with non-functional requirements: three experimental studies of a process-oriented approach. In: 17th international conference on software engineering. IEEE, pp 25–25. https://doi.org/10.1145/225014.225017

Cleland-Huang, J., Mazrouee S, Liguo H, Port D (2007) nfr. https://doi.org/10.5281/zenodo.268542

Cleland-Huang J, Settimi R, Zou X, Solc P (2007) Automated classification of non-functional requirements. Requir Eng 12(2):103–120. https://doi.org/10.1007/s00766-007-0045-1

Condori-Fernandez N, Lago P (2015) Can we know upfront how to prioritize quality requirements? In: 2015 IEEE fifth international workshop on empirical requirements engineering (EmpiRE). IEEE, pp 33–40. https://doi.org/10.1109/empire.2015.7431305

Cysneiros LM, do Prado Leite JS (1999) Integrating non-functional requirements into data modeling. In: Proceedings IEEE international symposium on requirements engineering. IEEE, pp 162–171

Cysneiros LM, do Prado Leite JCS (2004) Nonfunctional requirements: from elicitation to conceptual models. IEEE Trans Software Eng 30(5):328–350. https://doi.org/10.1109/tse.2004.10

Cysneiros LM, do Prado Leite JCS, Neto JDMS (2001) A framework for integrating non-functional requirements into conceptual models. Requir Eng 6(2):97–115. https://doi.org/10.1007/s007660170008

Article   MATH   Google Scholar  

Daneva M, Buglione L, Herrmann A (2013) Software architects’ experiences of quality requirements: what we know and what we do not know? In: international working conference on requirements engineering: foundation for software quality. Springer, pp 1–17. https://doi.org/10.1007/978-3-642-37422-7_1

Daneva M, Marczak S, Herrmann A (2014) Engineering of quality requirements as perceived by near-shore development centers’ architects in eastern europe: the hole in the whole. In: Proceedings of the 8th ACM/IEEE international symposium on empirical software engineering and measurement. ACM, p 19. https://doi.org/10.1145/2652524.2652534

De La Vara JL, Wnuk K, Berntsson-Svensson R, Sánchez J, Regnell B (2011) An empirical study on the importance of quality requirements in industry. In: Proceedings of the 23rd international conference on software engineering and knowledge engineering. pp 438–443. https://doi.org/10.1007/s10270-008-0108-x

Doerr J, Kerkow D, Koenig T, Olsson T, Suzuki T (2005) Non-functional requirements in industry-three case studies adopting an experience-based nfr method. In: 13th IEEE international conference on requirements engineering (RE’05). IEEE, pp 373–382. https://doi.org/10.1109/re.2005.47

Dörr J, Kerkow D, Von Knethen A, Paech B (2003) Eliciting efficiency requirements with use cases. In: Ninth international workshop on requirements engineering: foundation for software quality. In conjunction with CAiSE, vol 3

Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. In: Guide to advanced empirical software engineering. Springer, pp 285–311. https://doi.org/10.1007/978-1-84800-044-5_11

Ebert C (1998) Putting requirement management into praxis: dealing with nonfunctional requirements. Inf Softw Technol 40(3):175–185. https://doi.org/10.1016/s0950-5849(98)00049-4

Eckhardt J, Vogelsang A, Fernández DM (2016) Are “non-functional” requirements really non-functional? an investigation of non-functional requirements in practice. In: 2016 IEEE/ACM 38th international conference on software engineering (ICSE). IEEE, pp 832–842. https://doi.org/10.1145/2884781.2884788

Eckhardt J, Vogelsang A, Fernández DM (2016) Are “non-functional” requirements really non-functional? an investigation of non-functional requirements in practice. In: IEEE/ACM 38th international conference on software engineering (ICSE). IEEE, pp 832–842. https://doi.org/10.1145/2884781.2884788

Edwards R, Bencomo N (2018) Desire: further understanding nuances of degrees of satisfaction of non-functional requirements trade-off. In: Proceedings of the 13th international conference on software engineering for adaptive and self-managing systems. ACM, pp 12–18. https://doi.org/10.1145/3194133.3194142

Ernst NA, Mylopoulos J (2010) On the perception of software quality requirements during the project lifecycle. In: International working conference on requirements engineering: foundation for software quality. Springer, pp 143–157. https://doi.org/10.1007/978-3-642-14192-8_15

Farid WM, Mitropoulos FJ (2012) Normatic: A visual tool for modeling non-functional requirements in agile processes. In: Proceedings of IEEE southeastcon. IEEE, pp 1–8 https://doi.org/10.1109/secon.2012.6196989

Finkelstein A, Dowell J (1996) A comedy of errors: the london ambulance service case study. In: Proceedings of the 8th International Workshop on Software Specification and Design. pp. 2–4. IEEE

García-Mireles GA, De La Rubia MÁM, García F, Piattini M (2015) Identifying quality characteristic interactions during software development. In: International conference on evaluation of novel approaches to software engineering (ENASE). IEEE, pp 78–89. https://doi.org/10.5220/0005351800780089

García-Mireles GA (2016) Identifying relevant product quality characteristics in the context of very small organizations. Comput Sci Inf Syst 13(3):875–900. https://doi.org/10.2298/csis160809034g

Glinz M (2007) On non-functional requirements. In: 15th IEEE international requirements engineering conference (RE 2007). IEEE, pp 21–26. https://doi.org/10.1109/re.2007.45

Grimshaw DJ, Draper GW (2001) Non-functional requirements analysis: deficiencies in structured methods. Inf Softw Technol 43(11):629–634. https://doi.org/10.1016/s0950-5849(01)00171-9

Groen EC, Kopczyńska S, Hauer MP, Krafft TD, Doerr J (2017) Users-the hidden software product quality experts?: a study on how app users report quality aspects in online reviews. In: IEEE 25th international requirements engineering conference (RE). IEEE, pp 80–89. https://doi.org/10.1109/re.2017.73

Haigh M (2010) Software quality, non-functional software requirements and it-business alignment. Software Qual J 18(3):361–385. https://doi.org/10.1007/s11219-010-9098-3

Herrmann A, Paech B (2008) Moqare: misuse-oriented quality requirements engineering. Requir Eng 13(1):73–86

Ho CW, Williams L, Robinson B (2008) Examining the relationships between performance requirements and “not a problem” defect reports. In: 16th IEEE international requirements engineering conference. IEEE, pp 135–144

Höst M, Regnell B, Wohlin C (2000) Using students as subjects-a comparative study of students and professionals in lead-time impact assessment. Empir Softw Eng 5(3):201–214. https://doi.org/10.1023/A:1026586415054

In H, Boehm BW (2001) Using winwin quality requirements management tools: a case study. Ann Softw Eng 11(1):141–174

ISO/IEC 25010:2011(E) (2011) Systems and software engineering - systems and software quality requirements and evaluation (SquaRE)

ISO/IEC 9126-1:2001 (2001) Software engineering - product quality - part 1: quality model

Ivarsson M, Gorschek T (2011) A method for evaluating rigor and industrial relevance of technology evaluations. Empir Softw Eng 16(3):365–395. https://doi.org/10.1007/s10664-010-9146-4

Jacobs S (1999) Introducing measurable quality requirements: a case study. In: Proceedings IEEE international symposium on requirements engineering (Cat. No. PR00188). IEEE, pp 172–179

Jha N, Mahmoud A, Mining non-functional requirements from app store reviews. Empir Softw Eng 1–37. https://doi.org/10.1007/s10664-019-09716-7

Johansson E, Wesslén A, Bratthall L, Host M (2001) The importance of quality requirements in software platform development-a survey. In: Proceedings of the 34th annual hawaii international conference on system sciences. IEEE, p 10. https://doi.org/10.1109/HICSS.2001.927252

Jung HW (2007) Validating the external quality subcharacteristics of software products according to iso/iec 9126. Comput Stand Interfaces 29(6):653–661. https://doi.org/10.1016/j.csi.2007.03.004

Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering

Kopczyńska S, Nawrocki J, Ochodek M (2018) An empirical study on catalog of non-functional requirement templates: usefulness and maintenance issues. Inf Softw Technol 103:75–91. https://doi.org/10.1016/j.infsof.2018.06.009

Kopczyńska S, Nawrocki J (2014) Using non-functional requirements templates for elicitation: a case study. In: 2014 IEEE 4th international workshop on requirements patterns (RePa). IEEE, pp 47–54. https://doi.org/10.1109/repa.2014.6894844

Kurtanović Z, Maalej W (2017) Automatically classifying functional and non-functional requirements using supervised machine learning. In: 2017 IEEE 25th international requirements engineering conference (RE). IEEE, pp 490–495. https://doi.org/10.1109/re.2017.82

Kusters RJ, van Solingen R, Trienekens JJ (1999) Identifying embedded software quality: two approaches. Qual Reliab Eng Int 15(6):485–492

Lawrence B, Wiegers K, Ebert C (2001) The top risk of requirements engineering. IEEE Softw 18(6):62–63. https://doi.org/10.1109/52.965804

Leung HK (2001) Quality metrics for intranet applications. Inf Manag 38(3):137–152. https://doi.org/10.1016/s0378-7206(00)00060-4

Lian X, Cleland-Huang J, Zhang L (2017) Mining associations between quality concerns and functional requirements. In: 2017 IEEE 25th international requirements engineering conference (RE). IEEE, pp 292–301. https://doi.org/10.1109/re.2017.68

Lochmann K, Fernández DM, Wagner S (2012) A case study on specifying quality requirements using a quality model. In: 19th Asia-Pacific software engineering conference, vol 1. IEEE, pp. 577–582. https://doi.org/10.1109/apsec.2012.57

Lu M, Liang P (2017) Automatic classification of non-functional requirements from augmented app user reviews. In: Proceedings of the 21st international conference on evaluation and assessment in software engineering. ACM, pp 344–353. https://doi.org/10.1145/3084226.3084241

Maalej W, Nayebi M, Johann T, Ruhe G (2015) Toward data-driven requirements engineering. IEEE Softw 33(1):48–54. https://doi.org/10.1109/MS.2015.153

Mahmoud A, Williams G (2016) Detecting, classifying, and tracing non-functional software requirements. Requir Eng 21(3):357–381. https://doi.org/10.1007/s00766-016-0252-8

Mohagheghi P, Aparicio ME (2017) An industry experience report on managing product quality requirements in a large organization. Inf Softw Technol 88:96–109. https://doi.org/10.1016/j.infsof.2017.04.002

Mourão E, Pimentel JF, Murta L, Kalinowski M, Mendes E, Wohlin C (2020) On the performance of hybrid search strategies for systematic literature reviews in software engineering. Inf Softw Technol. https://doi.org/10.1016/j.infsof.2020.106294

Mourão E, Kalinowski M, Murta L, Mendes E, Wohlin C (2017) Investigating the use of a hybrid search strategy for systematic reviews. In: International symposium on empirical software engineering and measurement (ESEM). IEEE, pp 193–198. https://doi.org/10.1109/esem.2017.30

Mylopoulos J, Chung L, Nixon B (1992) Representing and using nonfunctional requirements: a process-oriented approach. IEEE Trans Softw Eng 6:483–497. https://doi.org/10.1109/32.142871

Niemelä E, Immonen A (2007) Capturing quality requirements of product family architecture. Inf Softw Technol 49(11–12):1107–1120. https://doi.org/10.1016/j.infsof.2006.11.003

Olsson T, Wnuk K, Gorschek T (2019) An empirical study on decision making for quality requirements. J Syst Softw 149:217–233. https://doi.org/10.1016/j.jss.2018.12.002

Ouhbi S, Idri A, Fernández-Alemán JL, Toval A (2013) Software quality requirements: a systematic mapping study. In: 20th Asia-Pacific software engineering conference (APSEC), vol 1. IEEE, pp 231–238. https://doi.org/10.1109/apsec.2013.40

Paech B, Kerkow D (2004) Non-functional requirements engineering-quality is essential. In: 10th international workshop on requirements engineering foundation for software quality

Phillips LB, Aurum A, Svensson RB (2012) Managing software quality requirements. In: 38th euromicro conference on software engineering and advanced applications. IEEE, pp 349–356. https://doi.org/10.1109/seaa.2012.65

Poort ER, Martens N, Van De Weerd I, Van Vliet H (2012) How architects see non-functional requirements: beware of modifiability. In: International working conference on requirements engineering: foundation for software quality. Springer, pp 37–51. https://doi.org/10.1007/978-3-642-28714-5_4

Rago A, Marcos C, Diaz-Pace JA (2013) Uncovering quality-attribute concerns in use case specifications via early aspect mining. Requir Eng 18(1):67–84

Rahimi M, Mirakhorli M, Cleland-Huang J (2014) Automated extraction and visualization of quality concerns from requirements specifications. In: IEEE 22nd international requirements engineering conference (RE). IEEE, pp 253–262. https://doi.org/10.1109/re.2014.6912267

Regnell B, Svensson RB, Olsson T (2008) Supporting roadmapping of quality requirements. IEEE Softw 25(2):42–47. https://doi.org/10.1109/ms.2008.48

Regnell B, Höst M, Svensson RB (2007) A quality performance model for cost-benefit analysis of non-functional requirements applied to the mobile handset domain. In: International working conference on requirements engineering: foundation for software quality. Springer, pp 277–291. https://doi.org/10.1007/978-3-540-73031-6_20

Saito Y, Monden A, Matsumoto K (2012) Evaluation of non functional requirements in a request for proposal (rfp). In: Joint conference of the 22nd international workshop on software measurement and the 2012 seventh international conference on software process and product measurement. IEEE, pp 106–111. https://doi.org/10.1109/iwsm-mensura.2012.23

Shahrokni A, Feldt R (2013) Industrial challenges with quality requirements in safety critical software systems. In: 39th Euromicro conference on software engineering and advanced applications. IEEE, pp 78–81. https://doi.org/10.1109/seaa.2013.57

Sharma VS, Ramnani RR, Sengupta S (2014) A framework for identifying and analyzing non-functional requirements from text. In: Proceedings of the 4th international workshop on twin peaks of requirements and architecture. ACM, pp 1–8. https://doi.org/10.1145/2593861.2593862

Sibisi M, Van Waveren CC (2007) A process framework for customising software quality models. In: AFRICON. IEEE, pp 1–8. https://doi.org/10.1109/afrcon.2007.4401495

Silva A, Pinheiro PR, Albuquerque A, Barroso J (2017) Evaluation of an approach to define elicitation guides of non-functional requirements. IET Softw 11(5):221–228. https://doi.org/10.1049/iet-sen.2016.0302

Singh P, Singh D, Sharma A (2016) Rule-based system for automated classification of non-functional requirements from requirement specifications. In: 2016 International conference on advances in computing, communications and informatics (ICACCI). IEEE, pp 620–626. https://doi.org/10.1109/icacci.2016.7732115

Slankas J, Williams L (2013) Automated extraction of non-functional requirements in available documentation. In: 2013 1st international workshop on natural language analysis in software engineering (NaturaLiSE). IEEE, pp 9–16. https://doi.org/10.1109/naturalise.2013.6611715

Svensson RB, Gorschek T, Regnell B, Torkar R, Shahrokni A, Feldt R, Aurum A (2011) Prioritization of quality requirements: state of practice in eleven companies. In: IEEE 19th international requirements engineering conference. IEEE, pp 69–78. https://doi.org/10.1109/re.2011.6051652

Svensson RB, Gorschek T, Regnell B, Torkar R, Shahrokni A, Feldt R (2011) Quality requirements in industrial practice-an extended interview study at eleven companies. IEEE Trans Softw Eng 38(4):923–935. https://doi.org/10.1109/tse.2011.47

Tenbergen B, Daun M (2019) Is requirements-engineering research delivering what it promised?: a review of its accomplishments and opportunities after 10 years. IEEE Softw 36(4):6–11. https://doi.org/10.1109/ms.2019.2909127

Teruel MA, Navarro E, López-Jaquero V, Montero F, Jaen J, González P (2012) Analyzing the understandability of requirements engineering languages for cscw systems: a family of experiments. Inf Softw Technol 54(11):1215–1228. https://doi.org/10.1016/j.infsof.2012.06.001

Tóth L, Vidács L (2018) Study of various classifiers for identification and classification of non-functional requirements. In: International conference on computational science and its applications. Springer, pp 492–503. https://doi.org/10.1007/978-3-319-95174-4_39

Vanhanen J, Mäntylä MV, Itkonen J (2009) Lightweight elicitation and analysis of software product quality goals: a multiple industrial case study. In: 2009 third international workshop on software product management. IEEE, pp 42–52. https://doi.org/10.1109/iwspm.2009.5

Wagner S, Lochmann K, Heinemann L, Kläs M, Trendowicz A, Plösch R, Seidl A, Goeb A, Streit J (2012) The quamoco product quality modelling and assessment approach. In: Proceedings of the 34th international conference on software engineering. IEEE Press, pp 1133–1142. https://doi.org/10.1109/icse.2012.6227106

Wang T, Liang P, Lu M (2018) What aspects do non-functional requirements in app user reviews describe? an exploratory and comparative study. In: 2018 25th Asia-pacific software engineering conference (APSEC). IEEE, pp 494–503. https://doi.org/10.1109/apsec.2018.00064

Wieringa R, Maiden N, Mead N, Rolland C (2006) Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requir Eng 11(1):102–107. https://doi.org/10.1007/s00766-005-0021-6

Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: International conference on evaluation and assessment in software engineering. p 38. https://doi.org/10.1145/2601248.2601268

Younas M, Jawawi D, Ghani I, Shah MA Extraction of non-functional requirement using semantic similarity distance. Neural Comput Appl 1–15. https://doi.org/10.1007/s00521-019-04226-5

Yu E (1997) Towards modelling and reasoning support for early-phase requirements engineering. In: International symposium on requirements engineering. pp 226–35. https://doi.org/10.1109/ISRE.1997.566873

Yusop N, Zowghi D, Lowe D (2008) The impacts of non-functional requirements in web system projects. Int J Value Chain Manag. https://doi.org/10.1504/ijvcm.2008.016116

Zhang W, Yang Y, Wang Q, Shu F (2011) An empirical study on classification of non-functional requirements. In: The twenty-third international conference on software engineering and knowledge engineering (SEKE 2011). pp 190–195

Zou J, Xu L, Yang M, Zhang X, Yang D (2017) Towards comprehending the non-functional requirements through developers eyes: an exploration of stack overflow using topic analysis. Inf Softw Technol 84:19–32. https://doi.org/10.1016/j.infsof.2016.12.003

Zubcoff J, Garrigós I, Casteleyn S, Mazón JN, Aguilar JA, Gomariz-Castillo F (2019) Evaluating different i*-based approaches for selecting functional requirements while balancing and optimizing non-functional requirements: a controlled experiment. Inf Softw Technol 106:68–84. https://doi.org/10.1016/j.infsof.2018.09.004

Download references

Open access funding provided by RISE Research Institutes of Sweden.

Author information

Authors and affiliations.

RISE Research Institutes of Sweden AB, Gothenburg, Sweden

Thomas Olsson & Efi Papatheocharous

Mälardalen University, Västerås, Sweden

Séverine Sentilles

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Thomas Olsson .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The work is partially supported by a research grant for the ORION project (reference number 20140218) from The Knowledge Foundation in Sweden.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Olsson, T., Sentilles, S. & Papatheocharous, E. A systematic literature review of empirical research on quality requirements. Requirements Eng 27 , 249–271 (2022). https://doi.org/10.1007/s00766-022-00373-9

Download citation

Received : 10 April 2020

Accepted : 17 January 2022

Published : 08 February 2022

Issue Date : June 2022

DOI : https://doi.org/10.1007/s00766-022-00373-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic literature review
• Quality requirements
• Non-functional requirements
• Empirical evidence
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

A Systematic Literature Review of Quality Management Initiatives in Dental Clinics

Affiliations.

• 1 Faculty of Economics and Business Administration, Department of Management, Babes-Bolyai University, 400591 Cluj-Napoca, Romania.
• 2 Faculty of Medicine, Department of Community Medicine, Public Health and Management, Iuliu Hatieganu University of Medicine and Pharmacy Cluj-Napoca, 400337 Cluj-Napoca, Romania.
• 3 Faculty of Economics and Business Administration, Department of Finance, Babes-Bolyai University, 400591 Cluj-Napoca, Romania.
• PMID: 34769604
• PMCID: PMC8582852
• DOI: 10.3390/ijerph182111084

By considering the recently proposed definitions and metrics, oral healthcare quality management (OHQM) emerges as a distinct field in the wider healthcare area. The goal of this paper is to systematically review quality management initiatives (QMIs) implementation by dental clinics. The research methodology approach is a review of 72 sources that have been analyzed using the Context-Intervention-Mechanism-Outcome Framework (CIMO). The analysis identifies five mechanisms that explain how quality management initiatives are implemented by dental clinics. The simplest QMIs implementations are related to (1) overall quality. The next ones, in terms of complexity, are related to (2) patient satisfaction, (3) service quality, (4) internal processes improvement, and (5) business outcomes. This paper is the first attempt to provide a critical review of this topic and represents an important advancement by providing a theoretical framework that explains how quality management is implemented by practitioners in this field. The results can be used by scholars for advancing their studies related to this emerging research area and by healthcare managers in order to better implement their quality management initiatives.

Keywords: CIMO framework; dental clinics; oral healthcare quality management; quality management initiative; systematic literature review.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Research process.

Similar articles

• The influence of contextual factors on healthcare quality improvement initiatives: a realist review. Coles E, Anderson J, Maxwell M, Harris FM, Gray NM, Milner G, MacGillivray S. Coles E, et al. Syst Rev. 2020 Apr 26;9(1):94. doi: 10.1186/s13643-020-01344-3. Syst Rev. 2020. PMID: 32336290 Free PMC article. Review.
• Benchmarking outcomes on multiple contextual levels in lean healthcare: a systematic review, development of a conceptual framework, and a research agenda. Reponen E, Rundall TG, Shortell SM, Blodgett JC, Juarez A, Jokela R, Mäkijärvi M, Torkki P. Reponen E, et al. BMC Health Serv Res. 2021 Feb 19;21(1):161. doi: 10.1186/s12913-021-06160-6. BMC Health Serv Res. 2021. PMID: 33607988 Free PMC article.
• The Experience and Effectiveness of Nurse Practitioners in Orthopaedic Settings: A Comprehensive Systematic Review. Taylor A, Staruchowicz L. Taylor A, et al. JBI Libr Syst Rev. 2012;10(42 Suppl):1-22. doi: 10.11124/jbisrir-2012-249. JBI Libr Syst Rev. 2012. PMID: 27820153
• Clinical Integration in a Dental School Clinic Through an Enhanced Patient Intake Process. Robinson FG, Fischbach H, Salisbury J, Stefanik D, Kearney R, Fields HW. Robinson FG, et al. J Dent Educ. 2019 Sep;83(9):1030-1038. doi: 10.21815/JDE.019.105. Epub 2019 May 13. J Dent Educ. 2019. PMID: 31085688
• Combined quality function deployment and logical framework analysis to improve quality of emergency care in Malta. Buttigieg SC, Dey PK, Cassar MR. Buttigieg SC, et al. Int J Health Care Qual Assur. 2016;29(2):123-40. doi: 10.1108/IJHCQA-04-2014-0040. Int J Health Care Qual Assur. 2016. PMID: 26959894 Review.
• Ferlie E.B., Shortell S.M. Improving the Quality of Health Care in the United Kingdom and the United States: A Framework for Change. Milbank Q. 2001;79:281–315. doi: 10.1111/1468-0009.00206. - DOI - PMC - PubMed
• Minkman M., Ahaus K., Huijsman R. Performance Improvement Based on Integrated Quality Management Models: What Evidence Do We Have? A Systematic Literature Review. Int. J. Qual. Health Care. 2007;19:90–104. doi: 10.1093/intqhc/mzl071. - DOI - PubMed
• Zabada C., Rivers P.A., Munchus G. Obstacles to the Application of Total Quality Management in Health-Care Organizations. Total. Qual. Manag. 1998;9:57–66. doi: 10.1080/0954412989261. - DOI
• Taner M.T., Sezen B., Antony J. An Overview of Six Sigma Applications in Healthcare Industry. Int. J. Health Care Qual. Assur. 2007;20:329–340. doi: 10.1108/09526860710754398. - DOI
• Nicolay C.R., Purkayastha S., Greenhalgh A., Benn J., Chaturvedi S., Phillips N., Darzi A. Systematic Review of the Application of Quality Improvement Methodologies from the Manufacturing Industry to Surgical Healthcare. Br. J. Surg. 2012;99:324–335. doi: 10.1002/bjs.7803. - DOI - PubMed

Publication types

• Search in MeSH

Related information

Linkout - more resources, full text sources.

• Europe PubMed Central
• PubMed Central

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

To read this content please select one of the options below:

Please note you do not have access to teaching notes, industry 4.0, quality management and tqm world. a systematic literature review and a proposed agenda for further research.

The TQM Journal

ISSN : 1754-2731

Article publication date: 26 June 2020

Issue publication date: 21 July 2020

The main purpose of this paper is to analyse the current literature situation in terms of relationships between Industry 4.0 and quality management and TQM. The author wanted to understand what topics and issues can be considered the most relevant referring to the so-called Quality 4.0, what the literature is missing opening avenues for further research.

Design/methodology/approach

This research employed a systematic literature review. In total, 75 papers from different sources were reviewed using specific inclusion and exclusion criteria.

Four categories of topics emerged, namely: creating value within the company through quality (big) data, analytics and artificial intelligence; developing Quality 4.0 skills and culture for quality people; customer value co-creation; cyber–physical systems and ERP for quality assurance and control. This paper also tried to understand if there is a definition of Quality 4.0 based on determined methods.

Research limitations/implications

Systematic literature review could have introduced some limitations in terms of the number and reliability of reviewed papers. Probably some interesting papers had been not intentionally missed.

Practical implications

Consultants and managers in developing and implementing their own Quality 4.0 models could use many practical and discussed implications concerning I4.0 technologies and quality management.

Originality/value

This is one of the first papers which employed the systematic literature review for researching Industry 4.0, quality management and TQM relationships.

• Quality 4.0
• Industry 4.0
• Quality management
• Systematic literature review

Chiarini, A. (2020), "Industry 4.0, quality management and TQM world. A systematic literature review and a proposed agenda for further research", The TQM Journal , Vol. 32 No. 4, pp. 603-616. https://doi.org/10.1108/TQM-04-2020-0082

Emerald Publishing Limited

Copyright © 2020, Emerald Publishing Limited

Related articles

All feedback is valuable.

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

A multi-level multi-product supply chain network design of vegetables products considering costs of quality: A case study

Roles Methodology, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Industrial Engineering, Iran University of Science and Technology, Narmak, Tehran, IR

Roles Supervision

Affiliation Agricultural Garden, Yaman Avenue, Shahid Chamran Highway, Tehran, IR

Roles Conceptualization

• Sareh Khazaeli, 
• Ramazan Kalvandi, 
• Hadi Sahebi

• Published: September 3, 2024
• https://doi.org/10.1371/journal.pone.0303054
• Peer Review
• Reader Comments

Effective logistics management is crucial for the distribution of perishable agricultural products to ensure they reach customers in high-quality condition. This research examines an integrated, multi-echelon supply chain for perishable agricultural goods. The supply chain consists of four stages: supply, processing, storage, and customers. This study investigates the quality-related costs associated with product perishability to maximize supply chain profitability. Key factors considered include the network design, location of processing and distribution centers, the ability to process raw products to minimize post-harvest quality degradation, the option to sell the excess produce to a secondary market due to unpredictable yields, and the decision not to fulfill demand from distant customers where significant quality loss and price drops would be involved, instead diverting those products to the aforementioned secondary market. Quantitative methods and linear mathematical programming are employed to model and validate the proposed supply chain using actual data from a real-world case study on vegetable supply chains. The main contribution of this research is the incorporation of quality costs into the objective function, which allows the supply chain to prioritize meeting nearby customers’ demands with minimal quality loss over serving distant customers where high quality loss is unavoidable. Additionally, deploying a faster transportation fleet can significantly improve the overall profitability of the perishable product supply chain.

Citation: Khazaeli S, Kalvandi R, Sahebi H (2024) A multi-level multi-product supply chain network design of vegetables products considering costs of quality: A case study. PLoS ONE 19(9): e0303054. https://doi.org/10.1371/journal.pone.0303054

Editor: Md. Monirul Islam, Bangladesh Agricultural University, BANGLADESH

Received: September 8, 2023; Accepted: April 18, 2024; Published: September 3, 2024

Copyright: © 2024 Khazaeli et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: The most critical data are presented in Supporting Information files. All are not presented due to the high space they need. If there is no space limitation in the paper, it can be published.

Funding: The author(s) received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

1. Introduction

Vegetables are perishable, edible, agricultural products that deteriorate during a limited shelf life [ 1 ]. Quality of perishable products is essential to the customer because such products deteriorate fast and endanger the consumer’s health [ 2 ]. There is a consensus in the literature on the reasons why people buy organic food; however, there is also a gap between the consumers’ generally positive attitude toward organic food and their relatively low level of actual purchases [ 3 ]. Quality of vegetables is one of the important measures to its customers due to the quality deterioration rate of products which relates to the health of consumers [ 2 ]. Time decay and shortages are common phenomena in products with short life cycles, and financial volatility necessitates a more accurate characterization of inventory costs based on time-adjusted value [ 4 ]. The supply chain management concept evolved when manufacturers experienced a strategic partnership with their direct suppliers. Then the logistics and transportation experts improved it one step forward and involved the distribution and transportation operations. Next, the concept of integrated logistics was recognized as the supply chain management [ 5 ]. Product quality is another novel concept in the supply chain management [ 6 ]. Moreover the quality deterioration often happens in traditional supply chains which, for the most part, are poorly planned [ 7 ]. From a product quality perspective, when processed products decay at a faster rate than raw materials, storing raw materials is favored [ 8 ]. Alternatively, when processing decreases the quality decay rate, a short time until processing is favored [ 9 ]. The supply chain (SC) of vegetables consists of four echelons: 1) purchasing raw materials, 2) processing, 3) distribution, and 4) customers to which products are delivered [ 10 ]. Since perishable products (agri-foods) have limited shelf life, logistic-related topics are important in business [ 11 ]. Transportation share in supply chain costs reached about 92% in the distribution sector in some traditional chains [ 12 ]. The post-harvest pre-customer-sent product loss [ 13 ] accounts for more than 40% of the supply chain costs even in industrialized and developed countries [ 14 ]. It occurs in terms of both the product quantity and agri-food quality loss throughout the chain and imposes quality costs on the chain [ 15 ]. Although considering the shelf life losses is in relation to an increase in transportation costs, it worth investing on transportation infrastructure due to less quality loss. Moreover, from a system’s point of view, integrating warehousing and transportation in the supply chain can highly affect the total cost, customer satisfaction and inventory level. Integrated models of providing and storing perishable products help to maximize meeting demands [ 11 ]. Integration of storing and distributing decisions leads to more efficiency than other operational integration [ 16 , 17 ]. Integration of strategic decision making and operational processes appears relevant, especially for such perishable products as agri-foods [ 18 ]. Recently some strategies were studied in supply chain management of perishable products to control the perishability of products which are inventory management [ 19 ], reverse logistic management [ 20 ], pricing [ 7 ], and robust optimization [ 21 ].

Notably, product quality is characterized by the product’s remaining shelf life and thus is time-dependent [ 22 ]. Taguchi described the deviation in performance using the quality loss function that measures the product’s quality loss in terms of the total loss to society due to functional variation and harmful side effects [ 23 ]. For perishable foods, product quality degradation must be identified because it significantly affects consumers’ decisions and retailer profitability [ 22 ]. On the other hand, computing the cost of quality loss for an integrated supply chain allows for exploring the interrelationships among business entities. It enables the supply chain to achieve a minimum total cost by investing in quality and, hence, increasing the overall benefit [ 24 ]. Today, lateral marketing is the most effective way of competing in mature/immature markets, where micro-segmentation and plenty of brands don’t leave any space for new opportunities [ 25 ]. One of problems in the perishable agricultural products’ supply chain is a high quality loss post-harvest, which leads to different quality costs and the customer dissatisfaction. A brief review of the literature reveals that rarely is there any established advanced multi-echelon vegetable supply chain wherein the profit is maximized by considering such features as product quality degradation, quality loss-related costs, and settling lateral markets. Due to this research gap, current study is aimed to maximize the profit of perishable products supply chain considering their related quality costs. The question in this research is how considering both the cost of qualities and the second market in the supply chain network design (SCND) of perishable products can affect the benefits of stakeholders, such as farmers and customers in the supply chain.

The research objective is to formulate a SCND of perishable products by considering different costs of qualities in the supply chain and settling a lateral market and processing the part of perishable products that have not entered the supply chain due to its high level of perishability and enters to the second market be used in specific form satisfying customers, in the mathematical mixed integer linear programming. The current study intends to make affecting decisions in different levels of decision making as: 1) strategic level; locating different centers in the supply chain, 2) tactical level; determining the processing type, and quantities of different products be delivered to the customers, and 3) operational level; selecting a suitable mode of transportation and quantities in the SCND. To address this challenging problem, vegetables, important perishable products, were examined in a case study by first studying the multi-echelon agri-food supply chain (AFSC) based on the post-harvest quality features.

The remainder of this paper is structured as follows: In the next section, a brief overview of related literature reviews on the quality management of perishable agricultural products is given. Section 3 describes the research methodology, a quantitative supply chain modeling approach in a linear programming framework. The case study and sensitivity analysis results in the optimum point are presented in Section 4, the research conclusions in Section 5, managerial implications in Section 6, and future research and limitations in Section 7.

2. Literature review

2.1. agricultural products supply chain.

Customers pay special attention to the quality and safety of agri-foods because they directly affect their health [ 26 ]. This quality can be measured by such different criteria as the purchasability [ 27 ], lifetime (day) left [ 28 ], color [ 29 ], freshness [ 30 ] and light-greenness of vegetables (L. in the Hunter Laboratory) [ 31 , 32 ]. Creating an efficiency-responsiveness balance in quality-based customer-oriented supply chains is worth considering [ 9 ]. The optimal operation strategy is acquired based on product quality [ 6 ]. Organizations that have instituted a system of quality cost measures have experienced dramatic positive results because it translates the implications of poor quality, activities of a quality program, and quality improvement efforts into a monetary language for managers to understand which factors are important in affecting profitability and the consumer need [ 24 ].

Decisions made in the supply chain of perishable products are strategic, tactical and, operational; strategic decisions that have long-term effects on firms are those made on the network design, supply chain network design [ 33 ] and the location of different equipment in the processing, distribution and, hub centers to make the best use of the capacity of the existing facilities [ 34 ]. In the strategic level of decision-making in the perishable products’ supply chain design, different ways to cope with increasing product quality decay can be identified. On the one hand, the network can be centralized to decrease handling time (for each transport to a hub, a fixed handling time is incorporated in the transport time) and hence decay. On the other hand, more hubs can be opened to decrease transport time and decay [ 9 ]. Moreover, technical models are popular and have public applications in harvest programming, product selection, and labor capacity in agricultural products supply chains. Besides strategic and tactical decisions, the supply chain also involves operational decisions for which it is assumed that the former two are already known and sufficient knowledge is available about production, demand, and transportation [ 35 ]. Pasha et al. studied an integrated bi-objective quality-based production-distribution agri-food MILP supply chain model in which profitability is maximized by defining the quality as a function of such decisions as the location of hubs and transportation strategy throughout the supply chain [ 17 ], whereas making decisions in an integrated way will reduce costs compared to individual decisions made at each level [ 36 , 37 ]. Moreover, in the greenery supply chains, De Keizer et al. presented a model in which decisions made on the greenhouse location (strategic) are based on the plant’s lifetime in that location [ 9 ]. As changes in the temperature and enthalpy levels change the food quality [ 38 ], Khazaeli et al. and Rong et. al determined the temperature of distribution centers and deliveries to meet the expectations of different customers as the operational decision-making in a supply chain management [ 39 , 40 ].

2.2. Quality of agricultural products

In most supply chain designs, cost, profit, quality, responsiveness and environment are the general decision-making factors [ 34 ]. Although cost and profit are still the main criteria in almost all quantitative mathematical programming models of the supply chain of perishable agricultural products, in recent years, other criteria, such as product quality [ 9 , 17 , 18 , 41 , 42 ] and environmental protection [ 43 ] have also been considered in some studies. The quality function of perishable agricultural products can be either complex or simple [ 44 ]. It has been shown that, the decrease of a single quality attribute of agricultural products can be approximated by one of the four basic types of mechanism which are zero-order reactions having linear kinetics, Michaelis Menten kinetics, first-order reactions having exponential kinetics, and autocatalytic reactions with logistic kinetics [ 45 , 46 ]. For the concept of keeping quality, it is convenient to assume zero-order reaction kinetics [ 28 ], and mostly the Michaelis Menten kinetics reduces to a linear one in the initial region of decay, which is the most important in quality assessment [ 47 ]. Therefore, the quality variable of vegetables in the initial region of decay can be considered in a widely used equation, in which the quality function changes by the time linearly. It is shown in Eq 1 .

Where, Q 0 is the initial quality, t is time and k is a degradation rate. In a dynamic environment, the well-known Arrhenius equation shows that the degradation rate (k) depends on the activation energy of the material, and the environmental factors [ 28 , 48 , 49 ].

The perishable products’ quality model shown in Eq 1 has been frequently used to capture the degradation of food products over time. For example, in the grocery retail chain, Wang and Li presented a pricing model to maximize food retailer’s profit in a dynamically identified food shelf life by using Eq 1 [ 50 ]. Chen and Chen proposed an on-site direct-sale dynamic supply chain inventory model, considering time-dependent quality losses for perishable foods [ 22 ]. Lejarza and Baldea presented a closed-loop, feedback-based control framework, that employs real-time product quality measurements for optimal supply chain management [ 51 ]. Moreover, Xu et al. presented a real time decision support framework to mitigate the quality degradation in the journey of agricultural perishable products from farm to the retailer in the supply chain based on the Eq 1 [ 52 ].

Generally, cost, benefit, and quality factors are the most important factors that are to be optimized in network designs. Mostly, agri-food should make a logical balance between two topics, which are the price reduction and the customer service improvement [ 38 ]. In the field of multi-objective supply chain network design, De Keizer et al. and Khazaeli et al. showed that, the quality of agricultural products causes cost in the supply chain’s network [ 18 , 39 ]. A review of quantitative supply chain research on the perishability of agri-food by considering related quality costs is summarized in Table 1 .

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0303054.t001

2.3. Research gaps and contributions

Due to the importance and necessity of developing SCM from a larger perspective to provide a win-win situation for each participant in the supply chain, in this paper, we aim to develop a novel mathematical model to design a supply chain network, based on quality function elements in the vegetables’ sector. The summary of the literature review outlines the gaps in the literature as follows:

• Despite the importance of the cost of qualities in designing supply chains due to the perishability of the products, the cost of quality concept has not been widely incorporated by researchers in the design of agricultural products’ supply chains.
• No research has paid attention to the lateral market to look at the quality problems from the side covering some target customers.
• Few researchers have considered the benefits of several stakeholders of the agricultural supply chains simultaneously. The stakeholders in agricultural Products’ supply chain are consumers, farmers, the environment, and society.

The proposed SCND is a multi-product, multi-echelon model with exact (certain) demand that makes decisions at strategic, tactical, and operational levels. It has focused on “quality” by considering the quality deterioration which is time-dependent in the initial region of decay, moreover, by defining costs of quality degradation in the quality-cost functions. Features that differentiate the present research from others are displayed in the last row in Table 1 . As previous researches have demonstrated, traditional supply chain of agri-food is unstructured, which generally leads to low quality and low benefit of agricultural products, the presented research is developed, in which the main contributions are as follows:

• ✓ Providing a network design model for an integrated multi-level supply chain of perishable products wherein profit is optimized by considering quality decay aspect of the products.
• ✓ Optimizing the profit of the supply chain of perishable products considering different quality costs for them due to unmet demand, product waste and reduced revenue of low-quality products.
• ✓ Introducing a strategy of selling perishable products to lateral markets before letting products enter the chain to prevent the production of low-quality products along it.
• ✓ Enabling the purchase of the farmer’s total agricultural product above the contract ceiling due to unpredictable production to prevent waste production and its scattering in the environment.
• ✓ Introducing a strategy of producing semi-processed, low-quality products (from those that did not enter the chain) to meet part of the market demand for lower-quality lower-price products.

The developed model is a four-echelon supply chain of perishable agricultural products in which the time-dependent quality of the products is considered. In addition, a lateral market is considered in the designed supply chain that does not stand higher than vertical marketing and completes the primary market.

In the end, the developed model is applied to a case study of a firm in the agricultural products industry with four echelons of farm-processing-distribution-customer centers. The vegetables selected as candidates for the present supply chain network design are Yarrow , Borage flower , and Melisa , due to their priority in agricultural studies and their application in various industries [ 55 ].

Although there are some studies done to minimize quality losses of perishable products by multi-objective problem-solving approaches [ 17 , 19 , 20 , 21 , 39 ], the programming in the present research is done as a single objective problem solving by profit objective function underlying quality loss costs.

3. Problem description and formulation

From the perspective of the research approach, this research is quantitative, done as a mathematical mixed integer linear programming (MILP) modeling with the objective function of profit by considering the cost of quality factors of products in the multi-echelon perishable products’ supply chain. It is applicable to the related supply chains. It focuses on an integrated multi-product SCND of agricultural products that provides, processes, stores and distributes materials. It considers customer demands and sells the farmers’ in-excess products to the second market. The designed model was solved using GAMS 24.1.2 software by exact solution method by epsilon-constraint. The model is validated by applying it in the case study of a multi-vegetable supply chain of a firm in a fertile area in Iran country. The designed supply chain of the firm is shown in Fig 1 .

https://doi.org/10.1371/journal.pone.0303054.g001

First, products through related contracts and in-excess products are bought from farmers in the study area. In the second echelon of the proposed supply chain, some or all of the purchased products are processed at related centers resulting in different degrees of product quality. Third, the products in the former echelon are stored in cool storage centers until being distributed and fourth, they are sold to wholesalers. Another part of the purchased products are transferred to the second market as lower quality products in different industries (tea bags, spices in food, etc.). Different road modes of transport are used between different echelons of the supply chain.

The modeling makes decisions at different echelons of the supply chain. Decisions made are (1) selecting farms and the quantity of raw products to be purchased from each of them, (2) the quantity of products sold to the second market, (3) the number of processing and storage facilities to be settled in the supply chain, (4) product flow and the vehicles to carry out the transportation between the active facilities i. e. from farms to wholesalers and (5) assignment of processing facilities to the products. They are made based on minimizing the total cost of the supply chain design considering the cost of qualities. In the following, assumptions and the modeling are described.

3.1. Assumptions

• The location of production centers is specified.
• Capacities of the processing centers, and also storage centers are determined.
• Customer demand for each type of processed product is pre-determined.
• Shortage to customer demand is allowed.
• The quality of products post-harvest in the supply chain is considered time-dependent.
• The deterioration rate of each product is considered specific, based on the activation energy of the material.
• The approach of quality costs is considered in measuring the quality of products in the objective function modeling.
• The transportation speed of each mode is assumed uniform.
• In-excess products are sold to the second market.
• Over-time quality loss-related cost, unmet customer demand and product waste are considered as quality costs.
• The cost of the lost product quality equals the price drop in proportion to the quality drop by a factor of ten (The coefficient (10) is proposed by experts based on pairwise comparisons of cost and quality criteria).
• The quality cost of the customer credit for each demand equals the revenue lost due to not meeting one unit demand.
• The quality cost of the product waste equals the revenue from the product sales not realized, causing that product to enter the environment as waste.
• Products are bought from farmers: 1) at a price for first-grade products based on the amount in the contract and 2) at a price for second-grade products for those over that in the contract (According to experts, the purchase-price-drop coefficient is 0.3 in the market).
• Products are sold to the supply chain customers at a price for first-grade products and those outside the supply chain are sold in the second market at a price for second-grade products (According to experts, the sell-price-drop coefficient is 0.3 in the market).

The mathematical model, its objective and its constraints are presented in the following.

3.2. Mathematical modelling

Symptoms used in the model consist of sets, related indexes, parameters and variables, objective functions and constraints, are as follows:

Sets and indexes.

https://doi.org/10.1371/journal.pone.0303054.t002

Parameters.

https://doi.org/10.1371/journal.pone.0303054.t003

https://doi.org/10.1371/journal.pone.0303054.t004

Profit objective function and constraints are described as follows:

Profit objective function.

The objective function is defined to maximize the supply chain profit. It is equal to the revenue from both, selling products to customers and the second market minus the total supply chain and quality costs ( Eq 2 ).

Revenue consists of: 1) that obtained by selling the supplied demanded product, which is equal to the unit price of the sold product multiplied by the customer met demand; the latter equals the amount supplied in the supply chain minus that over the customer demand, and 2) that obtained by selling: a) the supply chain-decided products and b) in-excess products sent to the second market which is equal to the price of each unit of the low-quality product multiplied by the amounts in a and b.

Costs relate to: 1) purchasing high-quality (on contract) and low-quality (in-excess) products from farmers (with their own related prices), 2) locating processing and storage centers, 3) processing operations, 4) storing products in storage centers, 5) different supply chain distances (ton-km), 6) ordering different transportation modes, 7) revenue lost due to reduced product quality, 8) credit lost due to unmet demand and 9) unsold wasted product.

Constraints

Quantities equations..

Constraints (3) to (10) ensure the product weight in different supply chain steps—from the farm to the customer (considering the amount of the farm production). Constraint (11) addresses in-excess low-grade products to be sold in the second market; these are produced, but not delivered to customers through the supply chain for different reasons.

Constraints (12) to (15) indicate that quantities of processed and stored products, respectively, in activated processing and storage centers are determined based on the capacities of these centers. If centers are not active, the quantities would be zero.

Travel time in the supply chain equation.

Constraint (16) indicates that the vehicles used in the transportation system of the supply chain have uniform speeds.

Number of vehicles.

Constraint (17) to (20) determines needed vehicles in different modes to transfer materials in different supply chain steps and the whole supply chain assuming full-capacity active vehicles.

Shortage and extra quantities constraints.

Constraints (21) to (23) determine the in-excess and shortage amounts.

Constraints (24) and (25) illustrate non-negativity and binary variables.

4. Case study

In this section, we implement the proposed model in an Iranian raw and processed vegetable products’ company, the Razian Company, as a case study. Iran country has been bestowed with a wide range of climate and physio-geographical conditions and as such is most suitable for growing various kinds of vegetables, its production of vegetables is increasing. Moreover, agricultural products are profitable fields for investment. Since Iran possesses a large variety of flora with manufacturers, in equal measure, analysis of the working of the vegetable market is critical [ 55 ]. There is an apparent shortage of related supply chain in Iran country. The goal of the case study is to evaluate the efficacy of the proposed model under real-world conditions and to address the needs of the firm in question. The case study used a four-echelon SCND, and materials were supplied, processed, and stored (echelons 1–3) in the firm area (origin) while the last-level centers were located all over the country; in addition, a center was established as a second market to collect the in-excess products, as shown in Fig 1 . The mentioned lateral market imposes no costs on the supply chain because it is closest to farms, and customers pay the transportation costs.

At first, the firm seasonally provided the vegetables from the suppliers. Suppliers were specified and contracted in advance in fertilized source centers (i = 4) of selected vegetables (n = 3). The farm centers were, in Kaboudrahang , Razan , Nahavand , and Malayer , and the vegetable products were Yarrow , Borage flower , and Melisa . Secondly, the firm used the related processing on vegetables, or the products remained raw. There are potential processing center (j = 5) candidates in the case study. Thirdly, the firm stored the products in the storage centers for packaging. There are potential storage center (k = 5) candidates in the case study. The five potential processing and storage center candidates were Kaboudrahang , Razan , Nahavand , Malayer , and Asadabad . Finally, the firm delivered the demanded products to the customer centers. The customers were trade representatives of each province all over the country (l = 30). Due to the importance of the case study data for the application of the presented model, some were obtained from the enterprise resource planning (ERP) of Razian company [ 56 ]. In addition, data on fixed and variable costs of different transportation modes were obtained from the recent case study research done in Iran [ 39 ]. Data on the price of different raw and processed vegetable products were gathered from the statistics of the Ministry of Agriculture [ 57 ]. Details of the most critical data of the case study are presented in the table in S1 Table in the supporting information.

The designed mathematical mixed integer linear programming (MILP) model was implemented and solved using GAMS 24.1.2 software and an Intel 2.13-GHz processor by exact solution method by epsilon-constraint. The designed network, product type, amount (tons) produced and sent to, e.g., Tehran (Capital), the transportation mode at different supply chain levels, and amount (tons) delivered to the second market are shown in Fig 2 .

(R indicates Yarrow product, C indicates Borage flower product, T indicates Melisa product). Optimally, 564 trailers and 33068 trucks were needed in the designed supply chain network. Generally, in presented agricultural products’ supply chain, some products have high quality-loss rates as well as demands for distances far from the cultivation center. This leads to a long post-harvest time for the product to reach the customer and, hence, a high rate of quality loss and a drop in the product price. This fact makes the supply chain decision maker set the lateral market due to not delivering those products to those customers and hence delivering them to the second market. It is considered newly in the present research due to make quality loss of products in the supply chain, the less, hence the profit the more.

https://doi.org/10.1371/journal.pone.0303054.g002

4.1. Results

As shown in Fig 2 , in the optimum point of maximizing profit by considering quality costs in perishable vegetables supply chain in the proposed MILP model, the processing, storage, and distribution centers are settled in similar locations, spatially. It leads to set process-storage-transfer type of hub centers, in compliance with the supply chain network proposed by Khazaeli et al. [ 39 ]. Out of 5 potential processing and storage/transfer centers, the model found all for the supply chain No. of facilities based on the center capacity and its setup costs (related parameters are listed in the table in S1 Table ). It is similar to the model proposed by De Keizer et al., in which to decrease transport time and hence decay in related supply chain design, the centers were decentralized, [ 9 ]. Therefore, more hubs were opened.

The model determined the amount and type of the delivered products between all supply chain levels, by the supply chain programming, and provided the information on the product (ton) if it was possible to supply to meet the customer demand. The details of provided products are presented in the table in S2 Table in the supporting file. Here, the supply chain management decides not to offer part of products to the customer and sells them at a second-grade price to the second market to maximize the chain profit by minimizing the quality loss-related cost along the chain (highlighted as unmet demands in the table in S2 Table ). In such a case, saving the low-quality cost of the perishable product will bring more revenue for the chain.

The model also selected the center-to-center transportation mode considering the vehicle speed to reduce time and, hence, the quality degradation and transportation costs. The table in S2 Table in the supporting file lists the number of each vehicle type required to transfer products. In result, the supply chain used trucks about 60 times more than trailers because of being faster. It used trailers, although with higher order costs, only in long distances, e.g., from storage centers to customer centers due to their more than ten times more capacity than trucks which led to fewer vehicle orders and, hence, less vehicle order costs. As shown in Fig 2 , in all supply chain steps, except the last, the model suggests using trucks because of their higher speed than trailers and their less order costs than trailers (The vehicle-related parameters are shown in the table in S1 Table ).

In this chain, some produced, but supply chain-decided undelivered to the supply chain were sold to the second market with price of high-grade products. The products produced more than that guaranteed in the farmer’s purchase contract, were sold to the second market with a much cheaper price (0.3 that of high-grade products). Both, amounted to 1820 tons of product Yarrow in Razan , 10020 tons of product Borage flower in Nahavand and 93.5 tons of product Melisa in Malayer and Nahavand , all were delivered to the second market.

Demands for all types of products were met except for fresh products, for which the demands were responded in centers closer to the previous echelon due, maybe, to their higher corruptibility and quality-loss rate than other types of products (The table in S1 Table in the supporting information lists the perishability rate of each processed products than the fresh one) and, hence, a price decline that makes them uneconomical to deliver to customers.

4.2. Benefit and quality loss of the products in the supply chain

In the designed supply chain, as shown for the optimum solution point in Fig 3A , the revenue and total cost are, respectively, 27.3 and 18.5 million USD; therefore, the benefit is 8.8 million USD. The final product quality and quality loss in the supply chain are 28,357 and 643 (Unit of quality), respectively ( Fig 3B ).

(a). Profit/ cost of the SC designed. (b). Final quality/ quality losses in the SC design.

https://doi.org/10.1371/journal.pone.0303054.g003

The revenue of the supply chain (27.3 million USD) is due to: 1) selling the chain-demanded supplied products 21.9 (Million USD), 2) selling products not supplied to the chain and sold to the secondary market based on the chain management decision 0.08 (Million USD) and 3) selling products supplied more than that specified in the contract 5.32 to the secondary market (Million USD) ( Fig 4A ).

(a). Supply chain revenue parts. (b). Farmers’ revenue parts.

https://doi.org/10.1371/journal.pone.0303054.g004

The revenue of farmers as main stakeholders, is 12.5 million USD, which goes to them by selling: 1) contract-demanded products delivered to the supply chain (10.2 million USD), 2) contract-demanded products supply chain-decided undelivered products (2.07 million USD) and 3) in-excess-of-contract products to the second market (0.24 million USD) ( Fig 4B ).

Total supply chain benefit (8.8 million USD) comes from supplying products to customers considering the demand (5.7 million USD) and products to the second market (3.1 million USD). In addition, the total revenue of farmers is (12.5 million USD) ( Fig 5 ).

https://doi.org/10.1371/journal.pone.0303054.g005

4.3. Supply chain cost breakdown considering quality cost and other supply chain costs

The supply chain cost (18.5 million USD) consists of 8 elements, among which purchasing, including buying raw materials for the supply chain (10.2 million USD) and in-excess materials (2.3 million USD) for selling to the second market, is the costliest, and revenue lost due to reduced product quality along the chain (5.4 million USD) stand next. Other costs in the case studied, in the order of higher values, include quality cost of unmet demand of fresh products in long distances (0.38 million USD), processing (0.1 million USD), logistic transportation (0.07 million USD), storage (0.03 million USD), establishing facility centers (0.02 million USD); product waste has zero cost. The percent share of total costs, including those of the network, supply chain logistics and quality costs is shown in Fig 6 .

https://doi.org/10.1371/journal.pone.0303054.g006

As shown in Fig 6 , 29% of the costs (5.4 million USD) in the supply chain of perishable product supply relate to the revenue lost due to the product quality loss by unmet fresh products. On the other hand, the quality cost of unmet demand for fresh products in long distances is 0.38 million USD. The most part of the mentioned costs are compensated by revenue earned by selling these products to the second market by 5.32 million USD.

The designed supply chain has other profits, which are: 1) preventing low-quality products from being produced at the request of the chain customers and 2) sending products produced over that specified in the contract (due to unpredicted agricultural products produced) to the secondary market and, hence, preventing them from entering the environment as waste.

The model accuracy was verified by changing its parameters and examining its responses to the changes. The validity of the proposed model has also been confirmed by comparing the results of the present SCND, with a vicinity secondary market ( Fig 2 ), and those of the existing chain, without such a market. Related experts have evaluated the proposed model, validated it, and concluded that the chain profit has increased due to its reduced quality costs. The sensitivity analysis is presented to evaluate the effect of changing some parameters on variables and the objective function, in the following.

4.4. Sensitivity analysis

Parameters to which model responses investigated in reaction, are the reaction rate of products and speed of different transportation modes as they relate to the quality loss of products and cost of supply chain during the time after harvest. Model responses to changes have been analyzed and explained orderly in the following:

Quantity of products and revenue versus reaction rate (k) of products.

The quality loss rate (k) of different products varies depending on their reactivity, and processing reduces this rate in fresh products. To prevent the quality cost resulting from the products’ quality loss and price decline, the chain provides just part of the fresh product demands, not far than a specific distance (The table in S2 Table in the supporting information). When the quality loss rate (k) changes, the amount of the customer-demanded met products as well as those not enter the chain change too; the latter are processed at the beginning of the chain immediately after they are purchased and then sold as low-grade products to the second market. The ratio of the customer-offered to customer demand for different types of products and the amount sold to the second market were examined considering the product quality loss rate (k). The effects of the quality loss rate (k) on the stakeholders’ profit and revenue have also been studied. A summary of the results is shown in Fig 7 .

(a) Changes in quantities of products. (b) Changes in revenue/ profit of farmers and SC parts.

https://doi.org/10.1371/journal.pone.0303054.g007

In the current chain, 96% of the demand for fresh products is met, and the rest is sold to the second market. As shown in Fig (7a), an increase in the quality loss rate (k) reduces the amount of fresh products. It increases the amount of those sold to the second market and supplied before entering the chain due to a sharp drop in fresh products, undesirability for customers, quality loss and price drop in the chain over time. A more increase in the mentioned rate (twice more) reduces the meeting rate of the customer-demanded fresh product from 96% to 35%; products sold to the second market increase from 64% to 100%, and the processed, dried and essence products, fully met, remain unchanged. Moreover, as shown in Fig (7b), an increase in the rate of product quality loss (k) does not reduce the farmer revenue, because the contract-specified products are bought from farmers at the original price.

As shown in Fig (7b), an increase in the quality-loss rate (k) of perishable products reduces the chain profit because some of these products, purchased from the farmer at the original contract price, do not enter the chain and are sold in the secondary market at lower prices (here, 0.3 times the contract price). Therefore, considering higher quality-loss rates (k) in the SCND will result in sharper reduced profits for the supply chain and the secondary market.

Supply chain cost/revenue versus speed of vehicles (v) changes.

Under present conditions and the speed (v) of the current fleet in the case study (V trailer = 80 and V truck = 100 (km/ hour)), the model meets 96% of the demand for fresh products and all that for the dried and essence products; Faster fleet speeds enable more demands to be met ( Fig 8A ).

(a) Change in quantities of products (b). Change in number of vehicles (c). Change in cost/ revenue.

https://doi.org/10.1371/journal.pone.0303054.g008

Increasing the speed (v) up to 50% will help the demand for fresh products to be met up to 100% and that for other products stays constant at 100%; however, reducing it up to 80% will not change the amount of processed products, but will cause the amount of the freshly supplied products to reach about 20% ( Fig 8a ).

As shown in Fig (8b), increasing the speed (v) leads to more use of faster vehicles (here, trucks). As shown, increasing the speed (v) to 100% will increase the number of needed trucks by 2%, but will not change the number of needed trailers. As mentioned earlier, trailers are used for outside-province long distances to respond to customers located far from the supply center. This will result in lower total long-distance transportation costs than trucks due to lower ton-km costs despite higher-order costs (The table in S1 Table in the supporting file).

Fig (8c) shows the minor increases in transportation costs and a noticeable reduction in the unmet-demand lost revenue due to the increased vehicle speed (v). Increasing the speed (v) up to 100% will increase the transportation costs by 11%, but reduces the unmet-demand lost revenue by 100%. This increased transportation cost of 0.008 million USD will prevent a revenue loss of 0.34 million USD, which is quite a significant figure.

It demonstrates that increasing the speed (v) will increase the number of vehicles, hence increase the transportation costs and the responded demand and ultimately prevent the revenue loss. Hence, increasing the speed (v) will lead to increased costs and enhanced chain revenue ( Fig 9 ).

https://doi.org/10.1371/journal.pone.0303054.g009

Since the increased revenue is greater than the increased cost, increasing the speed (v) will increase the chain profit; increasing the speed (v) up to 100% will increase the profit by 2.9 million USD (increased by 100%). Increasing the speed (v) will not affect the farmers’ revenue. The results comply with the findings of Patidar and Agrawal in research on traditional agricultural chains in India, in which the transportation share in supply chain costs reached about 92% in the distribution sector [ 12 ]. It shows the importance of transportation strategies in this sector.

A comparison of designed supply chain with traditional supply chain in the case study is demonstrated in Fig 10 .

https://doi.org/10.1371/journal.pone.0303054.g010

Regarding demands for fresh products, as shown in Fig 10 , their amounts in the two cases (with and without a secondary market) are 10868 and 10068 tons, respectively, showing an increase of about 0.08 times; this leads to a quality increase and, hence, customer satisfaction and profit increase. In both cases, demands for dry and essence products are fully satisfied. The comparison between the results of the present supply chain design in the case study and the results with the lateral market indicates that a lateral market in the supply chain will increase the chain profit and farmer income. However, in the optimal mode in this case study, they are increasing from 9.7 and 5.85 to 12.5 (about +50%) and 8.8 (about +20%), respectively.

The results show that newly designed supply chain is applicable in the field of the perishable products supply chain. It confirms the necessity of supplying innovative products of perishable ones such as processed agricultural products to meet new customer needs in a lateral market to the competitiveness. It complies with the findings of Malynka and Perevozova, who proposed the lateral markets in mature and immature markets in the brand creation process [ 25 ].

4.5. Managerial insight

Some lessons and insights for managers are as follows.

• All of the contract products are supposed to enter the chain; if not (for different reasons, e.g., chain management decision), some are sent to the second market, the presence of which prevents the produced products and resources (land, labor, energy, etc.) spent for those not entering the chain (for different reasons) from being wasted.
• Not considering a second market for fresh perishable agricultural products e. g., vegetables will lead to ignoring the post-harvest quality loss-related costs.
• Increasing the fleet speed is of great benefit to all the chain stakeholders including customers, chain management and environment because, on the one hand, it leads to increased response to the customer demand for more fresh products and selling customer-demanded high-quality products at prices proportionate to their grades will increase the chain profit and, hence, the total revenue, on the other, it prevents environmental pollution by letting more products to enter the supply chain and preventing wastes to be generated.

5. Conclusions

In this paper, a new approach is presented to optimize a logistics network design for distributing multiple products that are highly perishable and sensitive in quality and health of products to consumers, such as vegetables. Echelons of supply chain design include supply, processing, storage and customer. Considering the unpredictable amount of production of agricultural products and their perishability post-harvest, the second market which is accompanied by processing technologies to produce innovative products from the perishable products has been considered in the related supply chain network design, beside the main chain. The supply chain network design has been done based on maximization of profit by considering different quality costs in the supply chain. Quality costs include those due to: 1) quality-loss price-drop, 2) product waste and 3) losing credit with the customer for not meeting the desired demand. Since the chain integrity of these types of products is essential, the integrated one considered in this study is managed by the chain management deployed in the product supply center. Programming has been done based on the maximization of profit by the MILP model considering the quality costs of products in the supply chain. To evaluate the modeling, a case study was used on three vegetables cultivated and harvested in a fertile area in Iran country in September 2023. The model was subsequently validated by multiple sensitivity analyses performed on some of the essential parameters that had a greater effect on the results.

In this supply chain design, as it is demonstrated in Fig 2 , different decisions have been made at strategic, tactical and operational levels in order to maximize profit by considering the costs of quality in the supply chain. Decisions made are on the location of processing centers and storage centers, and product flow allocations in the designed supply chain. Moreover, the model decides on the operations of processing after harvest, such as drying and extracting, which leads to mitigated products’ quality decay. In the next echelon after processing in the supply chain, there are storage centers in which products are stored to be distributed to the retailers. In the tactical level of decision making, the presented model decides on the allocation of farmers to processing centers and also processing centers to storage centers, moreover the allocation of storage centers to the retailers as the customers, also the number of products produced by farmers enters the supply chain and remains to be supplied to the second market and not deployed in the supply chain is determined. In the operational level of decision-making, the quantity of products and mode of transportation between different levels in the supply chain have been determined to meet the customers’ needs.

Results of this research were compared with those of related recent studies [ 9 , 12 , 25 , 39 ]. The comparisons demonstrated good conformity, especially, in compliance with recent research in lateral besides vertical markets [ 25 ]. It seems innovative second markets are required to meet other parts of demand. Settling the lateral market seems strategic, especially in perishable products. The lateral market regulates supply and demand and helps reduce the quality-loss-related costs of the chain and responds to another part of the market that has specific customers.

6. Managerial implications

The proposed model is generic and can help managers in food quality, customer service, and other related operations as a tool to assist in decision-making in the perishable agricultural products supply chain. Specially, the research done can have the following applications:

• The decisions stemming from the presented model are determined based on the products’ degradation pattern to maximize its quality. The decisions include supplier selection, supply chain design, processing technology deployment, and vehicle deployment.
• A second market besides the chain and not higher than the vertical one in supply-based products such as vegetables, may result in a considerable increase in the chain profit without changing the resources, no reduction in the farmer income for unpredictable amounts of agricultural products production, and no wasted products preventing the environment from being polluted.
• The usage of lateral marketing is relevant, as it is the most effective way of competition in mature markets. However, when chains are designed for perishable products for optimum profit, the demand for some products with high quality-loss rates is not met due too long distances from distribution centers (if it is met, high-quality costs will be imposed on the chain). The related products are processed for secondary customers and delivered to them in the second market.
• Increased perishability rate of agricultural products reveals the effects and necessity of second markets next to the chain.
• Although, high-speed shipping fleets are expensive, using them will increase the chain profit because they reduce the post-harvest travel time and, hence, reduce the quality-loss-related costs of perishable products significantly. This way, the demands of more customers are met, customer credit costs will be prevented and the supply chain management and customers will both be benefitted. By applying the proposed model in the perishable agricultural products supply chain, the products are sold in the second market to meet the lateral part of the market.

As a result, different stakeholders such as farmers, customers, the environment, and the owner of the supply chain may benefit from the new supply chain network design.

7. Future research and limitations

Our framework is limited in some respects. With that said, this modeling limitations serve as a platform for extending it in future researches. One primary limitation of the presented model is that it does not consider the uncertainty in the amount of customers’ demand. Therefore, the proposed model does not work for the problem in uncertain conditions. Also, the proposed model in this research has been solved by the exact-type solving method of mathematical programming, which is proper for solving the small size of problems such as the studied case. Considering the limitations above, using mathematical models by uncertainty considerations in the supply chain parameters and applying meta-heuristic methods to solve medium and large-sized problems are suggested in the future research. From the managerial perspective, the presented research works by the assumption of that upstream suppliers, freight transportation, processing centers, and storage facilities are integrated and it needs to build alignment between their organizations to deploy the solutions proposed by the output of the proposed framework. For these efforts to be successful, for future research, it is suggested to study how to cooperate all parties involved in the supply chain, and design the coordination infrastructure in the supply chain to yield the positive effects of proposed supply chain network design, in practice.

Supporting information

S1 table. parameters of case study network design..

https://doi.org/10.1371/journal.pone.0303054.s001

S2 Table. Quantity (tons) of each product delivered to customers and number of vehicles in logistics of designed SC.

https://doi.org/10.1371/journal.pone.0303054.s002

Acknowledgments

The authors are indebted to Mr. Ja’fary, the manager of “ Razian” Co. ( https://razian.co/ ), for his invaluable help to gather data in the case study. Also, the authors are grateful to the two anonymous referees for their valuable comments, which have led to significant improvements in this paper.

• View Article
• Google Scholar
• PubMed/NCBI
• 8. van der Vorst J. G., Effective food supply chains: generating, modelling and evaluating supply chain scenarios. Wageningen University and Research, 2000.
• 46. Varoquaux P. and Wiley R. C., “Biological and biochemical changes in minimally processed refrigerated fruits and vegetables,” in Minimally processed refrigerated fruits & vegetables, Springer, 1994, pp. 226–268.
• 48. Chang R., “The second law of thermodynamics,” Physical Chemistry with Applications to Biological Systems, New York: Macmillan, pp. 129–163, 1981.
• 55. Iranian National Standardization Organization, “Iran Good Agricultural Practices (IRAN GAP)–Control points and compliance criteria for medicinal and ornamental plants–General requirements,” INSO, Tehran, 2015.
• 56. https://razian.co/

Comparative Study of 24 Publications on the Applications of AI, Robots, and Automation Frameworks on Quality Management in Construction Sites

Description.

The dataset presents a comprehensive review and analysis of the existing literature on the integration of advanced technologies in construction quality management. This dataset encompasses 24 peer-reviewed research papers, each scrutinizing the application of AI, robotics, and various automation frameworks within construction sites, with a focus on quality management processes. The data includes detailed information extracted from each paper, such as the title, authors, year of publication, country of origin, research methods used, technologies investigated, benefits, drawbacks, and the overall reception of the findings within the industry. The dataset also covers the application level of these technologies, the frameworks they relate to, key performance indicators (KPIs) and metrics developed, and their reported accuracy. This dataset is valuable for researchers and industry professionals who are interested in understanding the current state of technology integration in construction quality management. It highlights the advancements, challenges, and gaps in the application of AI, robotics, and automation within the construction sector, providing a critical resource for future studies and practical implementations. The data also reflects on the practical implications of these technologies, offering insights into their potential impact on the construction industry's quality management practices.

Steps to reproduce

The primary databases utilised were Web of Science and Emerald, selected for their coverage of high-quality academic literature. The search was conducted using carefully crafted Boolean phrases to ensure that all pertinent studies were captured. These phrases included combinations of keywords such as "AI," "artificial intelligence," "robots," "robotics," "automation," "construction quality management," and "construction defect detection." The search aimed to cover all relevant literature up to the time of the study. Inclusion and Exclusion Criteria Studies were included if they: Discussed the application of AI, robotics, or automation frameworks specifically within the context of construction quality management. Provided empirical data, detailed case studies, or in-depth discussions that were based on actual construction site scenarios. Were published in peer-reviewed journals or recognised conference proceedings to ensure academic credibility and methodological rigour. Studies were excluded if they: Lacked sufficient data or clear methodologies, making it difficult to assess their findings or replicate their results. Focused solely on theoretical concepts without practical application or empirical validation within construction sites. Were not directly related to the construction industry, such as those focused on manufacturing or unrelated industries. Phases of Data Collection and Analysis 1. Initial Screening: During this phase, the titles and abstracts of all identified studies were screened to exclude any articles that were clearly irrelevant to the research topic. This step ensured that only studies potentially relevant to the application of AI, robotics, and automation in construction quality management were selected for further review. 2. Full-Text Screening: In this phase, the full texts of the selected studies were reviewed in detail. The studies were assessed based on the inclusion and exclusion criteria, with a focus on the presence of empirical data and practical application. This phase was crucial in ensuring that only studies with a direct and relevant contribution to the topic were included in the final analysis. 3. Data Extraction: For each study that passed the full-text screening, key information was systematically extracted. This included details on the technologies investigated, the benefits and drawbacks reported, the level of application within the quality management framework, and the metrics or key performance indicators (KPIs) used. This structured data extraction allowed for a comprehensive comparison across studies, facilitating the identification of patterns, trends, and gaps within the literature.

Institutions

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Overcoming barriers: a comprehensive review of chronic pain management and accessibility challenges in rural america.

1. Introduction

2. materials and methods, 3.1. economics of care, 3.2. the role of the clinician and healthcare systems, 3.3. the impact of race, ethnicity, and cultural values, 3.4. opioids.

4. Discussion

4.1. challenges in managing chronic pain in rural settings, 4.2. current strategies for pain management in rural areas, 4.3. gaps in research and practice, 4.4. innovative approaches and future directions, 5. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Rikard, S.M. Chronic Pain among Adults—United States, 2019–2021. MMWR. Morb. Mortal. Wkly. Rep. 2023 , 72 , 379–385. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Zis, P.; Daskalaki, A.; Bountouni, I.; Sykioti, P.; Varrassi, G.; Paladini, A. Depression and chronic pain in the elderly: Links and management challenges. Clin. Interv. Aging 2017 , 12 , 709–720. [ Google Scholar ] [ CrossRef ]
• Fishbain, D.A.; Lewis, J.E.; Gao, J. The Pain Suicidality Association: A Narrative Review. Pain Med. 2014 , 15 , 1835–1849. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Khalid, S.; Sambamoorthi, U.; Umer, A.; Lilly, C.L.; Gross, D.K.; Innes, K.E. Increased Odds of Incident Alzheimer’s Disease and Related Dementias in Presence of Common Non-Cancer Chronic Pain Conditions in Appalachian Older Adults. J. Aging Health 2022 , 34 , 158–172. [ Google Scholar ] [ CrossRef ]
• Ditre, J.W.; Zale, E.L.; LaRowe, L.R. A Reciprocal Model of Pain and Substance Use: Transdiagnostic Considerations, Clinical Implications, and Future Directions. Annu. Rev. Clin. Psychol. 2019 , 15 , 503–528. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Rafferty, A.P.; Luo, H.; Egan, K.L.; Bell, R.A.; Gaskins Little, N.R.; Imai, S. Rural, Suburban, and Urban Differences in Chronic Pain and Coping among Adults in North Carolina: 2018 Behavioral Risk Factor Surveillance System. Prev. Chronic Dis. 2021 , 18 , E13. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Jensen, L.; Monnat, S.M.; Green, J.J.; Hunter, L.M.; Sliwinski, M.J. Rural Population Health and Aging: Toward a Multilevel and Multidimensional Research Agenda for the 2020s. Am. J. Public Health 2020 , 110 , 1328–1331. [ Google Scholar ] [ CrossRef ]
• Cheeseman Day, J.; Hays, D.; Smith, A. A Glance at the Age Structure and Labor Force Participation of Rural America. United States Census Bureau. 2016. Available online: https://www.census.gov/newsroom/blogs/random-samplings/2016/12/a_glance_at_the_age.html (accessed on 8 February 2024).
• Breuer, B.; Pappagallo, M.; Tai, J.Y.; Portenoy, R.K. U.S. board-certified pain physician practices: Uniformity and census data of their locations. J. Pain 2007 , 8 , 244–250. [ Google Scholar ] [ CrossRef ]
• Smith, T.J.; Hillner, B.E. The Cost of Pain. JAMA Netw. Open. 2019 , 2 , e191532. [ Google Scholar ] [ CrossRef ]
• Dahlhamer, J.; Lucas, J.; Zelaya, C.; Nahin, R.; Mackey, S.; DeBar, L.; Kerns, R.; Von Korff, M.; Porter, L.; Helmick, C. Prevalence of Chronic Pain and High-Impact Chronic Pain among Adults—United States, 2016. Morb. Mortal. Wkly. Rep. 2018 , 67 , 1001–1006. [ Google Scholar ] [ CrossRef ]
• United States Census Bureau. In Some States, More Than Half of Older Residents Live in Rural Areas. Census.gov. Available online: https://www.census.gov/library/stories/2019/10/older-population-in-rural-america.html (accessed on 29 January 2024).
• Danilov, A.; Danilov, A.; Barulin, A.; Kurushina, O.; Latysheva, N. Interdisciplinary approach to chronic pain management. Postgrad. Med. 2020 , 132 , 5–9. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ramsay-Seaner, K.; Letcher, A.; Hoffman, M.S.; Anderson, E.; Heckmann, C. Perceptions of prescription opioid use among rural farming and ranching communities: Preliminary implications for outreach and treatment. Subst. Abuse 2022 , 43 , 1245–1250. [ Google Scholar ] [ CrossRef ]
• Nguyen, L.H.; Dawson, J.E.; Brooks, M.; Khan, J.S.; Telusca, N. Disparities in Pain Management. Anesthesiol. Clin. 2023 , 41 , 471–488. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Eaton, L.; Langford, D.; Meins, A.; Rue, T.; Tauben, D.; Doorenbos, A. Use of Self-Management Interventions for Chronic Pain Management: A Comparison between Rural and Nonrural Residents. Pain. Manag. Nurs. 2018 , 19 , 8–13. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Tauben, D.; Stacey, B. Approach to the Management of Chronic Non-Cancer Pain in Adults. UpToDate. 2024. Available online: https://www.uptodate.com/contents/approach-to-the-management-of-chronic-non-cancer-pain-in-adults (accessed on 26 February 2024).
• Parchman, M.L.; Ike, B.; Osterhage, K.P.; Baldwin, L.M.; Stephens, K.A.; Sutton, S. Barriers and facilitators to implementing changes in opioid prescribing in rural primary care clinics. J. Clin. Transl. Sci. 2020 , 4 , 425–430. [ Google Scholar ] [ CrossRef ]
• Parlier, A.B.; Galvin, S.L.; Thach, S.; Kruidenier, D.; Fagan, E.B. The Road to Rural Primary Care: A Narrative Review of Factors That Help Develop, Recruit, and Retain Rural Primary Care Physicians. Acad. Med. 2018 , 93 , 130. [ Google Scholar ] [ CrossRef ]
• Decker, S.A.; Culp, K.R.; Cacchione, P.Z. Evaluation of musculoskeletal pain management practices in rural nursing homes compared with evidence-based criteria. Pain Manag. Nurs. 2009 , 10 , 58–64. [ Google Scholar ] [ CrossRef ]
• Elhakim, M.; Dexter, F.; Pearson, A.C.S. US critical access hospitals’ listings of pain medicine physicians and other clinicians performing interventional pain procedures. J. Clin. Anesth. 2019 , 58 , 52–54. [ Google Scholar ] [ CrossRef ]
• Rodgers-Melnick, S.N.; Trager, R.J.; Love, T.E.; Dusek, J.A. Engagement in Integrative and Nonpharmacologic Pain Management Modalities among Adults with Chronic Pain: Analysis of the 2019 National Health Interview Survey. J. Pain Res. 2024 , 17 , 253–264. [ Google Scholar ] [ CrossRef ]
• Day, M.A.; Thorn, B.E. The relationship of demographic and psychosocial variables to pain-related outcomes in a rural chronic pain population. Pain 2010 , 151 , 467–474. [ Google Scholar ] [ CrossRef ]
• Gessert, C.; Waring, S.; Bailey-Davis, L.; Conway, P.; Roberts, M.; VanWormer, J. Rural definition of health: A systematic literature review. BMC Public Health 2015 , 15 , 378. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Mares, J.G.; Lund, B.C.; Adamowicz, J.L.; Burgess, D.J.; Rothmiller, S.J.; Hadlandsmyth, K. Differences in chronic pain care receipt among veterans from differing racialized groups and the impact of rural versus urban residence. J. Rural. Health 2023 , 39 , 595–603. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Qudah, B.; Maurer, M.A.; Mott, D.A.; Chui, M.A. Discordance in Addressing Opioid Crisis in Rural Communities: Patient and Provider Perspectives. Pharmacy 2022 , 10 , 91. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Vallerand, A.H.; Fouladbakhsh, J.M.; Templin, T. Self-Treatment of Pain in a Rural Area. J. Rural. Health 2004 , 20 , 166–172. [ Google Scholar ] [ CrossRef ]
• Prunuske, J.P.; Hill, C.A.S.; Hager, K.D.; Lemieux, A.M.; Swanoski, M.T.; Anderson, G.W.; Lutfiyya, M.N. Opioid prescribing patterns for non-malignant chronic pain for rural versus non-rural US adults: A population-based study using 2010 NAMCS data. BMC Health Serv. Res. 2014 , 14 , 563. [ Google Scholar ] [ CrossRef ]
• Kapoor, S.; Thorn, B.E. Healthcare use and prescription of opioids in rural residents with pain. Rural Remote Health 2014 , 14 , 2879. [ Google Scholar ] [ CrossRef ]
• Brunner, W.M.; Pullyblank, K.; Scribani, M.B.; Krupa, N.L. Loneliness among Rural Self-Management Education Program Enrollees during the COVID-19 Pandemic. Am. J. Health Promot. 2022 , 36 , 1296–1303. [ Google Scholar ] [ CrossRef ]
• Cheeseman Day, J. Rates of Uninsured Fall in Rural Counties, Remain Higher than Urban Counties. United States Census Bureau. 2019. Available online: https://www.census.gov/library/stories/2019/04/health-insurance-rural-america.html (accessed on 12 February 2024).
• Aging in Rural Communities—PubMed. Available online: https://pubmed-ncbi-nlm-nih-gov.ezproxy.bu.edu/36404874/ (accessed on 12 February 2024).
• Tul, Y.; Unruh, A.; Dick, B.D. Yoga for chronic pain management: A qualitative exploration. Scand. J. Caring Sci. 2011 , 25 , 435–443. [ Google Scholar ] [ CrossRef ]
• Bridges, L.; Sharma, M. The Efficacy of Yoga as a Form of Treatment for Depression. J. Evid-Based Complement. Altern. Med. 2017 , 22 , 1017–1028. [ Google Scholar ] [ CrossRef ]
• Manchikanti, L.; Malla, Y.; Wargo, B.; Pampati, V.; Fellows, B. Complications of fluoroscopically directed facet joint nerve blocks: A prospective evaluation of 7500 episodes with 43,000 nerve blocks. Pain. Physician. 2012 , 15 , E143–E150. [ Google Scholar ] [ CrossRef ]
• Copenhaver, D.; Pritzlaff, S.; Jung, M.; Singh, N. Interventional Therapies for Chronic Pain. UpToDate. 2024. Available online: https://www.uptodate.com/contents/interventional-therapies-for-chronic-pain?topicRef=126111&source=see_link#H62304578 (accessed on 22 April 2024).
• A Chen, J.; DeFaccio, R.J.; Gelman, H.; Thomas, E.R.; Indresano, J.; Dawson, T.C.; Glynn, L.H.; Sandbrink, F.; Zeliadt, S.B. Telehealth and Rural-Urban Differences in Receipt of Pain Care in the Veterans Health Administration. Pain. Med. Malden Mass. 2022 , 23 , 466–474. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Spleen, A.; Lengerich, E.; Camacho, F.; Vanderpool, R. Health Care Avoidance among Rural Populations: Results From a Nationally Representative Survey. J. Rural. Health 2014 , 30 , 79–88. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Mittinty, M.M.; Vanlint, S.; Stocks, N.; Mittinty, M.N.; Lorimer Moseley, G. Exploring effect of pain education on chronic pain patients’ expectation of recovery and pain intensity. Scand. J. Pain 2018 , 18 , 211–219. [ Google Scholar ] [ CrossRef ]
• Products—Data Briefs—Number 372—August 2020. Available online: https://www.cdc.gov/nchs/products/databriefs/db372.htm (accessed on 12 February 2024).
• Ernst, M.; Niederer, D.; Werner, A.M.; Czaja, S.J.; Mikton, C.; Ong, A.D.; Rosen, T.; Brähler, E.; Beutel, M.E. Loneliness before and during the COVID-19 pandemic: A systematic review with meta-analysis. Am. Psychol. 2022 , 77 , 660–677. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Kong, L.J.; Lauche, R.; Klose, P.; Bu, J.H.; Yang, X.C.; Guo, C.Q.; Dobos, G.; Cheng, Y.W. Tai Chi for Chronic Pain Conditions: A Systematic Review and Meta-analysis of Randomized Controlled Trials. Sci. Rep. 2016 , 6 , 25325. [ Google Scholar ] [ CrossRef ]
• Sani, N.A.; Yusoff, S.S.M.; Norhayati, M.N.; Zainudin, A.M. Tai Chi Exercise for Mental and Physical Well-Being in Patients with Depressive Symptoms: A Systematic Review and Meta-Analysis. Int. J. Environ. Res. Public Health 2023 , 20 , 2828. [ Google Scholar ] [ CrossRef ]
• Jain, F.A.; Walsh, R.N.; Eisendrath, S.J.; Christensen, S.; Cahn, B.R. Critical Analysis of the Efficacy of Meditation Therapies for Acute and Subacute Phase Treatment of Depressive Disorders: A Systematic Review. Psychosomatics 2015 , 56 , 140. [ Google Scholar ] [ CrossRef ]
• Wu, Y.; Yan, D.; Yang, J. Effectiveness of yoga for major depressive disorder: A systematic review and meta-analysis. Front. Psychiatry 2023 , 14 , 1138205. [ Google Scholar ] [ CrossRef ]
• la Cour, P.; Petersen, M. Effects of Mindfulness Meditation on Chronic Pain: A Randomized Controlled Trial. Pain. Med. 2015 , 16 , 641–652. [ Google Scholar ] [ CrossRef ]
• Felter, C.E.; Zalewski, K.; Jermann, R.; Palmer, P.L.; Baier, A.E.; Falvey, J.R. Rural Health: The Dirt Road Less Traveled. Phys. Ther. 2022 , 102 , pzac112. [ Google Scholar ] [ CrossRef ]
• Levy, C.E.; Silverman, E.; Jia, H.; Geiss, M.; Omura, D. Effects of physical therapy delivery via home video telerehabilitation on functional and health-related quality of life outcomes. J. Rehabil. Res. Dev. 2015 , 52 , 361–370. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Vorenkamp, K.E.; Kochat, S.; Breckner, F.; Dimon, C. Challenges in Utilizing Telehealth for Chronic Pain. Curr. Pain Headache Rep. 2022 , 26 , 617–622. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Tauben, D.J.; Langford, D.J.; Sturgeon, J.A.; Rundell, S.D.; Towle, C.; Bockman, C.; Nicholas, M. Optimizing telehealth pain care after COVID-19. Pain 2020 , 161 , 2437–2445. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Strand, N.; Tieppo Francio, V.; Turkiewicz, M.; El Helou, A. Advances in Pain Medicine: A Review of New Technologies. Curr. Pain Headache Rep. 2022 , 26 , 605–616. [ Google Scholar ] [ CrossRef ]
• Oliveira, M.F.D.; Johnson, D.S.; Demchak, T.; Tomazoni, S.S.; Leal-Junior, E.C. Low-intensity LASER and LED (photobiomodulation therapy) for pain control of the most common musculoskeletal conditions. Eur. J. Phys. Rehabil. Med. 2021 , 58 , 282–289. [ Google Scholar ] [ CrossRef ]
• Ergonenc, T.; Stockman, J. New ultrasound-guided techniques in chronic pain management: An update. Curr. Opin. Anesthesiol. 2021 , 34 , 634–640. [ Google Scholar ] [ CrossRef ]
• LaVigne, J.E.; Alles, S.R. CCK2 receptors in chronic pain. Neurobiol. Pain 2022 , 11 , 100092. [ Google Scholar ] [ CrossRef ]
• Lamer, T.J.; Moeschler, S.M.; Gazelka, H.M.; Hooten, W.M.; Benden, M.A.; Murad, M.H. Spinal Stimulation for the Treatment of Intractable Spine and Limb Pain: A Systematic Review of RCTs and Meta-Analysis. Mayo Clin. Proc. 2019 , 94 , 1475–1487. [ Google Scholar ] [ CrossRef ]
• Xu, J.; Herndon, C.; Anderson, S.; Getson, P.; Foorsov, V.; Harbut, R.; Moskovitz, P.; Harden, R.N. Intravenous Ketamine Infusion for Complex Regional Pain Syndrome: Survey, Consensus, and a Reference Protocol. Pain Med. 2019 , 20 , 323–334. [ Google Scholar ] [ CrossRef ]
• Cohen, S.P.; Bhatia, A.; Buvanendran, A.; Schwenk, E.S.; Wasan, A.D.; Hurley, R.W.; Viscusi, E.R.; Narouze, S.; Davis, F.N.; Ritchie, E.C.; et al. Consensus Guidelines on the Use of Intravenous Ketamine Infusions for Chronic Pain from the American Society of Regional Anesthesia and Pain Medicine, the American Academy of Pain Medicine, and the American Society of Anesthesiologists. Reg. Anesth. Pain Med. 2018 , 43 , 521–546. [ Google Scholar ] [ CrossRef ]
• Iacob, E.; Hagn, E.; Sindt, J.; Brogan, S.; Tadler, S.C.; Kennington, K.S.; Hare, B.D.; E Bokat, C.; Donaldson, G.W.; Okifuji, A.; et al. Tertiary Care Clinical Experience with Intravenous Lidocaine Infusions for the Treatment of Chronic Pain. Pain Med. 2018 , 19 , 1245–1253. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ohta, R.; Sano, C. Integrating Clinical and Socio-Environmental Approaches in Managing Rheumatoid Arthritis with Social Determinants of Health: A Case Study of an Elderly Patient in Rural Japan. Cureus 2023 , 15 , e50915. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Cheung, J.C.-W.; Cheung, D.S.K.; Ni, M.; Chen, K.-W.; Mao, Y.-J.; Feng, L.; Lam, W.-K.; Wong, D.W.-C.; Leung, A.K.-L. X-reality for phantom limb management for amputees: A systematic review and meta-analysis. Eng. Regen. 2023 , 4 , 134–151. [ Google Scholar ] [ CrossRef ]
• Darnall, B.D.; Edwards, K.A.; Courtney, R.E.; Ziadni, M.S.; Simons, L.E.; Harrison, L.E. Innovative treatment formats, technologies, and clinician trainings that improve access to behavioral pain treatment for youth and adults. Front. Pain Res. 2023 , 4 , 1223172. [ Google Scholar ] [ CrossRef ]
• Health Resources & Services Administration. Licensing across State Lines. Telehealth.HHS.gov. 21 February 2024. Available online: https://telehealth.hhs.gov/licensure/licensing-across-state-lines (accessed on 14 March 2024).
• Katzman, J.G.; Comerci, G., Jr.; Boyle, J.F.; Duhigg, D.; Shelley, B.; Olivas, C.; Daitz, B.; Carroll, C.; Som, D.; Monette, R.; et al. Innovative Telementoring for Pain Management: Project ECHO Pain. J. Contin. Educ. Health Prof. 2014 , 34 , 68–75. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gordon, S.E.; Monti, S.M.; Catic, A.G.; Dufour, A.B.; Mattison, M.; Morgan, M.; Lipsitz, L.A. Project ECHO-AGE and Nursing Home Quality of Care. J. Am. Med. Dir. Assoc. 2015 , 16 , B27–B28. [ Google Scholar ] [ CrossRef ]
• Shojaei, H.; Lakha, S.F.; Lyon, A.; Halabecki, M.; Donaghy, M.; Mailis, A. Evolution of a chronic pain management program in a Northwestern Ontario community: From structural elements to practical application. BMC Health Serv. Res. 2022 , 22 , 1355. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Baker, M.B.; Liu, E.C.; Bully, M.A.; Hsieh, A.; Nozari, A.; Tuler, M.; Binda, D.D. Overcoming Barriers: A Comprehensive Review of Chronic Pain Management and Accessibility Challenges in Rural America. Healthcare 2024 , 12 , 1765. https://doi.org/10.3390/healthcare12171765

Baker MB, Liu EC, Bully MA, Hsieh A, Nozari A, Tuler M, Binda DD. Overcoming Barriers: A Comprehensive Review of Chronic Pain Management and Accessibility Challenges in Rural America. Healthcare . 2024; 12(17):1765. https://doi.org/10.3390/healthcare12171765

Baker, Maxwell B., Eileen C. Liu, Micaiah A. Bully, Adam Hsieh, Ala Nozari, Marissa Tuler, and Dhanesh D. Binda. 2024. "Overcoming Barriers: A Comprehensive Review of Chronic Pain Management and Accessibility Challenges in Rural America" Healthcare 12, no. 17: 1765. https://doi.org/10.3390/healthcare12171765

Article Metrics

Further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 03 September 2024

Evaluating coupling coordination between urban smart performance and low-carbon level in China’s pilot cities with mixed methods

• Xiongwei Zhu 1 ,
• Dezhi Li 1 , 2 ,
• Shenghua Zhou 1 ,
• Shiyao Zhu 3 &
• Lugang Yu 1  

Scientific Reports volume  14 , Article number:  20461 ( 2024 ) Cite this article

Metrics details

• Climate-change adaptation
• Climate-change impacts
• Environmental impact
• Sustainability

The construction models of smart cities and low-carbon cities are crucial for advancing global urbanization, enhancing urban governance, and addressing major urban challenges. Despite significant advancements in smart and low-carbon city research, a consensus on their coupling coordination remains elusive. This study employs mixed-method research, combining qualitative and quantitative analyses, to investigate the coupling coordination between urban smart performance (SCP) and low-carbon level (LCL) across 52 typical smart and low-carbon pilot cities in China. Independent evaluation models for SCP and LCL qualitatively assess the current state of smart and low-carbon city construction. Additionally, an Entropy–TOPSIS–Pearson correlation–Coupling coordination degree (ETPC) analysis model quantitatively examines their relationship. The results reveal that smart city initiatives in China significantly outperform low-carbon city development, with notable disparities in SCP and LCL between eastern, non-resource-based, and central cities versus western, resource-dependent, and peripheral cities. A strong positive correlation exists between urban SCP and overall LCL, with significant correlations in management, society, and economy, and moderate to weak correlations in environmental quality and culture. As SCP levels improve, the coupling coordination degree between the urban SCP and LCL systems also increases, driven primarily by economic, management, and societal factors. Conversely, the subsystems of low-carbon culture and environmental quality show poorer integration. Based on these findings, this study proposes an evaluation system for smart and low-carbon coupling coordination development, outlining pathways for future development from the perspective of urban complex systems.

Similar content being viewed by others

An empirical analysis of the coupling and coordinated development of new urbanization and ecological welfare performance in China’s Chengdu–Chongqing economic circle

Decomposing the comprehensive efficiency of major cities into divisions on governance, ICT and sustainability: network slack-based measure model

Impact of urban spatial structure elements on carbon emissions efficiency in growing megacities: the case of Chengdu

Introduction.

Cities, as centers of population and economy, play crucial roles in cultural exchange, social integration, transportation, communication, and disaster response in modern societal development 1 , 2 . According to the United Nations Human Settlements program’s “2022 World Cities Report”, as of 2021, the global urbanization rate has reached 56%, and it is projected that by 2050, an additional 2.2 billion people will live in cities, increasing the urbanization rate to 68% 3 . North America and European countries are approaching urbanization saturation, with little fluctuation expected, while urbanization in Asia and Africa will accelerate notably 4 . Particularly in China, the world’s second-largest economy, as of 2022, the urbanization rate is only 64.7%, ranking 96th globally, indicating significant potential for growth compared to developed countries like the USA and the UK 5 . The Chinese government places high importance on urbanization development. It was clearly stated in the “2020 State Council Government Work Report” that new urbanization is a key measure for achieving China’s modernization. Moreover, in the “14th Five-Year Plan (2021–2025) and the Long-Range Objectives Through the Year 2035”, detailed strategies are outlined for optimizing the urban layout and promoting urban–rural integration, among other policies to advance urbanization 6 . However, urbanization, as a process of continuous concentration of population and industrial elements in cities, while bringing opportunities for economic growth and social development, also presents a series of challenges such as environmental pressure, resource constraints, and increased demand for services 7 , 8 .

In 2008, the American company IBM introduced the concept of a “Smart Planet”, which garnered widespread attention globally 9 . The concept of a smart city, as a specific application within this framework, aims to enhance urban management and service efficiency through the integration and innovative application of Information and Communication Technology (ICT), thereby improving the quality of life for residents, optimizing resource use, reducing environmental impact, and promoting economic development and social progress 10 , 11 . Currently, the smart city construction model is seen as one of the effective means to advance global urbanization, improve urban governance, and solve major urban issues 12 . In 2009, IBM released the “Smart Planet: Winning in China” plan, outlining China’s five major thematic tasks in constructing a “Smart Planet” (sustainable economic development, corporate competitiveness, energy efficiency, environmental protection, and social harmony) 13 . The construction of smart cities, as a key measure to achieve these thematic tasks, has received significant attention from the Chinese government. In 2014, the Chinese government elevated smart city construction to a “national strategy”, considering it a cornerstone of China’s future economic and urban development strategies. By 2016, over 500 Chinese cities had initiated or announced smart city pilot construction plans, accounting for nearly half of all such projects planned or underway globally 14 . In recent years, with the continuous release of policy benefits related to smart city construction in China and substantial capital investment, China has become a leader in driving global smart city initiatives 15 . However, an undeniable fact is that while smart city construction models promote economic development and improve the quality of life for residents, the new infrastructure supporting the operation of smart cities, such as big data centers, 5G shared base stations, and Beidou ground-based augmentation stations, result in substantial energy consumption and significant carbon emissions 16 . Research shows that in 2018, the total electricity consumption of data centers in China supporting IT infrastructure reached 160.9 billion kilowatt-hours, exceeding the total electricity consumption of Shanghai for that year and accounting for about 2% of China’s total electricity consumption, with carbon emissions nearing 100 million tons 17 . The Environmental Defense Fund (EDF) predicts that by 2035, the total electricity consumption of China’s data centers and 5G base stations will reach 695.1–782 billion kilowatt-hours, accounting for 5–7% of China’s total electricity consumption, with total carbon emissions reaching 230–310 million tons 18 .

In 2022, global energy-related CO 2 emissions increased by 0.9%, reaching a record high of over 36.8 Gt. Concurrently, atmospheric CO 2 concentrations continued to rise, averaging 417.06 parts per million, marking the eleventh consecutive year with an increase exceeding 2 ppm 19 . According to the World Meteorological Organization (WMO), the global surface temperature in September 2023 was 1.44 °C higher than the twentieth century average, setting a new historical record 20 . The continuous rise in global temperatures has led to frequent occurrences of disastrous events such as extreme heat, torrential rains, floods, forest fires, and hurricanes in recent years, causing significant loss of life and property damage 21 . World Health Organization (WHO) data indicates that in 2022, there were at least 29 weather disaster events globally causing billions of dollars in losses, with approximately 61,672 deaths in Europe due to heatwave-related causes 22 . As global climate issues become increasingly severe, the call for global carbon emission reduction is growing louder. Cities, as highly concentrated areas of population and economic activities, according to the Global Report by the United Nations Human Settlements Programme (UN-Habitat), consume 60–80% of the global energy and contribute to over 75% of global CO 2 emissions 23 . As the largest global emitter of carbon, China’s CO 2 emissions in 2022 accounted for 27% of the global total 24 . Given China’s influence in the global economy, technological innovation, and international cooperation, international organizations and global climate policies generally believe that China’s efforts in carbon reduction are crucial to achieving the global 1.5 °C climate goal 25 . In recent years, the Chinese government has actively promoted the construction of low-carbon pilot cities. To date, three batches of low-carbon pilot cities have been implemented in China, bringing the total number of such cities to 81 26 .

However, the report “China’s Digital Infrastructure Decarburization Path: Data Centers and 5G Carbon Reduction Potential and Challenges (2020–2035)” indicates that compared to peak carbon emissions expected around 2025 in key sectors like steel, building materials, and non-ferrous metals in China, the “lock-in effect” of carbon emissions from digital infrastructure poses a significant challenge to achieving China’s peak carbon and carbon neutrality goals 27 , 28 , 29 . Given the urgency of global climate change, it raises the question of the correlation between smart cities and low-carbon cities: is it positive, negative, or non-existent? Should the pace of smart city development be slowed to achieve sustainable urban development goals, considering the significant carbon dioxide emissions resulting from current technological choices, social habits, and policy frameworks? To address these practical issues, it is first essential to conduct an objective and accurate assessment of urban SCP and LCL. However, due to the complexity and diversity of urban carbon emissions sources, current measurement and estimation techniques fail to capture all emission types. This limitation hampers the ability to obtain comprehensive, accurate, and timely city-level carbon emission data 30 , 31 . To address this challenge, this paper decomposes smart cities and low-carbon cities into their interdependent and interactive subsystems (i.e., economic, political, cultural, social, and ecological) viewed through the lens of urban complex systems. It then develops evaluation models for both city types and conducts empirical analyses in 52 representative Chinese pilot cities. Based on these analyses, the paper elucidates the coupling coordination degree between SCP and LCL and proposes a specific pathway for their coordinated development.

This paper is therefore structured as follows: “ Literature review ” section offers an overview of the relevant literature, laying the foundation for the introduction of SCP and LCL. Subsequently, SCP and LCL are identified clearly, and measurement based on a mixed method for the coupling coordination degree is established in “ Methodology ” section, followed by a case demonstration for the introduced method in “ Results ” section and the demonstration results analysis in “ Discussions and implications ” section. Finally, “ Conclusions ” section summarizes the study’s main findings and contributions, discusses its limitations, and suggests directions for future research.

Literature review

Evaluation of smart city: contents, methods, and subjects.

The evaluation of smart cities is a central research area within the smart city development field. Developing standardized evaluation criteria serves the dual purpose of defining smart city development boundaries and scientifically measuring its effectiveness. This, in turn, facilitates the achievement of development goals centered on evaluation-driven construction, improvement, and management 32 . We conducted data collection on “smart city*” AND “evaluation”, resulting in the selection of 82 articles. This involved an extensive search of the Wos Core Collection database for articles published in the period from January 2019 to January 2024.

To facilitate a clearer understanding for readers of current research on smart city evaluation, we have categorized it by evaluation contents , evaluation methods , and evaluation subjects .

Cluster1-evaluation contents (what to evaluate), including smart city evaluation dimensions and indicators. By analyzing the article content, it’s clear that most smart city evaluation approaches align with six core dimensions: economy, quality of life, governance, people, mobility, and environment 13 , 15 . Centered around these six dimensions, international organizations (ISO, ETSI, UN, and ITU) and scholars have established various sets of smart city evaluation indicators, considering the interdependencies among urban economic, environmental, and social factors, all in alignment with the goals of sustainable urban development 32 , 33 , 34 . Notably, Sharifi 35 compiled a comprehensive list of indicators incorporating a wide range of assessment schemes. This list not only covers the scope of the evaluation indicators (project/community/city) and their data types (primary/secondary) but also considers the stages of smart city development (planning/operation) and stakeholder involvement 36 . Subsequent research predominantly utilizes the same criteria as Sharifi 35 to identify indicator sets, taking into account the specific needs of each city and defining the spatial and temporal scales of the indicator sets 37 .

Cluster 2-evaluation methods (How to evaluate) , including smart city evaluation methods and tools. Research in this field focuses on three main areas: identifying evaluation indicators for smart cities, computing composite index, and developing evaluation models 38 , 39 . Methods for indicator identification mainly include literature review, case studies, brainstorming, the Delphi method, and data-driven techniques 40 , 41 . The Analytic Hierarchy Process (AHP) is commonly used for calculating composite indices, yet it faces issues like subjective biases and data size limitations 42 . Alternative methods, such as the Analytical Network Process (ANP) and the Decision-Making Trial and Evaluation Laboratory (DEMATEL), are used to address these drawbacks by simulating inter-indicator interactions. Additionally, techniques like Principal Component Analysis (PCA) and Data Envelopment Analysis (DEA) are applied for indicator weighting. Finally, smart city evaluation models are constructed to aggregate various dimensions and indicators into a unified score, facilitating project comparison and ranking, and highlighting areas needing improvement 43 , 44 .

Cluster 3-evaluation subjects (Who performs the evaluation) , including smart city stakeholders and participants. Smart city evaluations involve various stakeholders and participants. These complex processes see each entity, including government agencies, international organizations, academic institutions, industry sectors, and NGOs, contributing to the smart cities’ planning, development, and management 45 , 46 . Key organizations in this realm are the International Organization for Standardization (ISO), International Telecommunication Union (ITU), United Nations Human Settlements Programme (UN-Habitat), Smart Cities Council, European Institute of Innovation and Technology (EIT Urban Mobility), and World Council on City Data (WCCD). Additionally, numerous countries have established their own smart city evaluation standards to direct and review smart city progress 11 . Notable examples are the “One New York: The Plan for a Strong and Just City” in the USA, the “BSI PAS 180” in the UK, Singapore's “Smart Nation Initiative”, and China’s “National New-type Smart City Evaluation Indicator System”.

Evaluation of low-carbon city: contents, methods, and subjects

As more countries integrate low-carbon city development into their national strategies and plans, conducting scientific evaluations of cities’ current low-carbon development levels to encourage them to adopt corresponding measures for improvement has become a key strategy in advancing cities towards a low-carbon future 47 . In the Wos Core Collection database, we conducted a search for studies spanning January 2018 to January 2023 with “low-carbon city*” AND “evaluation” as keywords, subsequently identifying 98 pertinent articles through two rounds of screening.

This section, maintaining the research framework of “ Evaluation of smart city: contents, methods, and subjects ” section ( evaluation contents, methods, and subjects ), organizes low-carbon city research to enable comparison with smart city evaluations.

Cluster 1-evaluation contents (what to evaluate), including low-carbon city evaluation systems, dimensions, and indicators. Current research focusing on low-carbon cities primarily spans six key domains: urban low-carbon scale, energy, behavior, policy, mobility, and carbon sinks. The evaluation dimensions for low-carbon cities are mainly divided into two types: single-criterion systems concentrating on specific low-carbon aspects (such as low-carbon economy, low-carbon energy, etc.), and comprehensive multi-criteria systems assessing the overall urban low-carbon development 48 , 49 . Compared to single-criterion evaluation systems, comprehensive and multi-criteria evaluation systems are increasingly gaining attention from scholars. These scholars share the view that low-carbon city construction is a diverse, dynamic, interconnected process that requires comprehensive consideration of various urban aspects, including economy, society, and environment, and involves coordinating the actions of different stakeholders to achieve sustainable urban development 50 , 51 . Additionally, international institutions and many national governments have also published low-carbon city evaluation frameworks from the perspective of comprehensive and multi-criteria evaluation systems. The most notable examples include the United Nations Commission on Sustainable Development, which set 30 indicators from four dimensions: social, environmental, economic, and institutional, to evaluate the level of urban low-carbon development. The Chinese Academy of Social Sciences proposed the “China Low Carbon City Indicator System”, covering 8 dimensions such as economy, energy, facilities, and 25 specific indicators including energy intensity, per capita carbon emissions, and forest coverage rate.

Cluster 2-evaluation methods (How to evaluate) , including low-carbon city evaluation methods and tools. Firstly, identifying evaluation indicators as the initial step in constructing a low-carbon city evaluation model, current research methods not only include traditional methods like literature review and expert interviews but also increasingly involve scholars using dynamic perspectives based on urban complex systems, applying models like DPSR (Driving forces-Pressures-State-Response), STIRPA (Stochastic Impacts by Regression on Population, Affluence, and Technology), the Environmental Kuznets Curve (EKC), and STEEP (Social, Technological, Economic, Ecological, and Political) for indicator identification 52 , 53 . Secondly, weighting evaluation indicators, an essential part of model construction, typically involves methods like subjective weighting (expert scoring, Delphi method, AHP) 54 , objective weighting (PCA, Entropy weight method, variance analysis), and combined weighting (DEA) 55 . Each method has its characteristics and suitable scenarios and should be selected according to specific circumstances. Additionally, quantitative assessment of regional carbon emissions using methods like carbon footprint analysis, baseline emission comparison, and Life Cycle Assessment (LCA) is also becoming a research focus 56 .

Cluster 3-evaluation subjects (Who performs the evaluation) , including low-carbon city stakeholders and participants. The evaluation of low-carbon cities also involves multiple stakeholders (government, enterprises, residents, etc.) 57 . Among them, international organizations like the International Organization for Standardization (ISO), the International Energy Agency (IEA), and the World Meteorological Organization (WMO) have played significant roles in establishing low-carbon city evaluation standards and promoting global low-carbon city development. Additionally, due to economic, policy, and perception factors, current low-carbon city construction relies primarily on government financial input, with social capital and public participation in low-carbon city construction noticeably lacking 58 . Therefore, how to enhance the awareness of enterprises and residents as main actors in low-carbon city construction has become a current research focus.

Coupling coordination analysis between SCP and LCL

Smart cities and low-carbon cities, as important urban development models for the future, have seen an increasing focus on their interrelation by scholars in recent years, becoming an emerging research hotspot in the field. In the Wos Core Collection database, we searched for studies from January 2018 to January 2024 using the keywords “smart city*” “low-carbon city*” “correlation analysis” “coupling coordination analysis” and “urban sustainability”. After two rounds of screening, 24 related studies were selected for analysis.

From the perspective of research results, the current research conclusions about the correlation between low-carbon cities and smart cities primarily include two main points: (i) SCP and LCL cannot achieve coupling coordination development. Some scholars argue that SCP and LCL differ in their focus: SCP emphasizes urban technological and economic development, while LCL focuses more on urban ecological construction 17 . Particularly, De Jong identified 12 urban development concepts, including smart city, low-carbon city, eco-city, and green city. He believes that a clear distinction must be made in the conceptual definition of these types of cities to more accurately guide future urban planning 59 . Furthermore, some scholars argue that the relationship between SMC and LCC is negatively correlated. Deakin believes that the direct environmental benefits of IoT technology are insufficient to achieve urban sustainability goals 60 . Barr et al. argue that the logic of smart cities often leads city administrations to prioritize superficial changes and promote individual behavioral shifts, detracting from the crucial task of reconfiguring urban infrastructure for low-carbon lifestyles 61 , 62 . (ii) SCP and LCL can achieve coupling coordination development. Some scholars believe there is a positive correlation between SCP and LCL, with SCP potentially promoting the development of LCL. Specifically, the intelligent systems built by SCP can effectively match urban energy supply and demand, reducing urban carbon emissions, such as through smart grids and intelligent transportation networks 18 . It is worth noting that most of the studies on the coupling coordination relationship between urban SCP and LCL are based on perspectives of individual urban subsystems such as technology, economy, management, industrial structure, and society. They lack a comprehensive consideration of the city as a complex system 59 , 61 , 63 .

From the perspective of research methodologies, coupling coordination analysis is a fundamental statistical approach for examining relationships between two or more variables. This analysis typically employs techniques such as Pearson’s correlation coefficient, Spearman’s rank correlation coefficient, Kendall’s tau, partial correlation, point-biserial correlation, and multiple correlations. Each technique offers unique insights into the nature and strength of the interdependencies among variables 61 . The selection of an appropriate method depends on the data type (continuous, ordinal, or categorical), its distribution (e.g., normal distribution), and the specific objectives of the research.

In summary, although existing research has made significant contributions to the independent evaluation and advancement of smart cities and low-carbon cities, including their relevant construction content, main actors, as well as some specific measures such as empowering cities with data intelligence for low-carbon economic development and transitioning industrial structure to low-carbon, there are still some important knowledge gaps. On the one hand, current research primarily analyzes the coupling coordination relationship between urban SCP and LCL from the micro-perspective of individual urban subsystems such as economic and energy systems. This approach lacks a macroscopic perspective from the complex urban system, which is detrimental to the comprehensive development of cities 60 , 64 , 65 . On the other hand, current studies often only conduct basic qualitative comparisons of the relationship between the development levels of urban SCP and LCL from a quantitative or qualitative perspective. They lack a comprehensive analytical approach that integrates both qualitative and quantitative analyses for further exploration of the coupling coordination relationship between urban SCP and LCL. This shortfall hinders the sustainable development of cities.

To fill these knowledge gaps, this study employs a mixed-methods approach, combining qualitative and quantitative analyses, to examine the model of coupling coordination between urban SCP and LCL. It also develops recommendations to enhance this coupling coordination, aiming to support sustainable development goals. Furthermore, this research selects 52 typical low-carbon and smart pilot cities in China as case studies, ensuring both scientific validity and practical applicability of the findings. Additionally, to enhance the logical coherence and readability of this study, we posit that a coupling coordination relationship exists between urban SCP and LCL and thus propose Hypothesis 1 .

Hypothesis 1

There is a substantial degree of coupling coordination between the overall urban system’s SCP and LCL, yet there are disparities in this coordination degree among the subsystems of economy, society, politics, culture, and ecology.

Methodology

Research framework.

The construction of low-carbon and smart cities, as key pathways to urban sustainability, necessitates examining their interplay and fostering their collaborative development for achieving sustainability goals 66 . This research employs a sequential framework, including Conceptual, Data, Analysis, and Decision-making Layers, to methodically explore the coupling coordination relationship between SCP and LCL, with the framework illustrated in Fig.  1 .

Research framework.

Firstly , in the Conceptual Layer, this study aligns with the United Nations’ objectives for sustainable cities, encompassing economic growth, social equity, better life conditions, and improved urban environments. Integrating these with China’s “Five-Sphere Integrated Plan (economy, politics, culture, society, and ecological environment construction)” for urban development, the research dissects the components of smart city systems (such as information infrastructure, information security, public welfare services) and low-carbon city systems (including low-carbon construction, transportation, and industry), with the aim to collect indicators. Secondly , in the Data Layer, this research develops smart city and low-carbon city evaluation systems, grounded in national standards and official statistics, to qualitatively examine the correlation between SCP and LCL from a macro perspective. Thirdly, in the Analysis Layer, this study selects 52 cities, both smart and low-carbon pilot cities in China, as samples for quantitative analysis. The process involves standardizing indicators, scoring and ranking the cities based on their smart performance and low-carbon levels, followed by employing Pearson’s correlation coefficient and coupling coordination degree model to scientifically analyze the correlation between SCP and LCL. Finally, in the Decision-making Layer, the study examines the coupling coordination relationship between urban smart performance, the overall low-carbon level, and the low-carbon level across five dimensions, which is key for us to test Hypothesis 1 . It also formulates development paths for the coupling coordination of smart and low-carbon cities.

SCP index system construction

Since the concept of smart cities was introduced in 2008, many national governments have established smart city evaluation standards. Due to varying national conditions, SCP evaluation indicators differ across countries. As the sample cities in this study are Chinese smart pilot cities, the selection of SCP evaluation indicators primarily references relevant Chinese national standards. As a global pioneer in smart city development, China released the “Evaluation indicators for new-type smart cities (GB/T 33356-2016)” in 2016 and revised it in 2022. This national standard, with its evaluative indicators, clearly defines the key construction content and development direction of new smart cities, aiming to specifically enhance the effectiveness and level of smart city construction, gaining significant recognition within the industry.

This study, grounded in the concept of a city’s “Five-in-One” sustainable development, is guided by three principles of “Inclusive well-being & Ecological harmony”, “Digital space & Physical space”, and “New IT technologies & Comprehensive services”. It also adheres to the “people-oriented concept” and adopts an “urban complex dynamic perspective” in the process of smart city construction. Additionally, it follows the principle of “similar attributes of evaluation objects”. Based on these foundations, the study establishes three criteria for selecting evaluation indicators, including scientific, coordination, and representation. Drawing on the Chinese government’s smart city evaluation standards and utilizing a literature review methodology, this research constructs an SCP evaluation indicator system for cities, as detailed in Supplementary Appendix Table A1 . The SCP index system includes six primary indicators, including smart public service (SPE), precise governance (PG), information infrastructure (II), digital economy (DE), innovative development environment (IDE), and citizen satisfaction (SCS). It also features 24 secondary indicators, such as traffic information services, grassroots smart governance, and spatio-temporal information platforms. Importantly, to explore the correlation between smart cities and low-carbon cities more effectively, the study deliberately omits “Internet + Green Ecology” related indicators from the smart city evaluation system. To ensure the accuracy and representativeness of these indicators, they were validated through expert consultation, public participation, and comprehensive statistical methods.

LCL index system construction

Current international organizations and academic perspectives on low-carbon city evaluation systems are predominantly based on the urban complex systems approach, considering the interplay and interaction of aspects such as low-carbon society, economy, and technology. Consistent with the principles for selecting SCP evaluation indicators, the choice of LCL evaluation indicators in this study primarily adheres to relevant Chinese national standards and related literature.

As a proactive practitioner in global low-carbon city development, in 2021, the Chinese government released the “Sustainable Cities and Communities—Guides for low-carbon development evaluation (GB/T 41152-2021)”. This national standard evaluates the level of urban low-carbon development, clarifying the key directions for such development, and serves as a current guide for low-carbon city construction in China. Thus, this study, grounded in the “Five-in-One” sustainable urban development framework and guided by the principles of “carbon reduction & pollution reduction”, “green economic growth”, and “enhanced carbon sequestration capacity”, combines the previously established principles of scientific, coordination, and representative for selecting evaluation indicators. It establishes an LCL index system based on the Chinese government’s evaluation standards and relevant literature. Specifically, the LCL evaluation index system constructed in this study includes five primary indicators, including low-carbon economic (LCE), low-carbon society (LCS), low-carbon environmental quality (LCEQ), low-carbon management (LCM), and low-carbon culture (LCC), as well as 22 secondary indicators such as energy consumption per unit of GDP and carbon emission intensity, as shown in Supplementary Appendix Table A2 . Similarly, to ensure the accuracy and representativeness of the indicators, the specific indicators were validated through expert consultation, public participation, and comprehensive statistical methods.

Analysis model construction

In this study, an Entropy-TOPSIS-Pearson correlation-Coupling coordination degree (ETPC) analysis model is constructed to quantitatively analyze the coupling coordination relationship between Urban SCP and LCL. The entropy method is first applied for objective weighting of evaluation indices, ensuring data objectivity and reducing subjective bias, thus enhancing the model’s accuracy and fairness. Next, the TOPSIS method is used to rank sample cities based on their smart performance and low-carbon levels, providing a straightforward and intuitive ranking mechanism. The Pearson correlation method then examines the correlation between SCP and LCL, offering data-driven insights into the dynamic relationships between these variables. Finally, the coupling coordination model calculates the degree of coordination between SCP and LCL, providing a theoretical basis for subsequent enhancement pathways and policy recommendations. The ETPC model constructed in this study has several advantages and complementarities, allowing for a comprehensive analysis and evaluation of the research question from various perspectives. Additionally, the ETPC model can be broadly applied to other multidimensional evaluation and decision analysis issues, such as the coupling coordination between various public health interventions and community health levels, and the comprehensive effects of different economic policies on regional economic development and environmental impact. Specific analysis steps are outlined as follows.

Step 1: Conduct the data normalization process.

where x ij and y ij represent respectively the original and standardized value for the indicator j in referring to the sample case i ( i  = 1,2,3,…, m; j  = 1,2,3,…, n ), max (x j ) and min (x j ) denote respectively the largest and smallest value among all m samples for the indicator j , P ij represents the value proportion of indicator j in the sample case i to the summation value of the indicator from all cases.

Step 2: Calculate the weight and measure the comprehensive level based on entropy method.

The entropy weight method, an objective approach deriving weights from sample characteristics, mitigates expert bias, enhancing the objectivity and credibility of indicator weighting 67 . This study employs this method, determining weights through the calculation of each indicator’s information entropy, and measure the comprehensive level of the subsystem.

where m is the total number of sample cases, \({e}_{j}\) demonstrates the entropy value of the j indicator and \({\omega }_{j}\) denotes the weight of indicator j , and V represent the comprehensive level.

Step 3: Conduct a ranking of evaluation objects based on TOPSIS method.

A key limitation of the entropy method is its tendency to neglect the significance of indicators. The TOPSIS method, addressing this issue, is an ideal-solution-based ranking technique that aids in multi-objective decision-making among finite options 68 . In this approach, the study first determines positive and negative ideal solutions, measures each objective’s distance to these ideals, and subsequently ranks the subjects by the proximity of each objective to the ideal solution.

where \({ V}^{+}\) and \({V}^{-}\) respectively represent the best ideal solution and the worst ideal solution, \({D}_{i}^{+}\) and \({D}_{i}^{-}\) represent the distances from the objective to the positive and negative ideal solutions, respectively. \({C}_{i}\) indicates the closeness of the evaluation objective to the optimal solution, with \({C}_{i}\in \left[\text{0,1}\right]\) . A larger \({C}_{i}\) value suggests stronger smart and low-carbon development capabilities of the sample city.

Step 4: Analyze the correlation based on Pearson correlation method.

The Pearson correlation method is commonly used to measure the correlation coefficient between two continuous random variables, thereby assessing the degree of correlation between them 69 . In this study, based on the results from Steps 1–3, two sets of data are obtained representing the smart development level and low-carbon development level of sample cities, \(A:\left\{{A}_{1},{A}_{2},\dots ,{A}_{n}\right\}\) and \(B:\left\{{B}_{1},{B}_{2},\dots ,{B}_{n}\right\}\) . The overall means and covariance of both data sets are calculated, resulting in the Pearson correlation coefficient between the two variables.

where \({A}_{i}\) and \({B}_{i}\) respectively represent the SCP and LCL of sample cities. \(E\left(A\right)\) and \(E\left(B\right)\) are the overall means of the two data sets, \({\sigma }_{A}\text{ and }{\sigma }_{B}\) are their respective standard deviations, \(cov(A,B)\) is the covariance, and \({\rho }_{AB}\) is the Pearson correlation coefficient. When the correlation coefficient approaches 0, the relationship weakens, as it nears − 1 or + 1, the correlation strengthens.

Step 5: Analyze the coupling coordination degree based on the coupling coordination model.

The coupling coordination degree characterizes the level of interaction between different systems and serves as a scientific model for measuring the coordinated development level of multiple subsystems or elements 70 . This study has developed a model to measure the coupling coordination degree between two systems.

where C defines the coupling degree, \({f}_{1}\) and \({f}_{2}\) are the evaluation values of SCP and LCL respectively. CPD represents the coupling coordination degree. \(\alpha\) , \(\beta\) are the coefficient to be determined, indicating the importance of the systems. This study assumes that each system is equally important. Thus \(\alpha =\beta =1/2.\)

In this study, building upon the framework established by a preceding study, a classification system for the coupling coordination degree was developed. This system delineates the various types of coupling-coordinated development among SCP, LCL, LCS, LCM, LCEQ, and LCC. Current research on the division of coupling coordination degree intervals often uses an average distribution within the [0, 1] range 70 . However, due to the large sample size and the wide distribution range of coupling coordination degrees in this study, we have categorized these types into ten distinct levels based on their rank, as detailed in Table 1 .

Selection of sample cities and data collection

The Chinese government has prioritized the development of smart and low-carbon cities. Since 2010, it has launched 290 smart city pilots and 81 low-carbon city pilots across various regions, reflecting different levels of development, resource allocations, and operational foundations. To maintain the scientific integrity of our study, we established stringent criteria for selecting sample cities: (i) each city must be concurrently identified as both a smart and a low-carbon city pilot, and (ii) their government agencies must have issued data on key performance indicators for these initiatives. Following these criteria, our research has ultimately selected 52 cities as samples, as detailed in Fig.  2 . It is noteworthy that these 52 typical case cities are almost all provincial capitals in China, mostly located within the Yangtze River Delta, Pearl River Delta, Jingjinji (Beijing–Tianjin–Hebei), and Western Triangle economic regions. Additionally, according to the “Globalization and World Cities Research Network (GaWC) World Cities Roster 2022 (GaWC2022)”, these cities are ranked within the top 200 globally. Therefore, given the scope of this research, these case cities offer significant representativeness and can serve as valuable models for promoting development in other urban areas. The data for this paper were sourced from the “China Low-Carbon Yearbook (2010–2023)”, the “China Environmental Statistics Yearbook (2010–2023)”, and low-carbon city data published by the governments of the sample cities. Additionally, this study addressed any missing data by averaging the data from adjacent years and applying exponential smoothing.

52 sample cities and their geographic locations.

Weighting values between evaluation indicators

The entropy weighting values between the 20 indicators of SCP and the 19 indicators of LCL are calculated by applying the data described in “ Weighting values between evaluation indicators ” section to formula ( 1 )–( 5 ), and the results are shown in Supplementary Appendix Tables A3 and A4 . Specifically, within the SCP evaluation framework, SPE and II are assigned the highest weights, while LCS and LCM are allocated the highest weights within the LCL evaluation framework. Conversely, SCS and LCC have attributed the lowest weights in their respective contexts.

Evaluation of SCP and LCL in sample cities

Utilizing the data from “ Selection of sample cities and data collection ” section and the weighting values derived in “ Weighting values between evaluation indicators ” section, we can determine the SCP and LCL of sample cities using the TOPSIS method, as outlined in formulas ( 6 )–( 9 ). The results are illustrated in Supplementary Appendix Table A5 and Fig.  3 . In this study, the value of the closeness coefficient (C i ) is used to indicate the relative closeness of a particular sample city to the negative ideal point 71 . The negative ideal point represents the worst solution of the ideal, where the individual attribute values reach their worst in each alternative. Therefore, a larger value of closeness indicates better smart city performance or a lower carbon level of a sample city 72 . C LCL and C SCP respectively represent the low-carbon level closeness coefficient and the smart city performance closeness coefficient. In referring to Supplementary Appendix Table A5 , the best three cities of SCP are Shenzhen, Shanghai, and Hangzhou, whilst the worst three cities are Yan’an, Jincheng, and Xining. Furthermore, Chengdu, Qingdao, and Beijing are the best there low-carbon level performers. Whilst Jincheng, Urumqi, and Huhehaote are the three worst.

TOPSIS-based analysis of SCP with LCL in 52 sample cities.

In referencing Fig.  3 , this study considers SCP data of sample cities as the control variable and ranks them in ascending order based on TOPSIS results. We then examine changes in LCL data to ascertain the correlation between these variables, yielding two key research conclusions: on one hand, analysis of 52 sample cities demonstrates a general ascending trend in both SCP and LCL data curves. This trend suggests a positive correlation between these two parameters. On the other hand, the LCL data, in contrast to the consistent rise in SCP, exhibits notable fluctuations and wider dispersion. This indicates that the positive correlation between SCP and LCL, while present, is not markedly robust.

Correlation results of SCP and LCL in sample cities

Correlation analysis of urban SCP and overall-LCL. This analysis employs the closeness coefficient (C i ) to assess SCP and overall-LCL in sample cities for Hypothesis 1 in Eqs. ( 10 ) and ( 11 ). The results are presented in Table 2 . Additionally, a linear regression analysis is conducted to determine the presence and magnitude of the relationship between SCP and LCL in these cities, as shown in Fig.  4 .

The scatter and regression of SCP and LCL: ( A ) SCP & Overall-LCL; ( B ) SCP & LCM; ( C ) SCP & LCS; ( D ) SCP & LCE; ( E ) SCP & LCQE; ( F ) SCP & LCC.

Considering the closeness coefficient range, correlation is categorized into five levels: very weak ( \(\left|{\rho }_{AB}\right|<0\) .1), weak ( \(0.1\le \left|{\rho }_{AB}\right|<0\) .3), moderate ( \(0.3\le \left|{\rho }_{AB}\right|<0\) .5), strong ( \(0.5\le \left|{\rho }_{AB}\right|<0\) .7), and very strong ( \(0.7\le \left|{\rho }_{AB}\right|<1.0\) ) 73 . Table 1 indicates a strong positive correlation between SCP and overall LCL. Linear regression analysis in Fig.  4 A demonstrates a significant correlation between SCP and urban LCL ( R 2  = 0.42, p  < 0.001), with notable differences exist among cities, consistent with Hypothesis 1 .

Correlation analysis of SCP and each low-carbon dimension. Pearson correlation analysis effectively measures the strength of linear relationships between two variables, but it does not identify causal relationships between them. To address this limitation and explore the interaction between the two variables, this study sets and solves the closeness coefficient for each low-carbon dimension, which are low-carbon economy (C LCE ), low-carbon society (C LCS ), low-carbon environmental quality (C LCEQ ), low-carbon management (C LCM ), and low-carbon culture (C LCC ). It then calculates the correlation analysis results for SCP and each low-carbon dimension for Hypothesis 1 , as shown in Table 1 . Furthermore, the results of the linear regression analysis are presented in Fig.  4 .

In detail, strong correlations exist between SCP and LCM, LCS, and LCEQ. The correlation is moderate with LCE and weak with LCC. Furthermore, linear regression analysis shows that the links between SCP and low-carbon levels across five dimensions are significant with minimal variance. Cities with higher SCP typically show higher values in LCM ( R 2  = 0.38, p  = 0.000), LCS ( R 2  = 0.35, p  = 0.000), and LCE ( R 2  = 0.32, p  = 0.000) as depicted in Fig.  4 B–D. However, this trend is less pronounced in LCEQ ( R 2  = 0.17, p  = 0.000) and LCC ( R 2  = 0.06, p  = 0.001), which exhibit greater dispersion as shown in Fig.  4 E,F. The lower R 2 values for LCEQ and LCC compared to other dimensions suggest a greater influence of factors not included in the model. Furthermore, to ensure the credibility and reliability of the research findings, this study conducted a sensitivity analysis by identifying and removing outliers from the sample dataset using the Z-score method, in addition to the previously mentioned Pearson correlation analysis. The Pearson correlation coefficient for the original dataset of city SCP and LCL is 0.65, with a significant P-value. After removing the outliers, the Pearson correlation coefficient is 0.61, and the P-value remained significant. Therefore, the correlation between city SCP and LCL proposed in Research Hypothesis 1 is robust.

Coupling coordination degree of SCP and LCL in sample cities

The degree of coupling coordination comprehensively considers multiple aspects of urban complex systems, including economic, social, and environmental dimensions. By systematically evaluating the coordinated development level of urban SCP and LCL, this approach enables the analysis of the coupling and coordination relationships between SCP and LCL, as well as among various subsystems such as LCM, LCS, LCE, LCEQ, and LCC. This reveals the dynamic interactions and causality between SCP and LCL within urban complex systems. The coupling coordination degrees of SCP and LCL, along with their subsystems, in 52 typical smart and low-carbon pilot cities in China, are illustrated in Fig.  5 .

Coupled coordination degree of SCP and LCL, LCS, LCEQ, LCE, LCM, LCC.

Characteristics of objective changes in the coupled coordination degree between SCP and LCL. Based on the coupling coordination model and Eqs. ( 12 ) to ( 14 ), the coupling coordination degree of the urban complex system in SCP and LCL regions is calculated for Hypothesis 1 , as illustrated in Fig.  5 .

From the holistic perspective of urban complex systems, as the level of urban SCP continuously improves, the coupling coordination degree between SCP and LCL among 52 pilot cities in China shows an upward trend. This indicates that as the functional indices of urban SCP and LCL both strengthen, their interaction and coordination also enhance. Among these, Jincheng has the lowest coupled coordination degree at 0.5201, while Beijing boasts the highest at 0.8622. Within the 52 pilot cities, 5.78% exhibit a barely coupling coordination level, 51.93% display a primary coupling coordination level, 25% achieve an intermediate coupling coordination level, and 17.31% reach a good coupling coordination level. Moreover, the average coupling coordination degree of the 52 pilot cities is 0.598, suggesting that the SCP and LCL of the pilot cities can achieve coupled coordinated development.

Characteristics of objective changes in the coupled coordination degree among SCP, LCM, LCS, LCE, LCEQ, and LCC for Hypothesis 1 are illustrated in Fig.  5 .

From the perspective of urban subsystems, the coupling coordination degrees of LCS & SCP, LCE & SCP, and LCM & SCP all exhibit characteristics of steady fluctuations with an upward trend, while the coupling coordination degree of LCC & SCP shows greater volatility in its upward trend. The coupling coordination degree of LCEQ & SCP demonstrates a trend of initially rising and then declining. Furthermore, the average values of the coupling coordination degrees for LCS & SCP, LCE & SCP, LCM & SCP, LCEQ & SCP, and LCC & SCP are 0.478, 0.761, 0.779, 0.710, and 0.485, respectively. Among these, the pilot cities’ subsystems of LCE, LCM, and LCEQ with SCP exhibit an intermediate level of coupling coordination, while the coupling coordination degrees of LCS and LCC with SCP are on the verge of a dysfunctional recession. This indicates that the causal relationships between urban SCP and the subsystems of urban LCM, LCS, LCE, LCEQ, and LCC vary. Overall, Hypothesis 1 holds true both from the perspective of the city's overall system and from the perspective of its various subsystems.

Discussions and implications

Relationship between scp and lcl of different cities.

Considering the evaluation results of the urban SCP and LCL, four grades of the overall points can be classified, namely, excellent (0.7–1.0), average (0.5–0.7), below average (0.4–0.5), and poor (0–0.4). Subsequently, the sample cities in Supplementary Appendix Table A5 were classified based on these gradations. In the sample, cities with excellent SCP constitute 9.62%, about double the proportion with excellent LCL. Cities with average SCP account for 48.08%, whereas those at average LCL represent only 26.92%. Notably, cities with poor LCL comprise 26.92%, nearly triple the rate of those with poor SCP. The findings suggest that China’s SCP currently outperforms its low-carbon city initiatives, largely attributable to the rapid advancement of the Internet and Information and Communication Technology (ICT) in recent years. What’s more, Fig.  4 illustrates that urban SCP significantly positively influences the urban LCL, though substantial variations exist among different cities. The relevant types can be summarized into the following four categories.

Quadrant I-high SCP and high LCL, including only six cities (Shenzhen, Shanghai, Beijing, Ningbo, Xiamen, and Qingdao). These cities are not only among China’s earliest smart city pilots but also recent focus areas for the government’s “Carbon Peak Pioneer Cities” initiative. By actively exploring innovative models, systems, and technologies for smart and low-carbon co-development, these cities provide valuable practical experiences for others. For instance, Shenzhen has developed a multi-level, multi-component greenhouse gas monitoring network and technology system for “carbon flux, carbon concentration, carbon emissions”, while Ningbo has constructed a “smart zero-carbon” comprehensive demonstration port area.

Quadrant II-poor SCP and poor LCL, numerous cities in Fig.  4 A, such as Jincheng, Lhasa, and Urumqi, exhibit poor SCP and LCL. Despite China having the most smart and low-carbon city pilots globally, its development level in these areas still lags significantly behind typical developed countries. While China’s infrastructure like networking and computing power has reached a certain scale, issues persist with insufficient integration and intensity in infrastructure construction and operation, as well as problems with aging infrastructure and low levels of intelligence. Furthermore, although China’s low-carbon pilot cities have made positive progress in promoting low-carbon development, most still have incomplete carbon emission statistical systems and inadequate operational mechanisms, leading to generally poor overall low-carbon development levels.

Quadrant III-high LCL but poor SCP, such as Kunming, Xining, and Guiyang. These cities possess resources conducive to low-carbon development, such as Kunming and Guiyang with their rich forest carbon sinks, and Xining with abundant clean energy sources like solar and wind power. However, they are mostly situated in China’s central and southwest areas with underdeveloped physical and economic conditions. Leveraging their abundant low-carbon resources, and utilizing big data and IoT technology, achieving sustainable green economic growth through carbon credits and trading markets, as well as green finance, represents a significant future development direction for these cities.

Quadrant IV-high SCP but poor LCL, including Suzhou, and Jinhua Zhongshan, decoupling economic development from carbon emissions presents a significant development challenge for these cities. Specifically, for Suzhou, one of the world’s largest industrial cities, the main challenge is achieving decarburization in the energy sector and transitioning high-emission manufacturing industries to low-carbon alternatives.

What’s more, as illustrated in Fig.  5 , the degree of interaction between SCP and LCL across the 52 pilot cities in China positively impacts the balanced and comprehensive performance of these cities. This, in turn, fosters the coordinated development of urban systems as a whole. Moreover, the continual increase in the coupled coordination degree between SCP and LCL with the enhancement of SCP in pilot cities indicates that smart city construction contributes to urban low-carbon development. Future urban development in China should fully leverage the industrial upgrading effect, carbon sequestration effect, and energy utilization effect of smart city construction. However, the increasing slope of the SCP & LCL coupled coordination degree curve in Fig.  5 suggests significant regional differences in the level of SCP & LCL coupled coordination development across Chinese cities. Smart city construction has a more pronounced decarburization effect in central and western cities, southern cities, non-environmentally focused cities, and resource-based cities, with cities in the northwest showing notably poorer levels of SCP & LCL coupled coordination development. This serves as a warning for future urban development in China.

Relationships between SCP and LCL in each urban subsystem

The relationship between urban SCP and LCL across five dimensions is illustrated in Fig.  4 B–F. There is a strong positive correlation between SCP and LCM, LCS, and LCE, while a moderate correlation is observed with LCEQ, and a weak correlation with LCC. Furthermore, the degree of coupling coordination between SCP and subsystems such as LCS, LCEQ, LCE, LCM, and LCC is examined in Fig.  5 . The results of the coupling coordination vividly illustrate the synergistic interactions and developmental harmony between urban SCP and various systems.

Among these, the coupling coordination degree curve fluctuation between SCP & LCM is stable, situated at an intermediate coupling coordination level, indicating the dominant role of the Chinese government in the construction of smart cities and low-carbon cities, as well as the effectiveness of policy implementation. However, this also suggests that in promoting urban smart and low-carbon construction, China faces the risk of adopting “one-size-fits-all” mandatory policies, neglecting to advance construction in phases with emphasis, tailored to the city's resource endowment and economic development status. The coupling coordination degree curve changes between SCP&LCE and SCP&LCL show the highest degree of fit, indicating that low-carbon economic development brought about by digital empowerment and upgrading of the urban industrial structure is a key driving factor for promoting the coupled coordination development of urban smart and low-carbon initiatives. Transforming traditional industrial structures and pursuing low-carbon upgrades of the economic structure present challenges for urban development in China today. The coupled coordination degree of SCP & LCS is on the verge of a dysfunctional recession, highlighting the imbalance in the development between China's SCP and LCS, especially in terms of new infrastructure construction, such as smart transportation and logistics facilities, smart energy systems, smart environmental resources facilities, etc. The current construction of new infrastructure in China is far from meeting the living needs of the broad masses of people.

It is noteworthy that with the continuous improvement of the SCP in sample cities, the coupling performance degree between SCP and LCEQ exhibits two phases: an initial stage of synergistic enhancement followed by a stage of diminished synergy. In the early phase of synergistic development, the SCP and LCEQ systems of cities, driven by shared goals of sustainable urban development, strategy adjustments, resource sharing, and technological progress, facilitated effective collaboration and integration between systems. However, upon reaching a certain stage, intensified resource competition, declining management efficiency, and environmental changes led to internal system fatigue, resulting in weakened synergy. This indicates that once the technological effects generated by smart city construction reach a certain level, it becomes crucial to enhance the city's capacity for autonomous innovation. Addressing the bottleneck issues of core technologies and transforming the development mode of smart low-carbon technology from “imitative innovation” represent significant breakthroughs for further promoting the coupled coordination of SCP and LCEQ in China’s future.

Moreover, as the SCP of sample cities continuously improves, the coupled coordination degree between SCP and LCC shows two phases: initial stable fluctuations and subsequent rapid growth. The turning point in the curve change occurs at a coupled coordination degree of 0.6, denoted as the primary coupling coordination point. Among these, the low-carbon awareness rate of urban residents, as a key indicator of LCC, shows that the majority of urban residents in China are still in the cognitive awakening stage regarding low-carbon consciousness. At this stage, residents begin to recognize the severity of climate change and environmental degradation, along with the importance of smart low-carbon lifestyles in mitigating these issues. The government continuously promotes this awareness through media reports, educational activities, official propaganda, and community initiatives. As residents gain a deeper understanding of the issues, their attitudes shift from initial indifference or skepticism to a stronger identification with and support for the values and concepts of smart low-carbon living. This shift encourages residents to experiment with new smart low-carbon lifestyles, gradually finding suitable smart low-carbon behavioral patterns that become habitual. Ultimately, when smart low-carbon lifestyles are fully internalized as part of residents’ values, they not only practice smart low-carbon living at the individual level but also actively participate in promoting society’s smart low-carbon construction. Therefore, this study posits that the emergence of the coupled coordination degree turning point between SCP and LCC is not only a process of individual behavioral change but also a reflection of social and cultural transformation. This process is time-consuming and influenced by multiple factors, including policy guidance, economic incentives, educational dissemination, and the social atmosphere.

Implications for promoting coupling coordination development between urban SCP and LCL

Low-carbon and smartness are vital features of modern, sustainable urban development and key supports for it. This study posits that urban low-carbon and smart development should not be disjointed but rather synergistic and complementary. To better achieve sustainable urban development goals, a model should be constructed with “low-carbon” as the cornerstone of sustainable development and “smartness” as the technological assurance for low-carbon growth. Specifically, this study proposes the “urban smart low-carbon co-development model”, which entails a deep integration of intelligent technologies such as the Internet of Things (IoT) and big data with urban construction, governance services, and economic development. This model leverages digitalization to facilitate decarburization, thereby achieving urban sustainable development goals such as energy-efficient and green urbanization, ecological and livable environments, and streamlined governance services.

Furthermore, to better coordinate smart development with low-carbon city construction, enhance low-carbon city building through digitalization, and explore exemplary practices and models of smart low-carbon city construction, this study finds it necessary to establish an evaluation system for smart and low-carbon urban co-development. Therefore, based on the aforementioned urban SCP and LCL evaluation indicator system, this study initially conducted a literature review of past research, selecting 5 primary indicators and 20 secondary indicators from 48 articles to evaluate the degree of coupling coordination development between urban SCP and LCL. Subsequently, the Delphi method was employed to finalize the list of evaluation indicators, with 10 experts from various regions and diverse backgrounds in China refining the list and determining the weights of each indicator, as shown in Supplementary Appendix Table A6 . The final Smart Low-Carbon City Coupling Coordination Development Evaluation Indicator System, as presented in Table 3 , comprises 5 primary indicators and 18 secondary indicators. This evaluation system aims to emphasize the utilization of next-generation information technologies such as 5G, artificial intelligence, cloud computing, and blockchain to expand urban green ecological spaces, strengthen ecological environment governance, and enhance the level of intelligent urban governance, meeting the development needs of smart low-carbon cities.

The policy implications from the analysis results suggest that actions should be taken by government departments in China to reduce the uneven performance between urban SCP and LCL across various cities. These actions include, for example: Firstly, guiding the innovative development of urban SCP and LCL through policies, such as enhancing government digital services and administrative platforms, continuously promoting the development of emerging industries and the upgrading of traditional industries, and actively promoting green energy technologies. Secondly, categorizing and advancing the coordinated development of smart and low-carbon cities—comprehensive development should be pursued simultaneously in large cities in eastern and central China, while in smaller cities in western China, priorities should include enhancing urban innovation capabilities and improving infrastructure to lay a solid foundation for the coupled coordination of urban SCP and LCL. Thirdly, constructing a multi-stakeholder governance system to maximize the leading role of the government, the main role of enterprises, and the active participation of residents. By fostering a positive social atmosphere and cultural attributes, this will enhance the sense of participation and achievement among different social groups, creating a sustainable development model for urban SCP and LCL coordination. Lastly, emphasizing the development of SCP and LCL coordination in county-level cities is crucial. While large Chinese cities have already begun to form a pattern of coordinated SCP and LCL development, county-level cities, though with weaker infrastructures, possess tremendous potential. Focusing on low-carbon production, circulation, and consumption, and strengthening smart and low-carbon constructions in county-level cities will be a vital task for future urban development in China.

Conclusions

The global urbanization process brings opportunities for economic growth and social development, but also presents a series of challenges, such as environmental pressures and resource constraints 3 . The evaluation of urban SCP and LCL creates a link between the policy-making in urban resources environment management and the objectives of sustainable development goals (SDGs 11.4, 11.6, and 11.b) at the city level 74 . Currently, there is no unified consensus on the coupling coordination development between urban SCP and LCL. This study proposes a method combining qualitative and quantitative analysis from the perspective of urban complex systems to analyze the coupling coordination relationship between SCP and LCL. This new method clearly interprets a strong positive correlation between urban smart performance and the overall low-carbon level. Specifically, there are strong correlations between SMC and LCM, LCS, and LCE, with a moderate correlation to LCQE and a weak correlation with LCC. Several innovative insights for this method are highlighted: (i) sustainable development based on SCP and LCL assessment; (ii) emphasizing the “people-centric” concept in urban development; (iii) analyzing from the perspective of urban complex systems.

This study selected 52 typical smart and low-carbon pilot cities in China as sample cities to analyze the coupled coordination relationship between urban SCP and LCL. And the main findings from this analysis can be summarized as follows: (i) smart city initiatives outperform low-carbon city development, with notable differences in SCP and LCL effectiveness across eastern, central, and non-resource-based cities versus western, peripheral, and resource-dependent ones in China. (ii) A strong positive link between urban SCP and low-carbon levels, especially between SCP and LCM, LCS, and LCE, with moderate and weak correlations to LCEQ and LCC, respectively. (iii) An increasing urban SCP levels enhance the coupling coordination within the urban SCP and LCL system. SCP & LCE, SCP & LCM, and SCP & LCS subsystems align well with the overall system, driving the coupled coordination of urban SCP and LCL. In contrast, SCP & LCC and SCP & LCEQ have lesser alignment, affected by factors like technology, policy, economic incentives, education, and societal attitudes. Based on the evaluation results, this study posits that the development of urban low-carbon and smart initiatives should not be disjointed but rather synergistic and complementary. This study constructs an evaluation indicator system for the co-development of smart low-carbon cities aimed at better guiding the future coupling coordination development of smart and low-carbon cities.

The novelty of this study not only addresses the practical dilemma of obtaining comprehensive, accurate, and timely urban-level carbon emission data, a challenge due to existing measurement and estimation technologies being unable to capture all types of carbon emissions, but also assesses the urban SCP and LCL. Simultaneously, by combining qualitative and quantitative analysis methods, it fills the research gap on the nature of the coupled coordination relationship between urban SCP and LCL. Moreover, from the perspective of urban complex systems, this study dissects the urban low-carbon level into LCC, LC, LCE, LCEQ, and LCS, exploring their respective coupled coordination relationships with SCP. This clarifies the impact mechanism between SCP and LCL, providing a theoretical basis for smart low-carbon city co-development. The limitations of the study are also appreciated. Firstly, the study only selected a sample of cities in China, and the limited number of samples may not fully substantiate the research conclusions. Secondly, the indicator system constructed by this study is still not perfect, leading to certain inaccuracies in the evaluation results. In this regard, future studies are recommended to conduct a more comprehensive comparison analysis on the coupled coordination relationship between SCP and LCL at city, regional, and national levels, which would be beneficial in better guiding the practice of urban sustainability.

Data availability

All data generated or analysed during this study are included in this published article [and its Supplementary Information files].

Zheng, H. W., Shen, G. Q. & Wang, H. A review of recent studies on sustainable urban renewal. Habit. Int. 41 , 272–279. https://doi.org/10.1016/j.habitatint.2013.08.006 (2014).

Article   CAS   Google Scholar  

Bibri, S. E. & Krogstie, J. Smart sustainable cities of the future: An extensive interdisciplinary literature review. Sustain. Cities Soc. 31 , 183–212. https://doi.org/10.1016/j.scs.2017.02.016 (2017).

Article   Google Scholar  

Chen, M., Liu, W. & Lu, D. Challenges and the way forward in China’s new-type urbanization. Land Use Policy 55 , 334–339. https://doi.org/10.1016/j.landusepol.2015.07.025 (2016).

Liang, W. & Yang, M. Urbanization, economic growth and environmental pollution: Evidence from China. Sustain. Comput. Inform. Syst. 21 , 1–9. https://doi.org/10.1016/j.suscom.2018.11.007 (2019).

Guan, X., Wei, H., Lu, S., Dai, Q. & Su, H. Assessment on the urbanization strategy in China: Achievements, challenges and reflections. Habit. Int. 71 , 97–109. https://doi.org/10.1016/j.habitatint.2017.11.009 (2018).

Wu, H., Hao, Y. & Weng, J.-H. How does energy consumption affect China’s urbanization? New evidence from dynamic threshold panel models. Energy Policy 127 , 24–38. https://doi.org/10.1016/j.enpol.2018.11.057 (2019).

Liu, H., Cui, W. & Zhang, M. Exploring the causal relationship between urbanization and air pollution: Evidence from China. Sustain. Cities Soc. 80 , 783. https://doi.org/10.1016/j.scs.2022.103783 (2022).

Tang, F. et al. Spatio-temporal variation and coupling coordination relationship between urbanisation and habitat quality in the Grand Canal, China. Land Use Policy 117 , 6119. https://doi.org/10.1016/j.landusepol.2022.106119 (2022).

Kim, J. Smart city trends: A focus on 5 countries and 15 companies. Cities 123 , 551. https://doi.org/10.1016/j.cities.2021.103551 (2022).

Silva, B. N., Khan, M. & Han, K. Towards sustainable smart cities: A review of trends, architectures, components, and open challenges in smart cities. Sustain. Cities Soc. 38 , 697–713. https://doi.org/10.1016/j.scs.2018.01.053 (2018).

Yigitcanlar, T., Kankanamge, N. & Vella, K. How are smart city concepts and technologies perceived and utilized? A systematic geo-twitter analysis of smart cities in Australia. J. Urban Technol. 28 , 135–154. https://doi.org/10.1080/10630732.2020.1753483 (2021).

Yigitcanlar, T. et al. Can cities become smart without being sustainable? A systematic review of the literature. Sustain. Cities Soc. 45 , 348–365. https://doi.org/10.1016/j.scs.2018.11.033 (2019).

Guo, Q. & Zhong, J. The effect of urban innovation performance of smart city construction policies: Evaluate by using a multiple period difference-in-differences model. Technol. Forecast. Soc. Change 184 , 2003. https://doi.org/10.1016/j.techfore.2022.122003 (2022).

Ismagilova, E., Hughes, L., Dwivedi, Y. K. & Raman, K. R. Smart cities: Advances in research—An information systems perspective. Int. J. Inf. Manag. 47 , 88–100. https://doi.org/10.1016/j.ijinfomgt.2019.01.004 (2019).

Caragliu, A. & Del Bo, C. F. Smart innovative cities: The impact of Smart City policies on urban innovation. Technol. Forecast. Soc. Change 142 , 373–383. https://doi.org/10.1016/j.techfore.2018.07.022 (2019).

Yigitcanlar, T. et al. Understanding ‘smart cities’: Intertwining development drivers with desired outcomes in a multidimensional framework. Cities 81 , 145–160. https://doi.org/10.1016/j.cities.2018.04.003 (2018).

Liu, Z. et al. Decision optimization of low-carbon dual-channel supply chain of auto parts based on smart city architecture. Complexity 2020 , 5951. https://doi.org/10.1155/2020/2145951 (2020).

Guo, Q., Wang, Y. & Dong, X. Effects of smart city construction on energy saving and CO 2 emission reduction: Evidence from China. Appl. Energy 313 , 879. https://doi.org/10.1016/j.apenergy.2022.118879 (2022).

Cheng, J., Yi, J., Dai, S. & Xiong, Y. Can low-carbon city construction facilitate green growth? Evidence from China’s pilot low-carbon city initiative. J. Clean. Prod. 231 , 1158–1170. https://doi.org/10.1016/j.jclepro.2019.05.327 (2019).

Sun, W. & Huang, C. Predictions of carbon emission intensity based on factor analysis and an improved extreme learning machine from the perspective of carbon emission efficiency. J. Clean. Prod. 338 , 414. https://doi.org/10.1016/j.jclepro.2022.130414 (2022).

Shi, B., Li, N., Gao, Q. & Li, G. Market incentives, carbon quota allocation and carbon emission reduction: Evidence from China’s carbon trading pilot policy. J. Environ. Manag. 319 , 650. https://doi.org/10.1016/j.jenvman.2022.115650 (2022).

Sun, L. et al. Carbon emission transfer strategies in supply chain with lag time of emission reduction technologies and low-carbon preference of consumers. J. Clean. Prod. 264 , 664. https://doi.org/10.1016/j.jclepro.2020.121664 (2020).

Matsumura, E. M., Prakash, R. & Vera-Munoz, S. C. Firm-value effects of carbon emissions and carbon disclosures. Acc. Rev. 89 , 695–724. https://doi.org/10.2308/accr-50629 (2014).

Lv, M. & Bai, M. Evaluation of China’s carbon emission trading policy from corporate innovation. Financ. Res. Lett. 39 , 565. https://doi.org/10.1016/j.frl.2020.101565 (2021).

Jia, Z. & Lin, B. Rethinking the choice of carbon tax and carbon trading in China. Technol. Forecast. Soc. Change 159 , 187. https://doi.org/10.1016/j.techfore.2020.120187 (2020).

Huo, T., Xu, L., Liu, B., Cai, W. & Feng, W. China’s commercial building carbon emissions toward 2060: An integrated dynamic emission assessment model. Appl. Energy 325 , 828. https://doi.org/10.1016/j.apenergy.2022.119828 (2022).

Lin, B. & Huang, C. Analysis of emission reduction effects of carbon trading: Market mechanism or government intervention? Sustain. Prod. Consump. 33 , 28–37. https://doi.org/10.1016/j.spc.2022.06.016 (2022).

Zhang, M. & Liu, Y. Influence of digital finance and green technology innovation on China’s carbon emission efficiency: Empirical analysis based on spatial metrology. Sci. Total Environ. 838 , 463. https://doi.org/10.1016/j.scitotenv.2022.156463 (2022).

Zhu, X. & Li, D. How to promote the construction of low-carbon cities in China? An urban complex ecosystem perspective. Sustain. Dev. https://doi.org/10.1002/sd.2897 (2024).

He, C., Zhang, D., Huang, Q. & Zhao, Y. Assessing the potential impacts of urban expansion on regional carbon storage by linking the LUSD-urban and InVEST models. Environ. Model. Softw. 75 , 44–58. https://doi.org/10.1016/j.envsoft.2015.09.015 (2016).

Nowak, D. J., Greenfield, E. J., Hoehn, R. E. & Lapoint, E. Carbon storage and sequestration by trees in urban and community areas of the United States. Environ. Pollut. 178 , 229–236. https://doi.org/10.1016/j.envpol.2013.03.019 (2013).

Article   CAS   PubMed   Google Scholar  

Wang, T. et al. Mobility based trust evaluation for heterogeneous electric vehicles network in smart cities. IEEE Trans. Intell. Transp. Syst. 22 , 1797–1806. https://doi.org/10.1109/tits.2020.2997377 (2021).

Huovila, A., Bosch, P. & Airaksinen, M. Comparative analysis of standardized indicators for Smart sustainable cities: What indicators and standards to use and when? Cities 89 , 141–153. https://doi.org/10.1016/j.cities.2019.01.029 (2019).

Nizetic, S., Djilali, N., Papadopoulos, A. & Rodrigues, J. J. P. C. Smart technologies for promotion of energy efficiency, utilization of sustainable resources and waste management. J. Clean. Prod. 231 , 565–591. https://doi.org/10.1016/j.jclepro.2019.04.397 (2019).

Sharifi, S., Saman, W. & Alemu, A. Identification of overheating in the top floors of energy-efficient multilevel dwellings. Energy Build. https://doi.org/10.1016/j.enbuild.2019.109452 (2019).

Shafiq, M., Tian, Z., Sun, Y., Du, X. & Guizani, M. Selection of effective machine learning algorithm and Bot–IoT attacks traffic identification for internet of things in smart city. Future Gener. Comput. Syst. Int. J. Esci. 107 , 433–442. https://doi.org/10.1016/j.future.2020.02.017 (2020).

Huang, S., Liu, A., Zhang, S., Wang, T. & Xiong, N. N. BD-VTE: A novel baseline data based verifiable trust evaluation scheme for smart network systems. IEEE Trans. Netw. Sci. Eng. 8 , 2087–2105. https://doi.org/10.1109/tnse.2020.3014455 (2021).

Reed, M. S. et al. Evaluating impact from research: A methodological framework. Res. Policy 50 , 147. https://doi.org/10.1016/j.respol.2020.104147 (2021).

Venable, J., Pries-Heje, J. & Baskerville, R. FEDS: A framework for evaluation in design science research. Eur. J. Inf. Syst. 25 , 77–89. https://doi.org/10.1057/ejis.2014.36 (2016).

Kristan, M. et al. A novel performance evaluation methodology for single-target trackers. IEEE Trans. Pattern Anal. Mach. Intell. 38 , 2137–2155. https://doi.org/10.1109/tpami.2016.2516982 (2016).

Article   PubMed   Google Scholar  

Li, H. Research progress on evaluation methods and factors influencing shale brittleness: A review. Energy Rep. 8 , 4344–4358. https://doi.org/10.1016/j.egyr.2022.03.120 (2022).

Lyu, H.-M., Zhou, W.-H., Shen, S.-L. & Zhou, A.-N. Inundation risk assessment of metro system using AHP and TFN-AHP in Shenzhen. Sustain. Cities Soc. 56 , 103. https://doi.org/10.1016/j.scs.2020.102103 (2020).

Buyukozkan, G. & Guleryuz, S. An integrated DEMATEL-ANP approach for renewable energy resources selection in Turkey. Int. J. Prod. Econ. 182 , 435–448. https://doi.org/10.1016/j.ijpe.2016.09.015 (2016).

Ervural, B. C., Zaim, S., Demirel, O. F., Aydin, Z. & Delen, D. An ANP and fuzzy TOPSIS-based SWOT analysis for Turkey’s energy planning. Renew. Sustain. Energy Rev. 82 , 1538–1550. https://doi.org/10.1016/j.rser.2017.06.095 (2018).

Gao, Z. et al. EEG-based spatio-temporal convolutional neural network for driver fatigue evaluation. IEEE Trans. Neural Netw. Learn. Syst. 30 , 2755–2763. https://doi.org/10.1109/tnnls.2018.2886414 (2019).

Manzano, A. The craft of interviewing in realist evaluation. Evaluation 22 , 342–360. https://doi.org/10.1177/1356389016638615 (2016).

Zeng, S., Jin, G., Tan, K. & Liu, X. Can low-carbon city construction reduce carbon intensity? Empirical evidence from low-carbon city pilot policy in China. J. Environ. Manag. 332 , 363. https://doi.org/10.1016/j.jenvman.2023.117363 (2023).

Liu, X., Li, Y., Chen, X. & Liu, J. Evaluation of low carbon city pilot policy effect on carbon abatement in China: An empirical evidence based on time-varying DID model. Cities 123 , 582. https://doi.org/10.1016/j.cities.2022.103582 (2022).

Tan, S. et al. A holistic low carbon city indicator framework for sustainable development. Appl. Energy 185 , 1919–1930. https://doi.org/10.1016/j.apenergy.2016.03.041 (2017).

Article   ADS   Google Scholar  

Shi, X. & Xu, Y. Evaluation of China’s pilot low-carbon city program: A perspective of industrial carbon emission efficiency. Atmos. Pollut. Res. 13 , 446. https://doi.org/10.1016/j.apr.2022.101446 (2022).

Yang, S., Pan, Y. & Zeng, S. Decision making framework based Fermatean fuzzy integrated weighted distance and TOPSIS for green low-carbon port evaluation. Eng. Appl. Artif. Intell. 114 , 5048. https://doi.org/10.1016/j.engappai.2022.105048 (2022).

Fang, G., Gao, Z., Tian, L. & Fu, M. What drives urban carbon emission efficiency?—Spatial analysis based on nighttime light data. Appl. Energy 312 , 772. https://doi.org/10.1016/j.apenergy.2022.118772 (2022).

Yang, S., Jahanger, A. & Hossain, M. R. How effective has the low-carbon city pilot policy been as an environmental intervention in curbing pollution? Evidence from Chinese industrial enterprises. Energy Econ. 118 , 523. https://doi.org/10.1016/j.eneco.2023.106523 (2023).

Huang, G., Li, D., Zhu, X. & Zhu, J. Influencing factors and their influencing mechanisms on urban resilience in China. Sustain. Cities Soc. 74 , 210. https://doi.org/10.1016/j.scs.2021.103210 (2021).

Li, W. et al. Carbon emission and economic development trade-offs for optimizing land-use allocation in the Yangtze River Delta, China. Ecol. Indic. 147 , 950. https://doi.org/10.1016/j.ecolind.2023.109950 (2023).

Wu, H. et al. Exploring the impact of urban form on urban land use efficiency under low-carbon emission constraints: A case study in China’s Yellow River Basin. J. Environ. Manag. 311 , 866. https://doi.org/10.1016/j.jenvman.2022.114866 (2022).

Zhao, S. et al. Has China’s low-carbon strategy pushed forward the digital transformation of manufacturing enterprises? Evidence from the low-carbon city pilot policy. Environ. Impact Assess. Rev. 102 , 184. https://doi.org/10.1016/j.eiar.2023.107184 (2023).

Pan, A., Zhang, W., Shi, X. & Dai, L. Climate policy and low-carbon innovation: Evidence from low-carbon city pilots in China. Energy Econ. 112 , 129. https://doi.org/10.1016/j.eneco.2022.106129 (2022).

De Jong, M., Joss, S., Schraven, D., Zhan, C. & Weijnen, M. Sustainable-smart-resilient-low carbon-eco-knowledge cities; making sense of a multitude of concepts promoting sustainable urbanization. J. Clean. Prod. 109 , 25–38. https://doi.org/10.1016/j.jclepro.2015.02.004 (2015).

He, B.-J. et al. Co-benefits approach: Opportunities for implementing sponge city and urban heat island mitigation. Land Use Policy 86 , 147–157. https://doi.org/10.1016/j.landusepol.2019.05.003 (2019).

Nizetic, S., Solic, P., Lopez-de-Ipina, D. & Patrono, L. Internet of Things (IoT): Opportunities, issues and challenges towards a smart and sustainable future. J. Clean. Prod. 274 , 877. https://doi.org/10.1016/j.jclepro.2020.122877 (2020).

Abduljabbar, R. L., Liyanage, S. & Dia, H. The role of micro-mobility in shaping sustainable cities: A systematic literature review. Transp. Res. D Transp. Environ. 92 , 734. https://doi.org/10.1016/j.trd.2021.102734 (2021).

Anh Tuan, H., Van Viet, P. & Xuan Phuong, N. Integrating renewable sources into energy system for smart city as a sagacious strategy towards clean and sustainable process. J. Clean. Prod. 305 , 7161. https://doi.org/10.1016/j.jclepro.2021.127161 (2021).

March, H. & Ribera-Fumaz, R. Smart contradictions: The politics of making Barcelona a self-sufficient city. Eur. Urban Reg. Stud. 23 , 816–830. https://doi.org/10.1177/0969776414554488 (2016).

Yigitcanlar, T. & Lee, S. H. Korean ubiquitous-eco-city: A smart-sustainable urban form or a branding hoax? Technol. Forecast. Soc. Change 89 , 100–114. https://doi.org/10.1016/j.techfore.2013.08.034 (2014).

Kumar, S., Sharma, D., Rao, S., Lim, W. M. & Mangla, S. K. Past, present, and future of sustainable finance: Insights from big data analytics through machine learning of scholarly research. Ann. Oper. Res. https://doi.org/10.1007/s10479-021-04410-8 (2022).

Article   PubMed   PubMed Central   Google Scholar  

Li, Y., Gao, P., Tang, B., Yi, Y. & Zhang, J. Double feature extraction method of ship-radiated noise signal based on slope entropy and permutation entropy. Entropy 24 , 22. https://doi.org/10.3390/e24010022 (2022).

Article   ADS   CAS   Google Scholar  

Zavadskas, E. K., Mardani, A., Turskis, Z., Jusoh, A. & Nor, K. M. D. Development of TOPSIS method to solve complicated decision-making problems: An overview on developments from 2000 to 2015. Int. J. Inf. Technol. Decis. Making 15 , 645–682. https://doi.org/10.1142/s0219622016300019 (2016).

Edelmann, D., Mori, T. F. & Szekely, G. J. On relationships between the Pearson and the distance correlation coefficients. Stat. Probab. Lett. 169 , 960. https://doi.org/10.1016/j.spl.2020.108960 (2021).

Article   MathSciNet   Google Scholar  

Wang, S., Kong, W., Ren, L., Zhi, D. & Dai, B. Research on misuses and modification of coupling coordination degree model in China. J. Nat. Resour. 36 , 793–810 (2021).

CAS   Google Scholar  

Baak, M., Koopman, R., Snoek, H. & Klous, S. A new correlation coefficient between categorical, ordinal and interval variables with Pearson characteristics. Comput. Stat. Data Anal. 152 , 7043. https://doi.org/10.1016/j.csda.2020.107043 (2020).

Saadatmorad, M., Talookolaei, R.-A.J., Pashaei, M.-H., Khatir, S. & Wahab, M. A. Pearson correlation and discrete wavelet transform for crack identification in steel beams. Mathematics 10 , 689. https://doi.org/10.3390/math10152689 (2022).

De Winter, J. C. F., Gosling, S. D. & Potter, J. Comparing the Pearson and Spearman correlation coefficients across distributions and sample sizes: A tutorial using simulations and empirical data. Psychol. Methods 21 , 273–290. https://doi.org/10.1037/met0000079 (2016).

Belz, F. M. & Binder, J. K. Sustainable entrepreneurship: A convergent process model. Bus. Strategy Environ. 26 , 1–17. https://doi.org/10.1002/bse.1887 (2017).

Download references

The funding provided by Government of Jiangsu Province (BE2022606 and BM2022035).

Author information

Authors and affiliations.

Department of Construction and Real Estate, School of Civil Engineering, Southeast University, Nanjing, 210018, China

Xiongwei Zhu, Dezhi Li, Shenghua Zhou & Lugang Yu

Engineering Research Center of Building Equipment, Energy, and Environment, Ministry of Education, Southeast University, Nanjing, 210018, China

Department of Wood Science, The University of British Columbia, Vancouver, V6T 1Z4, Canada

You can also search for this author in PubMed   Google Scholar

Contributions

X.Z: conceptualization, methodology, formal analysis, investigation, writing-original draft. D. L: supervision, project administration, funding acquisition. S. Z: writing-review & editing. S.Z: writing-review & editing. L.Y: data curation.

Corresponding author

Correspondence to Dezhi Li .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary tables., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Zhu, X., Li, D., Zhou, S. et al. Evaluating coupling coordination between urban smart performance and low-carbon level in China’s pilot cities with mixed methods. Sci Rep 14 , 20461 (2024). https://doi.org/10.1038/s41598-024-68417-4

Download citation

Received : 28 March 2024

Accepted : 23 July 2024

Published : 03 September 2024

DOI : https://doi.org/10.1038/s41598-024-68417-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Smart city performance (SCP)
• Low-carbon level (LCL)
• ETPC analysis model
• Coupling coordination degree
• Smart low-carbon coupling coordination development paths

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Anthropocene newsletter — what matters in anthropocene research, free to your inbox weekly.