The umbrella review, based on a publicly available protocol (PROSPERO CRD42022296284), was conducted and reported according to relevant guidelines16,17,36, including the U-REACH methodological guidelines18 and the Preferred Reporting Items for Overviews of Reviews (PRIOR) reporting guidelines37. The PRIOR checklist is available in Supplementary Information section 1.
Search strategy and eligibility criteria
We searched MEDLINE, Web of Science, Embase, CINAHL and PsycINFO with terms related to two constructs (‘autism’ and ‘meta-analysis’), with no language, publication type or date of publication restrictions, up to 31 December 2023. The list of full search terms is reported in Supplementary Information section 2. Screening of the titles and abstracts, as well as study selection, was performed independently by the first author and several members of the team. Disagreements were resolved by the senior authors (R.D., S.C. and M.S.). References of included studies and Google Scholar were searched to identify additional eligible references, but all relevant reports found with these manual searches were already included in the database searches.
We included systematic reviews coupled with a meta-analysis of both randomized and non-randomized CCTs that assessed the efficacy of any CAIM on both core autism symptoms and key autism-related symptoms in autistic participants of any age. Contrary to what was envisioned in the protocol, we chose to include both randomized and non-randomized controlled trials, rather than including only randomized controlled trials. This choice was made because, in the field of autism, several promising types of intervention—such as long-term or intensive interventions—are difficult to assess in randomized trials. This choice ensures a complete and consistent mapping of the literature regardless of the intervention type. Even if less than 10% of studies were non-randomized studies in our primary analysis, we performed a sensitivity analysis restricted to randomized clinical trials.
A review was considered ‘systematic’ if it was identified as such by its authors and searched at least two scientific databases in combination with explicit inclusion and exclusion criteria. The definition of autism followed that used by primary authors, typically in line with international classifications (Diagnostic and Statistical Manual of Mental Disorders from 3rd to 5th editions, or International Classification of Diseases (ICD) 9 or 10). Based on the mean age of the participants within each meta-analysis, we grouped the presentation of the results into four distinct age groups: (1) preschool children (mean age ranging from 0 to 5 years), (2) school-aged children (6–12 years), (3) adolescents (13–19 years), and (iv) adults (≥20 years). For each meta-analysis, we evaluated whether the average age of the participants from the different CCTs was homogeneous (see our exact criteria in Supplementary Information section 3).
In accordance with the standard classification of the National Institutes of Health National Center for Complementary and Alternative Medicine6,7, we identified 19 CAIM types (see the list and complete description at https://ebiact-database.com/interventions/). The prespecified primary outcomes of interest were autism core symptoms (overall symptoms, social communication impairment, restricted/stereotyped/repetitive behaviours and sensory peculiarities) and CAIM-related safety (acceptability (risk of all-cause discontinuation), tolerability (risk of discontinuation due to treatment-related adverse events) and adverse events (risk of experiencing at least one or any specific adverse events)). Secondary outcomes were language skills (overall language, receptive language and expressive language), functioning (overall cognitive functioning, adaptive behaviours and quality of life), disruptive behaviours and psychiatric comorbidities (attention deficit hyperactivity disorder (ADHD), anxiety and emotional/depressive symptoms). We included sleep quality and quantity as post-hoc outcomes, given the significance of sleep for autistic individuals and their families that emerged from the screened papers38.
Data extraction and checking
As described in detail in Supplementary Information section 4, we extracted information regarding the characteristics of clinical trials (for example, risk of bias), participants (for example, mean age) and interventions (for example, dosage) from meta-analytic reports. In instances where the age of the participants was not reported in the meta-analysis, we obtained this directly from the report describing the clinical trial. Similarly, when the estimated effect sizes of individual studies were deemed unplausible (for example, SMD ≥5), we conducted the necessary checks using the metaConvert R package (version 1.0.2)39 and excluded meta-analyses containing inaccuracies or errors. More details on data extraction and data quality checks are available in Supplementary Information sections 4 and 5.
Assessment of the methodological quality
In line with recommendations for umbrella reviews16,17,18,36, we obtained the quality of primary studies by extracting this information directly from the meta-analytic reports. We evaluated the quality of the meta-analyses using the AMSTAR-2 tool40. The AMSTAR-2 scoring was performed independently by the first author and several members of the team.
Overlapping meta-analyses
Following the identification of several overlapping meta-analyses, that is, independent meta-analyses that assessed the same PICO combination, the most recent meta-analysis with the highest methodological quality was selected for the primary analysis, rather than the largest meta-analysis (see more details in ‘Main deviations from the protocol’ section). Then, we assessed the concordance of the results of overlapping meta-analyses, such as the percentage overlap of the 95% CI around the pooled effect size or the agreement on the GRADE ranking, in a secondary analysis. More details on the selection process for overlapping meta-analyses are available in Supplementary Information section 6.
Assessment of the levels of evidence
When registering the protocol, we did not describe any specific system for assessing the levels of evidence regarding the effects of each intervention. This was because, at that time, there was no consensus on the best approach in umbrella reviews of RCTs, given that the traditional gold-standard framework for meta-analyses of interventions (the GRADE framework) becomes difficult to use when the number of comparisons to grade is high. In this Article, we chose to rely on the algorithmic version of the GRADE framework recently implemented in the metaumbrella R software. These criteria are inspired by standard guidelines for rating the levels of evidence in large evidence syntheses, such as Confidence in Network Meta-Analysis (CINeMA)32. For all meta-analyses, the levels of evidence started with a ‘high’ ranking and could then be downgraded depending on the presence of risk of bias, inconsistency, indirectness, imprecision and publication bias (Table 1).
Development and update of the EBIA-CT platform
To tackle the recognized gap between scientific evidence about CAIM in autism and its accessibility for clinicians and both autistic persons and their families, we developed the EBIA-CT platform (https://ebiact-database.com/) to disseminate our results in a user-friendly manner. Building upon a preliminary version created during our prior work on psychosocial interventions15, we significantly enhanced the platform design and features during this project. This improvement process was informed by qualitative feedback from clinicians, stakeholders and researchers, ensuring the platform is user-friendly and effectively disseminates the findings of this umbrella review. Integrating the EBIA-CT platform into our umbrella review methodology allowed us to proactively adhere to the U-REACH guidelines, a framework specifically designed to promote the wide dissemination of umbrella review results. Furthermore, the platform is designed as a ‘living resource’, with planned annual updates for 5 years, following our established methodology and aligning with living review principles. In terms of ethical considerations, no new primary data were collected for this umbrella review or platform development. Therefore, formal ethics committee consultation was not deemed necessary, as is standard practice for projects focused on synthesizing and disseminating existing research findings. The platform is organized into three main sections:
-
(1)
Interventions: This section provides comprehensive details about each intervention we identified, including the target population (for example, age range and cognitive functioning level), implementation guidelines and meta-analysis results across age groups. It serves as a valuable resource for autistic individuals and their families seeking information on specific interventions.
-
(2)
Preferences: In this section, users can explore age-specific intervention outcomes, providing an overall mapping of the efficacy and safety of all interventions. This section supports informed discussions between clinicians and autistic individuals or their families, facilitating evidence-based decision-making tailored to individual needs.
-
(3)
Database: This section provides an overview of the information stored in the platform, and grants access to the raw data from the umbrella review, enabling users to download datasets for external analysis. It is especially useful for researchers identifying knowledge gaps and for guideline developers formulating recommendations.
Data analysis
As described in more detail in Supplementary Information section 7, all analyses were performed in the R environment (version 4.1.1) using the ‘metaumbrella’ package (version 1.1.0)41. We used the SMD as the main effect size measure for the assessment of efficacy, and the RR as the main effect size measure for the assessment of safety (that is, acceptability, tolerability and adverse events). Although we preregistered that the effect measure would be SMD only, we preferred analysing these safety outcomes on their natural effect measures, given their dichotomous nature. For meta-analyses of mean difference and OR, we converted this information for each trial to SMD (respectively RR) before conducting the calculations. Regardless of the metric used (RR or SMD), the direction of the effect was reversed when needed, such that a positive effect size (RR >1 or SMD >0) systematically reflected an improvement, that is, a symptom reduction, a competence improvement or a high safety.
We reran each meta-analysis identified in the review, to ensure consistent calculations and assessments of the level of evidence across meta-analyses. We systematically used a random-effects model with a restricted maximum likelihood estimator, or a Paul–Mandel estimator for τ2. The CI for the pooled effect size was based on the standard normal quantile method. The I2, Q statistics and τ2 statistics, as well as the 95% prediction interval, were used to assess heterogeneity. Egger’s regression asymmetry, the excess of statistical significance bias test and the proportion of participants in studies at high risk of reporting bias were used to examine the presence of small study effects42,43. When meta-analyses contained dependent effect sizes, we removed this dependence using the standard aggregating approach proposed by Borenstein and colleagues44 (Supplementary Information section 8). Because we chose to include both randomized and non-randomized studies, we conducted a sensitivity analysis restricted to randomized controlled trials. However, less than 10% of studies were non-randomized studies in our primary analysis, and the results of this analysis were very similar to those reported in the main manuscript.
Main deviations from the protocol
We made two important changes to our preregistered protocol. Although we initially planned to select the largest meta-analyses in cases of overlap, we ultimately prioritized the meta-analysis with the highest methodological quality—in line with our willingness to prioritize high-quality evidence for clinical decision-making. In addition, because the methodological landscape of umbrella reviews was still evolving at the time of registration and there were no gold-standard methods for assessing the quality of a very large body of evidence coming from umbrella reviews of RCTs, we did not prespecify an evidence grading system in our protocol. Instead, we relied on recent algorithmic GRADE criteria that have been proposed independently of the current work and implemented in the main umbrella review software34. Although these departures were made to increase methodological rigour, they represent post-hoc decisions that should be taken into account when interpreting our results and reflect the recent nature of the field of umbrella reviews.
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.
Source link
