Graphical summary of a meta-analysis of over 1,000 cases of pediatric glioma (the most common brain tumour), which identified the genetic mutations involved (highlighted by colour) as well as their importance and impact. Sourced from: https://www.cell.com
These days, anyone seeking a clearer understanding of a problem risks to be loaded with an avalanche of information. But how is one supposed to deal with it when inquiring not out of pure curiosity but in order to tackle a matter that directly influences human lives? It has been reported by Pubmed alone, one of the major biomedical databanks, that 1.3 million research papers in the related area are published worldwide every year. Even assuming that only 1% of them are concerned with the work of a certain doctor or a researcher, it would be necessary to read 30 papers every day, including weekend. Therefore, here has been an upturn in demand for methods of synthesizing and summarizing information, which, in particular, resulted in ‘studies examining other studies’ or systematic reviews – the apex of evidence-based medicine. One type of systematic review is meta-analysis. This article will discuss what a meta-analysis is and what applications and limitations it has.
A systematic review summarizes the results of other research studies in order to identify, examine and explain possible differences. It is more objective compared to a traditional descriptive literature review, as it adheres to rigid standards and protocols1-2. When special statistical techniques are used in an analysis of pooled results, it is called a quantitative systematic review, or meta-analysis1-3. In other words, meta-analysis utilizes mathematical approaches to combining data from numerous individual studies and drawing new inferences based on the overall dataset collected. However, it is necessary to ensure that the data being pooled are similar in type1-2. Karl Pearson4, who combined several typhoid vaccination studies, first conducted Meta-analysis back in 1904. But the term “meta-analysis” was coined by Gene Glass, an American statistician, in 19764.
How is a meta-analysis conducted? Initially, the aim and end point of the study should be formulated in a most explicit way, as this will determine its further progress. Next, a systematic search is carried out to find information on the topic concerned. All studies included in analysis must strictly meet the established criteria. Namely, “bad” papers are discarded, and “good” ones are accepted, for which a rationale must be provided. Publications selected for inclusion in analysis are assessed and quantitative data are summarized. An important step is assessing the combined results for heterogeneity1-4, which is followed by the meta-analysis itself, where a summary estimate of the overall effect size is derived using specialized statistical models, after which the sensitivity of analysis is determined and all possible limitations and factors that influence the final estimate are identified3-4. Finally, the results are interpreted as follows: conclusions are made on whether or not the treatment or drug produces an effect or if surgery has an advantage over drug therapy, etc.
Today, meta-analysis is the pinnacle of evidence-based medicine. It offers a greater statistical power (and therefore accuracy of estimation) compared to individual studies, due to a larger number of observations collected across multiple studies. Meta-analysis is also more reliable as it has more rigorous methodology1-3. As a tool to summarize and integrate large collections of evidence, meta-analysis is helpful for doctors, to stay afloat the sea of information, which is expanding each day; for scientists – to plan and design further studies; and even for health care organizers – to develop guidelines and legislation2. Meta-analysis debunked the myth that regular use of vitamin C helps prevent and treat common colds. Before that, owing to the Nobel Prize winner Linus Pauling, vitamin C had long been thought to be able to prevent or mitigate a common cold5.
Systematic reviews and meta-analysis are now part of routine clinical practice internationally, being increasingly applied in the non-clinical studies. However, a major obstacle here is that non-clinical studies are far more heterogeneous than standardized clinical trials, and therefore it is rarely possible to combine them correctly when, for example, it is necessary to derive an estimate of size of effect. Nonetheless, meta-analyses that have been used in non-clinical studies have pinpointed a number of problems typical of this type of research. The main one is that it lacks strict methodological procedures for experiments and presentation of results. This leads to an overestimated effect size and subsequent poor data reproducibility in the clinical setting4.
The Cochrane Collaboration is the largest and most respected international association involved in producing systematic reviews and meta-analyses of clinical trials2. The most up-to-date results of meta-analyses stored in the association’s databank guarantee high quality information on the effectiveness of medical interventions. Compelling evidence to support the critical significance of meta-analysis in medicine was provided by a study on the role of corticosteroid therapy in pregnant women at risk of premature labour. The analysis clearly showed that short-term treatment with corticosteroids could save the life of a newborn. After the results were made known to the public, corticosteroid therapy became widespread despite being formerly considered ineffective6. Therefore, it was due to meta-analysis that this simple treatment option became available and saved the lives of probably thousands of premature infants. The schematic illustration of the results of that meta-analysis was adopted as the logo for Cochrane.
Still, one should not think that meta-analysis is a panacea or a ‘sheet anchor’. Like any method, it has its failings and weak spots. Similarly to other statistical techniques, it is also vulnerable to errors, often yielding misleading results2. This has been noted by meta-analysts themselves, who are normally very critical of analyses they produce7. Although meta-analysis at first glance offers a mathematically rigorous approach, it is not distortion-free. For instance, automating the search for source material – perhaps the cornerstone of the entire process – is not yet practicable, which means it completely relies on analysts2,7. That matters when considering that the conclusion of meta-analysis is directly dependent on the studies the researcher selects to include in it. In doing so, one may choose (using a proper ‘scientific’ justification, of course) data largely according to their own liking and thereby produce an anticipated result7. One of the drawbacks of meta-analysis is that it is generally retrospective, dealing with known data. Using different types of sources, algorithms and interpretations can lead to conflicting inferences7.
An illustrative example could be a meta-analysis of corticosteroids used in acute bacterial meningitis. In 2009, the analysis demonstrated a clear benefit of steroids, which were prescribed for both adults and children with the above-mentioned condition. However, a more comprehensive analysis of 2010 showed that steroids are of no use in treating meningitis7. The reasons for such conflicting conclusions may be numerous. Meta-analysis is prone to intrinsic errors as well as random and systematic (bias) errors pertaining to the studies it comprises2,7. There have been efforts to counter this, but not very successful so far. Sometimes bias is introduced in the summary estimate because ‘positive’ results are more readily submitted for publication (i.e., publication bias – negative data are not so willingly published)2,3,7 and because of selective reporting of results or selective data analysis7. This is prompting critics to describe meta-analysis as ‘garbage in, garbage out’7.
Apart from biased estimates, meta-analysis is often complicated by combining heterogeneous studies that may initially have different objectives, eligibility criteria, methodology and procedure. In this case, critics ironically note that meta-analysis mixes “apples and oranges and an occasional lemon” 2,7. That way, the heterogeneity of data included in meta-analysis resulted in calcium preparations being recognized effective for reducing the risk of preeclampsia (high blood pressure) in pregnant women8. Yet a subsequent large-scale clinical trial demonstrated a lack of effect9. Scientists and analysts have been doing their utmost to improve homogeneity of studies and at least ensure comparisons within “citrus variants”, which is difficult to achieve, though.
No doubt, the use of systematic reviews and meta-analyses is a valuable decision-making tool for not only individual doctors or investigators, but also health care providers in general. No other tool that could provide better reliability for estimating the size of effect is available to date. Besides, the cost of conducting a meta-analysis is low compared to arranging a new large-scale clinical trial. Nonetheless, one should not overestimate the potentials of this approach. According to analysts themselves, it is a good means of integrating and structuring information7 rather than a source of unfailingly objective evaluations or a rigorous statistical discipline that produces unambiguous and simple answers to questions concerning complicated clinical problems9.
2. Lukina YV, Martsevich SY, Kutishenko NP. Systematic review and meta-analysis: the pitfalls of the methods. Ratsionalnaya farmakoterapia v kardiologii [Ration Pharmacother Cardiol]. 2016. 12(2):180-185.
3. Kalinin AL, Litvin LL, Trizna NM Use of data of evidence-based medicine in clinical practice. Problemy zdorovia i ekologii [Iss Health Ecol]. 2008. 3(17): 27-32.
4. Sena ES, Currie GL, McCann SK, Macleod MR, Howells DW. Systematic reviews and meta-analysis of preclinical studies: why perform them and how to appraise them critically. J Cereb Blood Flow Metab. 2014. 34(5): 737-42.
6. Douglas RM, Hemilä H, Chalker E, Treacy B. Vitamin C for preventing and treating the common cold. Cochrane Database Syst Rev. 2007 Jul 18;(3): CD000980.
8. Ioannidis JP. Meta-research: The art of getting it wrong. Res Synth Methods. 2010. 1(3-4): 169-84.
10. Hofmeyr GJ, Betrán AP, Singata-Madliki M, Cormick G, Munjanja SP, Fawcus S, Mose S, Hall D, Ciganda A, Seuc AH, Lawrie TA, Bergel E, Roberts JM, von Dadelszen P, Belizán JM. Prepregnancy and early pregnancy calcium supplementation among women at high risk of pre-eclampsia: a multicentre, double-blind, randomised, placebo-controlled trial. 2019. The Lancet. 393 (10169): 330-339.
11. Thompson SG, Pocock SJ. Can meta-analyses be trusted? 1991. The Lancet. 338 (8775): 1127-1130.