RANDOMIZED CONTROLLED TRIALS IN ORTHOPEDICS AND TRAUMATOLOGY: SYSTEMATIC ANALYSIS ON THE NATIONAL EVIDENCE

1 - Third-year Resident in the Department of Orthopedics and Traumatology, Federal University of São Paulo (Unifesp).
2 - Physician in the Shoulder and Elbow Sector of the Discipline of the Hand and Upper Limb, Department of Orthopedics and Traumatology, Federal University of São Paulo (Unifesp).
3 - Adjunct Professor in the Department of Orthopedics and Traumatology, Federal University of São Paulo (Unifesp).
4 - Titular Professor of the Department of Orthopedics and Traumatology, Federal University of São Paulo (Unifesp).
Work performed in the Paulista School of Medicine, Federal University of São Paulo (Unifesp).
Correspondence: Rua Borges Lagoa 593, 5° andar, São Paulo, SP. E-mail: vym70@yahoo.com.br
Work received for publication: March 2, 2010; accepted for publication: March 9, 2010.

Critical assessment of the orthopedic literature has become a necessary tool for those who seek up-to-date knowledge(1-3). Within this context, acquisition of information in a systematic, evidence-based manner, and consequent categorization into levels of evidence(4) becomes mandatory, in view of the great number of published papers and the frequent methodological traps that may lead readers to faulty conclusions that are potentially dangerous for medical practice(5-8).

Efforts towards defining adequate and uniform methodological criteria for publishing good-quality studies have led researchers to rethink the planning and publication of research, so that greater scientific rigor and greater external validity are achieved(9-11). Among these efforts, with regard to treatment, randomized controlled trials (RCTs) are the type of study that presents the best level of evidence(12,13), given that their methodological design makes it possible to reach comparative randomized conclusions regarding the best treatment option for each clinical question, using validated tools for measuring the significant outcomes from the clinical condition under examination. Today, RCTs attract greater prestige and attention within the literature.

Despite the attribution of levels of evidence, assessments on studies in an individualized manner cannot be neglected, given that inconsistencies in such attributions occur frequently, as made clear by some authors(5,14). A previous study demonstrated that there was equivalence between the quality of Brazilian national and foreign production between 1988 and 2002(15). Since then, it was recently demonstrated regarding the American literature that there had been an improvement in the quality of published papers, despite a considerable number of potentially correctable methodological faults(9,16).

It would be hoped that, in the Brazilian literature, quantitative and qualitative improvements in the level of evidence of published studies would be seen, in line with the worldwide trend(17).

The aims of the present study were:

1. To investigate whether there was any increase in the numbers of RCTs published, comparing the periods 2000-2004 and 2004-2009 (quantitative evaluation);
2. To evaluate these studies in accordance with their methodological quality (Jadad score and Cochrane Collaboration score) (qualitative evaluation).

MATERIALS AND METHOD

The investigation was conducted by two independent researchers (V.Y.M, C.D.M), who used an electronic search to select all the editions, and perform manual extraction of all the studies, that were described as RCTs in the two Brazilian indexed journals that focus mainly on publishing research relating to orthopedics and traumatology: Acta Ortopédica Brasileira (AOB) and Revista Brasileira de Ortopedia (RBO).

We defined RCTs as follows: I. Trials that have a design that is planned before data gathering; II. Trials involving treatment; and III. Trials in which a given patient can be included in any of the allocation groups with the same chance(13).

The two investigators independently evaluated all the titles and structured abstracts of these journals and set aside for detailed evaluation the studies that were presented as RCTs that, a priori, were published between 2000 and 2009. Studies that presented imprecise or inconclusive abstracts were also set aside and doubts regarding them were resolved by reading the full text and/or contacting the principal investigator of the study. These were included or excluded from the study as soon as the missing information was obtained and/or after reaching a consensus between the researchers.

The studies included were scored in accordance with the score of Jadad et al(18) and the score of the Cochrane Bone, Joint and Muscle Trauma Group(19). After reading the studies in full, the scores were considered separately (S1 and S2), and as the mean between them (S1/ S2), in order to analyze the groups. These groups were defined as follows: Group I: studies published before 2004; Group II: studies published between 2004 and 2009. Together, an evaluation according to the journal of publication was performed (AOB and RBO), for the same scores defined above.

This study was approved by the research ethics committee of our institution (no. 0016/10).

Inclusion criteria

RCTs published in indexed Brazilian journals focusing on orthopedics and traumatology (AOB and RBO) between 2000 and 2009 were included.

Exclusion criteria

Studies that resulted in methodological doubts that remained even after a consensus meeting between the investigators, and for which no contact with the principal investigator through traditional means could be achieved, were excluded.

Statistical analysis

The premise of normality of distribution of the sample was assessed using the Kolmogorov-Smirnov test. The Mann-Whitney test was used for comparisons between the means for the scores in the different groups. To evaluate the reliability, with the aim of assessing the internal consistency of the inter-observer Jadad scores, the Cohen kappa test was used, with paired samples(20). The method proposed by Landis and Koch(21) for interpreting the degree of concordance was used: I. < 0 - poor; II. 0 to 0.20 - slight; III. 0.21 to 0.40 - fair; IV. 0.41 to 0.60 - moderate; V. 0.61 to 0.80 - excellent,substantial; VI. 0.81 to 1.00 - almost perfect. The chisquare test was used for proportional evaluation between the different periods (proportion of RCTs in period 1 and proportion of RCTs in period 2).

RESULTS

Twenty-two studies were included, of which 10 came from AOB(22-31) and 12 from RBO(32-43) (Table 1). Evaluation of the inter-observer reliability (S1 and S2) for the Jadad score resulted in a kappa value of 0.611 (substantial, excellent).

Comparison of the methodological quality between the studies, according to the Jadad and Cochrane collaboration scores, did not demonstrate any statistical differences when the studies were grouped according to publication period (Tables 2 and 3). When grouped according to publication period, there were no statistically significant differences (Tables 4 and 5), except for examiner 2, for the Jadad score, in which the methodological quality in the period from 2004 to 2009 was favored (Mann-Whitney test; p = 0.02). The ratio between all the studies published and the RCTs did not show any difference between the two journals (p = 0.867).

DISCUSSION

The results from this study demonstrate a trend towards improvement in the methodological quality of the RCTs published within our setting, even though neither of the investigators found statistical significance. This last observation goes against the efforts coming from foreign and national initiatives(1,2,4,10,44,45). Another factor to which attention is drawn is the equivalence of the methodological quality of the two national journals, at least with regard to this single analysis.

The absolute number of RCTs (close to 2%) demonstrates two difficulties in real situations: the first is the difficulty of carrying out RCTs with blinding in specialties of a surgical nature; and the second is the characteristic of placing value on case series, which reflects a list of patients who are treated uniformly without methodological criteria. In a certain way, this is congruent with the worldwide literature, which presents similar percentages of RCTs, with improved methodological rigor now demonstrated(46,47). Despite the merit of case series, therapeutics cannot be guided by studies of level III or IV, at least for diseases that are known to occur with high frequency. It is within this panorama that protocols with adequate methodological quality involving several research centers can be created. This suggests that, in this respect, orthopedics and traumatology societies have important activities(48).

Some bias should be taken into consideration in this investigation: the difficulty involved in using the evaluation scales for RCTs (Jadad and Cochrane) and, because of the low frequency of trials encountered, there is the possibility that there may have been a beta error. The first of these is due to the inherent difficulties of classification systems, and the second is due to aspects of Brazilian realities.

CONCLUSION

The relative quality and quantity of RCTs in the journals analyzed was similar.

There was a trend towards improved quality, but there was no increase in the quantity of RCTs over the two periods analyzed.

Future studies on clinical treatment questions should focus on stimulating the elaboration of research protocols with methodological refinement (RCTs), with the aim of furnishing the best level of evidence and establishing inter-institution cooperation.-

1. Reis FB, Lopes AD, Faloppa F, Ciconelli RM. A importância da qualidade dos estudos para a busca da melhor evidência. Rev Bras Ortop. 2008; 43(6):209- 16.
2. Schunemann HJ, Bone L. Evidence-based orthopaedics: a primer. Clin Orthop Relat Res. 2003;(413):117-32.
3. Belangero WD. Reflexões sobre a metodologia na pesquisa em ortopedia e traumatologia. Acta Ortop Bras. 2001;9(3):59-61.
4. Camanho GL. Nível de evidência [Editorial]. Rev Bras Ortop. 2009;44(6):1-2.
5. Poolman R, Struijs P, Krips R, Sierevelt I, Lutz K, Bhandari M. Does a "Level I Evidence" rating imply high quality of reporting in orthopaedic randomised controlled trials? BMC Med Res Methodol. 2006;6:44.
6. Poolman RW, Kerkhoffs GM, Struijs PA, Bhandari M, International Evidence- Based Orthopedic Surgery Working Group. Don't be misled by the orthopedic literature: tips for critical appraisal. Acta Orthop. 2007;78(2):162-71.
7. Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM, Schünemann HJ, et al. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol. 2004;57(12):1232-6.
8. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273(5):408-12.
9. Bhandari M, Guyatt G, Lochner H, Sprague S, Tornetta P. Application of the Consolidated Standards of Reporting Trials (CONSORT) in the Fracture Care Literature. J Bone Joint Surg Am. 2002;84(3):485-9.
10. Moher D, Jones A, Lepage L, CONSORT Group (Consolitdated Standards for Reporting of Trials). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA. 2001;285(15):1992-5.
11. Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Ann Intern Med 2007;147(8):W163- 94.
12. Cochrane Handbook for Systematic Reviews of Interventions 4.2.4 [updated March 2005]. In: Higgins JPT, Green S, eds. The Cochrane Library. Chichester, UK: John Wiley & Sons, Ltd.; 2005.
13. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet. 2006;352(9128):609-13.
14. Dulai SK, Slobogean BLT, Beauchamp RD, Mulpuri K. A quality assessment of randomized clinical trials in pediatric orthopaedics. J Pediatr Orthop. 2007;27(5):573-81.
15. Amatuzzi ML. Análise da evolução qualitativa de publicações em Ortopedia e Traumatologia comparação entre a "Revista Brasileira de Ortopedia" e "The Journal of Bone and Joint Surgery". Rev Bras Ortop;39(9):527-35.
16. Dijkman BG, Abouali JA, Kooistra BW, Conter HJ, Poolman RW, Kulkarni AV, et al. Twenty years of meta-analyses in orthopaedic surgery: has quality kept up with quantity? J Bone Joint Surg Am. 2010;92(1):48-57.
17. Bhandari M, Richards R, Sprague S, Schemitsch E. The quality of reporting of randomized trials in the Journal of Bone and Joint Surgery from 1988 through 2000. J Bone Joint Surg Am. 2002;84(3);388-96.
18. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17(1):1-12.
19. Bhandari M, Morrow F, Kulkarni A, Tornetta P. Cochrane Bone, Joint and Muscle Trauma Group Meta-analyses in orthopaedic surgery. A systematic review of their methodologies. J Bone Joint Surg Am. 2006;83(1):15-24.
20. Petrie A. Statistics in orthopaedic papers. J Bone Joint Surg Br 2006;88(9):1.121- 36.
21. Landis J, Koch G. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74.
22. Benegas E, Amódio DT, Correia LFM, Malavolta E, Ramadan LB, Ferreira Neto AA, et al. Estudo comparativo prospectivo e randomizado entre o tratamento cirúrgico das fraturas diafisárias do úmero com placa em ponte e haste intramedular bloqueada (análise preliminar). Acta Ortop Bras. 2007;15(2):87-92.
23. Cabral CMN, Melim AM, Sacco CN, Marques AP. Fisioterapia em pacientes com sÌndrome fÍmoro-patelar: comparação de exercÌcios em cadeia cinética aberta e fechada. Acta Ortop Bras. 2008;16(3):180-5.
24. Ferracini AM, Faloppa F, Daltro GC, Crisóstomo Júnior DC, Reis FB, Belotti JC. Estudo prospectivo e randomizado de pacientes com fraturas expostas da diáfise do fêmur submetidos a osteossíntese com placa e haste intramedular bloqueada a foco aberto. Acta Ortop Bras. 2008;16(2):102-6.
25. Garcia Filho RJ, Korukian M, Santos FPE, Viola DCM, Puertas EB. Ensaio clínico randomizado, duplo-cego, comparativo entre a associação de cafeína, carisoprodol, diclofenaco sódico e paracetamol e a ciclobenzaprina, para avaliação da eficácia e segurança no tratamento de pacientes com lombalgia e lombociatalgia agudas. Acta Ortop Bras. 2006;14(1):11-6.
26. Hamra A, Miguel OF, Marteli TK, Barci LP, Leme FM. DHS e OPS: estudo comparativo da falência de osteossÌntese. Acta Ortop Bras. 2009;17(2):35-9.
27. Nascimento TF, D'Elia LFB, Gonçalves LO, Dobashi ET. Estudo randomizado do tratamento cirurgico da sindrome do tunel do carpo. Acta Ortop Bras. 2007;15(4):218-21.
28. Pereira EA, Mattar Júnior R, Azze RJ. Comparative study between endoscopic technique by a proximal port and mini palmary incision in the surgical treatment of carpal tunnel syndrome. Acta Ortop Bras. 2003;11(Suppl 1):48-57.
29. Rejaili WA, Chueire AG, Cordeiro JA, Petean FC, Carvalho Filho GD. Avaliação do uso do Hylano GF-20 no pós-operatório de artroscopia de joelho por artrose. Acta Ortop Bras. 2005;13(1):20-3.
30. Silva AL, Imoto DM, Croci AT. Estudo comparativo entre a aplicação de crioterapia, cinesioterapia e ondas curtas no tratamento da osteoartrite de joelho. Acta Ortop Bras. 2007;15(4):204-9.