Integration of Orthopaedic Journals and the Internet

The orthopaedic journals are the most experienced institutions when it comes to high quality information. Most of them have now moved to post a full text version of their "papers" on the Internet but try to support their subscription base by requiring a high price for access to this information. Since the material is posted anyway it would make financial sense to charge a low price and see higher readership. As this changes, the nature of publication is likely to change as Internet-specific features bear fruit. These features include faster turn-around time, integration of feedback into the work and the ease of updating. Those interested in informatics will be trying to hasten and smooth this transition.

Table 1 Comparison of Journal Articles and Internet Pages

The peer review system used by the journals is often cited as a key advantage ¹ and by contrast the Internet is condemned because there "is no peer-review" ² . This comment ignores the fact that the majority of medical teaching is not peer reviewed. Bedside teaching, operative technique, rounds and teaching seminars are seldom subjected to rigorous review, nor are most presentations at workshops and teaching courses. Yet the majority of orthopaedic CME occurs in this situation. Peer review has its critics. There are relatively few studies on the inter-rater reliability of peer review in orthopaedics. ^{3 4} Further, there is considerable controversy over journal bias ^{5 6 7} and the fairness and appropriateness of blind peer-review ^{8 9 10} as currently practiced by the journals ^{11 12} . Fister's summary (2005) ¹³ was "We now have plenty of evidence to support the contention that peer review is "expensive, slow, subjective and biased, open to abuse, patchy at detecting important methodological defects, and almost useless at detecting fraud or misconduct." ¹⁴ " Overall the Cochrane Review of the subject ¹⁵ concluded "little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research. However, the methodological problems in studying peer review are many and complex. At present, the absence of evidence on efficacy and effectiveness cannot be interpreted as evidence of their absence." Since peer review will be an important part of orthopaedic informatics for the foreseeable future it is important for reviewers to learn how to do it effectively ^{16 17} .
Peer review on the Internet has the potential for being more immediate, open and productive because alteration and updating of electronic material in response to critique is faster and easier. The "wiki" model ¹⁸ in which the material presented may be amended by multiple users is also very attractive ¹⁹ , provided there is supervision by a competent scientific editor. Orthopaedia is an orthopaedic 'wiki' with participation limited to those with an orthopaedic qualification. The assumption inherent in open authorship is that different points of view will be aired and synthesized with overall improvement in the way in which the material is presented. With all these pressures it is safe to predict that "publication" will not remain unchanged. ^{20 21 22}
There are now many Open Access Journals ²³ in the orthopaedic field. These journals "publish" on the Internet without charging for access to the material ²⁴ . Authors (or their institutions) are responsible for a fee to cover the costs of the website. By and large these journals offer a traditional approach to peer review and do not take full advantage of the electronic medium and hypertext. These advantages include linkage to other works, illustrations and videos on the Internet and exploiting the full potential of feedback from the readership. Few journals allow links to be incorporated in the text, instead requiring the traditional "endnotes". Works of orthopaedic scholarship rarely provoke responses apart from decorous letters to the editor. This leads to a false sense of completion when a work is published. It would be most stimulating to consider a new work as a challenge for others to take up or round out. This current paper, for example, has made an effort to collect the important examples of published orthopaedic informatics. There are bound to be gaps and deficiencies; it would be more efficient to modify the paper to correct these deficiencies as they come to light than to publish a series of addenda, (conflicting) corrections, or to prepare a whole new paper. This notion of "Collaborative Scholarship" presents, of course, a whole new set of problems - What is authorship? Who decides to accept  additions? How are the academic rewards of an on-going scholarly "thread" to be apportioned?

Footnotes

Ref	Notes
1	Spine JN, 2000. "Peer review: quality improvement at its best." Spine (Phila Pa 1976) 25 (18): 2277-9 [PubMed]
2	Leach RE, 2000. "The misinformation boom." Iowa Orthop J 20: 96-100 [PubMed]
3	Bhandari M, Templeman D, Tornetta P, 2004. "Interrater reliability in grading abstracts for the orthopaedic trauma association." Clin Orthop Relat Res (423): 217-21 [PubMed] Abstract: Only a small proportion of submitted abstracts to the annual meeting of the Orthopaedic Trauma Association can be accepted for podium presentation. Annual program committee members must ensure that the selection of abstracts is free from bias and transparent to investigators. The objectives of this study are to examine the consistency of reviewers in grading abstracts submitted for podium presentations at the 2001 and 2002 Annual Meetings of the Orthopaedic Trauma Association and to evaluate whether the grades of the actual podium presentations at the meeting are consistent with the grades based on abstracts only. Reviewers independently graded all abstracts submitted to the Orthopaedic Trauma Association for presentation in a blinded manner. Abstracts submitted by members of the review panel were independently adjudicated by six reviewers who were not members of the committee. Before final decision-making, all reviewers met to discuss the abstracts submitted for oral presentation. Among the 440 papers reviewed in 2001 and 438 papers reviewed in 2002, the interreviewer reliability for abstract review was 0.23 and 0.27, respectively. Despite disagreements in the quality of the abstracts, reviewers achieved consensus by discussions to determine the final program. Agreement among unblinded reviewers of the 67 and 73 podium presentations during the 2001 and 2002 meetings, respectively, did not improve interreviewer agreement. Of the papers of the 2002 meeting that ultimately ranked in the top 20 after the full presentation of the papers, 15 papers originally had been ranked less than 20 in the initial grading. Only one of the top three papers of the meeting originally was ranked in the top three before the meeting.
4	Poolman RW, Keijser LC, de Waal Malefijt MC, Blankevoort L, Farrokhyar F, Bhandari M, Dutch Orthopedic Association Scientific Committee, 2007. "Reviewer agreement in scoring 419 abstracts for scientific orthopedics meetings." Acta Orthop 78 (2): 278-84 [PubMed] Abstract: BACKGROUND: The selection of presentations at orthopedic meetings is an important process. If the peer reviewers do not consistently agree on the quality score, the review process is arbitrary and open to bias. The aim of this study was: (1) to describe the inter-reviewer agreement of a previously designed scoring scheme to rate abstracts submitted for presentation at meetings arranged by the Dutch Orthopedic Association; (2) to test whether the quality of reporting of submitted abstracts increased in the years after the introduction of the scoring scheme; and (3) to examine whether a review process with a larger workload had lower interrater agreement. METHODS: We calculated intraclass correlation coefficients (ICC) to measure the level of agreement among reviewers using the International Society of the Knee (ISK) quality-of-reporting system for abstracts. Acceptance rate and quality of the abstracts are described. RESULTS: Of 419 abstracts, 229 (55%) were accepted. Inter-reviewer agreement to rate abstracts was substantial (0.68; 95% CI: 0.47-0.83) to almost perfect (0.95; 95% CI: 0.92-0.97) and did not change over the eligible time period. A smaller proportion of abstracts were accepted after 2004. The mean ISK abstract score (with a maximum of 100 points) for accepted abstracts ranged from 60 (95% CI: 58-63) to 64 (95% CI: 62-66). The mean ISK abstract score for rejected abstracts varied from 46 (95% CI: 40-51) to 51 (95% CI: 47-55). Average scores for accepted and rejected abstracts did not change with time. The degree of workload of the reviewers did not influence their level of agreement. INTERPRETATION: The ISK abstract rating system has an excellent interobserver agreement. Other scientific orthopedic meetings should consider adopting this ISK rating system for further evaluation in a local or international setting.
5	Leopold SS, Warme WJ, Fritz Braunlich E, Shott S, 2003. "Association between funding source and study outcome in orthopaedic research." Clin Orthop Relat Res (415): 293-301 [PubMed] Abstract: The current report tests the hypotheses that commercial funding, country of origin, and presence of a coinvestigator with training in statistics are related to the likelihood of a published orthopaedic study arriving at a positive conclusion. All articles from the Journal of Bone and Joint Surgery (American), the Journal of Arthroplasty, and the American Journal of Sports Medicine published in 1 year were reviewed. The blinded review process classified each article as to study design and outcome (positive or negative), according to previously published definitions. Commercial funding was significantly associated with a positive outcome; 78.9% of commercially funded studies concluded with a positive outcome, compared with 63.3% of nonfunded studies. The presence of a statistician or epidemiologist as a coinvestigator, and the place of origin of the study were not associated with outcome. Only 21% of published studies were prospective, 3.5% were randomized, and 10.5% stated an experimental hypothesis; these factors were not associated with study outcome. Published studies that received funding from commercial parties were significantly more likely to report a positive outcome than studies that received no such funding. This does not imply the presence of a corrupting or causative influence of industry sponsorship on research outcomes; additional research is needed to determine whether such nonscientific factors actually affect study outcome or likelihood of publication.
6	Yousefi-Nooraie R, Shakiba B, Mortaz-Hejri S, 2006. "Country development and manuscript selection bias: a review of published studies." BMC Med Res Methodol 6: 37 [PubMed] Abstract: BACKGROUND: Manuscript selection bias is the selective publication of manuscripts based on study characteristics other than quality indicators. One reason may be a perceived editorial bias against the researches from less-developed world. We aimed to compare the methodological quality and statistical appeal of trials from countries with different development status and to determine their association with the journal impact factors and language of publication. METHODS: Selection criteria: Based on the World Bank income criteria countries were divided into four groups. All records of clinical trials conducted in each income group during 1993 and 2003 were included if they contained abstract and study sample size. Data sources: Cochrane Controlled Trials Register was searched and 50 articles selected from each income group using a systematic random sampling method in years 1993 and 2003 separately. Data extraction: Data were extracted by two reviewers on the language of publication, use of randomization, blinding, intention to treat analysis, study sample size and statistical significance. Disagreement was dealt with by consensus. Journal impact factors were obtained from the institute for scientific information. RESULTS: Four hundred records were explored. Country income had an inverse linear association with the presence of randomization (chi2 for trend = 5.6, p = 0.02) and a direct association with the use of blinding (chi2 for trend = 6.9, p = 0.008); although in low income countries the probability of blinding was increased from 36% in 1993 to 46% in 2003. In 1993 the results of 68% of high income trials and 64.7% of other groups were statistically significant; but in 2003 they were 66% and 82% respectively. Study sample size and income were the only significant predictors of journal impact factor. CONCLUSION: The impact of country development on manuscript selection bias is considerable and may be increasing over time. It seems that one reason may be more stringent implementation of the guidelines for improving the reporting quality of trials on developing world researchers. Another reason may be the presumptions of the researchers from developing world about the editorial bias against their nationality.
7	Lynch JR, Cunningham MR, Warme WJ, Schaad DC, Wolf FM, Leopold SS, 2007. "Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not." J Bone Joint Surg Am 89 (5): 1010-8 [PubMed] Abstract: BACKGROUND: Prior studies implying associations between receipt of commercial funding and positive (significant and/or pro-industry) research outcomes have analyzed only published papers, which is an insufficiently robust approach for assessing publication bias. In this study, we tested the following hypotheses regarding orthopaedic manuscripts submitted for review: (1) nonscientific variables, including receipt of commercial funding, affect the likelihood that a peer-reviewed submission will conclude with a report of a positive study outcome, and (2) positive outcomes and other, nonscientific variables are associated with acceptance for publication. METHODS: All manuscripts about hip or knee arthroplasty that were submitted to The Journal of Bone and Joint Surgery, American Volume, over seventeen months were evaluated to determine the study design, quality, and outcome. Analyses were carried out to identify associations between scientific factors (sample size, study quality, and level of evidence) and study outcome as well as between non-scientific factors (funding source and country of origin) and study outcome. Analyses were also performed to determine whether outcome, scientific factors, or nonscientific variables were associated with acceptance for publication. RESULTS: Two hundred and nine manuscripts were reviewed. Commercial funding was not found to be associated with a positive study outcome (p = 0.668). Studies with a positive outcome were no more likely to be published than were those with a negative outcome (p = 0.410). Studies with a negative outcome were of higher quality (p = 0.003) and included larger sample sizes (p = 0.05). Commercially funded (p = 0.027) and United States-based (p = 0.020) studies were more likely to be published, even though those studies were not associated with higher quality, larger sample sizes, or lower levels of evidence (p = 0.24 to 0.79). CONCLUSIONS: Commercially funded studies submitted for review were not more likely to conclude with a positive outcome than were nonfunded studies, and studies with a positive outcome were no more likely to be published than were studies with a negative outcome. These findings contradict those of most previous analyses of published (rather than submitted) research. Commercial funding and the country of origin predict publication following peer review beyond what would be expected on the basis of study quality. Studies with a negative outcome, although seemingly superior in quality, fared no better than studies with a positive outcome in the peer-review process; this may result in inflation of apparent treatment effects when the published literature is subjected to meta-analysis.
8	Medscape General Medicine.  2005;7(1):11. Standardization vs Diversity: How Can We Push Peer Review Research Forward? Karen Shashok Available at http://www.medscape.com/viewarticle/498238
9	Goldbeck-Wood S, 1999. "Evidence on peer review-scientific quality control or smokescreen?" BMJ 318 (7175): 44-5 [PubMed]
10	Thörn A, 2002. "[Peer review: a closed system in need of reform]" Lakartidningen 99 (30-31): 3106-8 [PubMed] Abstract: Peer review is the process by which experts assess the quality of manuscripts submitted for publication in scientific journals. During the last decade, this system for quality control has been debated a lot and investigated scientifically. By now, there is a widespread consciousness among scientists regarding the deficiencies of this system for quality control. It is slow, expensive, subjective, prone to bias and easily abused. To comply with its important function, the peer review system has to be reformed. The most important reforms would be to make it an open process--that is, the removal of the anonymity of the reviewing experts, and to oblige all parties in the publication process, authors, editors, reviewers, to declare potential conflicts of interest.
11	Ann C. Weller, BA,MA Editorial Peer Review: its Strengths and Weaknesses Information Today, Inc. Copyright 2001 (ASIST Monograph Series) 342 pages ISBN: 1-57387-100-1
12	Jefferson T, Alderson P, Wager E, Davidoff F, 2002. "Effects of editorial peer review: a systematic review." JAMA 287 (21): 2784-6 [PubMed] Abstract: CONTEXT: Editorial peer review is widely used to select submissions to journals for publication and is presumed to improve their usefulness. Sufficient research on peer review has been published to consider a synthesis of its effects. METHODS: To examine the evidence of the effects of editorial peer-review processes in biomedical journals, we conducted electronic and full-text searches of private and public databases to June 2000 and corresponded with the World Association of Medical Editors, European Association of Science Editors, Council of Science Editors, and researchers in the field to locate comparative studies assessing the effects of any stage of the peer-review process that made some attempt to control for confounding. Nineteen of 135 identified studies fulfilled our criteria. Because of the diversity of study questions, methods, and outcomes, we did not pool results. RESULTS: Nine studies considered the effects of concealing reviewer/author identity. Four studies suggested that concealing reviewer or author identity affected review quality (mostly positively); however, methodological limitations make their findings ambiguous, and other studies' results were either negative or inconclusive. One study suggested that a statistical checklist can improve report quality, but another failed to find an effect of publishing another checklist. One study found no evidence that training referees improves performance and another showed increased interrater reliability; both used open designs, making interpretation difficult. Two studies of how journals communicate with reviewers did not demonstrate any effect on review quality. One study failed to show reviewer bias, but the findings may not be generalizable. One nonrandomized study compared the quality of articles published in peer-reviewed vs other journals. Two studies showed that editorial processes make articles more readable and improve the quality of reporting, but the findings may have limited generalizability to other journals. CONCLUSIONS: Editorial peer review, although widely used, is largely untested and its effects are uncertain.
13	Fister K, 2005. "At the frontier of biomedical publication: Chicago 2005." BMJ 331 (7520): 838-40 [PubMed]
14	Godlee F, Jefferson T. Introduction. In: Godlee F, Jefferson T, eds. Peer review in health sciences. London: BMJ Books, 1999: xi-v.
15	Jefferson T, Rudin M, Brodney Folse S, Davidoff F, 2007. "Editorial peer review for improving the quality of reports of biomedical studies." Cochrane Database Syst Rev (2): MR000016 [PubMed] Abstract: BACKGROUND: Scientific findings must withstand critical review if they are to be accepted as valid, and editorial peer review (critique, effort to disprove) is an essential element of the scientific process. We review the evidence of the editorial peer-review process of original research studies submitted for paper or electronic publication in biomedical journals. OBJECTIVES: To estimate the effect of processes in editorial peer review. SEARCH STRATEGY: The following databases were searched to June 2004: CINAHL, Ovid, Cochrane Methodology Register, Dissertation abstracts, EMBASE, Evidence Based Medicine Reviews: ACP Journal Club, MEDLINE, PsycINFO, PubMed. SELECTION CRITERIA: We included prospective or retrospective comparative studies with two or more comparison groups, generated by random or other appropriate methods, and reporting original research, regardless of publication status. We hoped to find studies identifying good submissions on the basis of: importance of the topic dealt with, relevance of the topic to the journal, usefulness of the topic, soundness of methods, soundness of ethics, completeness and accuracy of reporting. DATA COLLECTION AND ANALYSIS: Because of the diversity of study questions, viewpoints, methods, and outcomes, we carried out a descriptive review of included studies grouping them by broad study question. MAIN RESULTS: We included 28 studies. We found no clear-cut evidence of effect of the well-researched practice of reviewer and/or author concealment on the outcome of the quality assessment process (9 studies). Checklists and other standardisation media have some evidence to support their use (2 studies). There is no evidence that referees' training has any effect on the quality of the outcome (1 study). Different methods of communicating with reviewers and means of dissemination do not appear to have an effect on quality (3 studies). On the basis of one study, little can be said about the ability of the peer-review process to detect bias against unconventional drugs. Validity of peer review was tested by only one small study in a specialist area. Editorial peer review appears to make papers more readable and improve the general quality of reporting (2 studies), but the evidence for this has very limited generalisability. AUTHORS' CONCLUSIONS: At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research. However, the methodological problems in studying peer review are many and complex. At present, the absence of evidence on efficacy and effectiveness cannot be interpreted as evidence of their absence. A large, well-funded programme of research on the effects of editorial peer review should be urgently launched.
16	Levine AM, Heckman JD, Hensinger RN, 2004. "The art and science of reviewing manuscripts for orthopaedic journals: Part I. Defining the review." Instr Course Lect 53: 679-88 [PubMed] Abstract: The peer review process is integral to the functioning of all scientific journals and plays a pivotal role in the publication of new scientific material. Timeliness, freedom from bias, and proper review etiquette are essential for an effective review. Each type of scientific article demands a slightly different reviewing technique. Clinical research articles require that a proper question be asked and that the study methodology allows the question to be answered effectively. All studies dealing with human subjects need to be approved by the Institutional Review Board. Clinical review articles have specific criteria to determine how they should be constructed and when they should be published, whereas basic science research articles should be evaluated for their experimental method as well as the relevance of the conclusions to the data. Finally, case reports have an important place in the scientific literature, but the rationale for publishing any individual case needs to be clearly established.
17	Levine AM, Heckman JD, Hensinger RN, 2004. "The art and science of reviewing manuscripts for orthopaedic journals: Part II. Optimizing the manuscript: practical hints for improving the quality of reviews." Instr Course Lect 53: 689-97 [PubMed] Abstract: Manuscripts submitted to musculoskeletal journals have several key components that need to be critically evaluated. There are specific methods to assess the abstract, illustrations, references, and other major elements of a manuscript under review. If each of these elements is assessed methodically, not only does the quality of the review improve, but it becomes more useful for the journal editor. Additionally, the method in which the review is conveyed has a marked impact on its usefulness. There should be a concise evaluation of the entire work, stating whether a publication should or should not be pursued. For poor manuscripts, several bulleted points that indicate the fatal flaw(s) are sufficient, but for good manuscripts, a systematic itemization of weaknesses will improve the quality of the manuscript. Reviews should not be derogatory and should be prompt and to the point.
18	Main Page - Wikipedia, the free encyclopedia (2007) Website Available at http://en.wikipedia.org/wiki/Main_Page
19	Frishauf P, 2006. "Are traditional peer-reviewed medical articles obsolete?" MedGenMed 8 (1): 5 [PubMed]
20	BMJ. 2002 Dec 21;325(7378):1478-81. Papyrus to PowerPoint (P 2 P): metamorphosis of scientific communication. LaPorte RE, Linkov F, Villasenor T, Sauer F, Gamboa C, Lovalekar M, Shubnikov E, Sekikawa A, Sa ER. Available at http://www.bmj.com/cgi/content/full/325/7378/1478
21	LaPorte RE, Marler E, Akazawa S, Sauer F, Gamboa C, Shenton C, Glosser C, Villasenor A, Maclure M, 1995. "The death of biomedical journals." BMJ 310 (6991): 1387-90 [PubMed]
22	Medscape General Medicine 4(4), 2002. Medscape General Medicine: The Next Steps in an Ongoing Experiment in Medical Publishing George D. Lundberg, MD; Bill Silberg; Christina Myers; Sara Mariani, MD, PhD; Mindy Hung, MA; Steve Zatz, MD; Roger Holstein Available at http://www.medscape.com/viewarticle/443817
23	Rankin JA, Franklin SG, 2004. "Open access publishing." Emerg Infect Dis 10 (7): 1352 [PubMed]
24	Eura Medicophys 2007; 43: 203-213 Open Access in the Biomedical field: a unique opportunity for researchers (and research itself) Giglia E Available at http://www.minervamedica.it/index2.t?show=R33Y2007N02A0203