Thursday, July 31, 2008

Phil Davis: Open access publishing, article downloads, and citations: The word is still out

[See also: Free Access to Science Papers Found Not to Increase Citations URL: Accessed: 2008-08-01. (Archived by WebCite® at]

31.7.2008. Today, Davis’ et al. [1] have published a paper containing preliminary results from their Open Access RCT. While parts of the paper will be welcomed by most Open Access advocates as far as the access/usage data are concerned, showing a significant increase in access and use of Open Access articles compared to non-OA articles (though these results are far from surprising), other parts of the paper are more controversial (to be diplomatic). Davis et al. failed to show a citation advantage after 9-12 months, from which they conclude that “the citation advantage from open access reported widely in the literature may be an artifact of other causes.”. Jumping to these conclusions after only 9-12 months is actually quite outrageous and the fact that the BMJ publishes “negative” results of an ongoing trial before it is even “completed” is deeply disturbing (by the way – where is the trial registration ID and/or the link to the study protocol? What period of time was stipulated in the protocol as the primary outcome comparison point? Didn’t the BMJ commit to publishing only RCTs which have been registered, if so, where is the trial registration number?[2]). While it is legitimate to publish results of an ongoing RCT prematurely if surprisingly large, statistically significant differences between the intervention and control group emerge, it is generally not considered ethical to frame results from an ongoing RCT as negative prematurely before it makes sense to compare the groups.
In general – and on a positive note-, I fully agree that RCT’s are needed in this area to get to the bottom of the issue of citations and knowledge uptake (Disclaimer: I am the Principal Investigator of a similar, CIHR-funded project), although the practical difficulties in doing such RCT’s - where due to self-archiving it is almost impossible to generate an “uncontaminated” control group - should not be underestimated. I also congratulate Davis to having been able to convince publishers to participate in such a trial (I know from personal experience how hard this is).
However, to conclude or even imply that any citation advantage is an “artifact” after looking only at citations that occur within the same year of the cited article (9-12 months after publication) is as interesting and valid as doing a RCT on the effectiveness of a painkiller and comparing the pain between control and intervention patients after only one minute, concluding that the painkiller doesn’t work if there is no statistically significant difference between the groups after 60 seconds. It is unfortunate that Davis et al. could not wait longer to report their citation data after a reasonable period of time before implying that there are no differences – a reasonable period would be to let citations accumulate for at least 2-3 years. The way the paper stands has the potential to mislead readers who are not familiar with citation dynamics and who do not know that usually, in the first 9-12 months, papers receive very few - if any - citations at all. True – according to their data there are no statistically significant differences in citation counts after 9-12 months between Open Access and non-Open Access articles. But apparently there are also no citation differences between other groups of articles for which one would clearly expect such a difference, for example articles featured on the cover-page or articles selected for press-releases versus normal articles. Ironically, Davis himself argued elsewhere that “Coverage in the popular press is well known to amplify the transmission of scientific information to the research community. As a result, articles that receive coverage in newspapers are more likely to be cited (Phillips et al. 1991; Kiernan 2003"). Now his own analysis fails to show any effect of cover page or press release coverage on citations, as one would expect from the previous studies (see his Table 3 - and according to his supplementary file, being on the cover page even significantly reduces the odds of being cited). And ironically, according to his data (Table 3), even "self-archiving" is not an independent predictor for higher citations after 9-12 months, although - according to Davis' own argumentation (with which I agree) - these are self-selected, better quality articles! What does this say about the validity of his other conclusions? Doesn’t this hint at the fact that the observation period might be much too short? The three variables "cover page", "press-release", and "self-archiving" should be seen as internal controls. I would believe his results (in any follow-up analysis) if these variables, which are clearly associated with quality/citation differences, emerge as independent significant predictors. But as long as Davis fails to show that these variables behave in an expected way (expected is that they are predictors for citations), then I am unsurprised that OA status also does not behave in the "expected" way, and have to assume that either his data collection or analysis is flawed.
Davis et al. allowed only 9-12 months of time for the Open Access advantage to develop. For a citation event to occur, a cited paper has to be indexed, a citing author has to discover the cited paper, write his citing paper, the citing paper has to be submitted, peer-reviewed, published, and indexed by Thomson/ISI – a process that normally takes much longer than 9-12 months, which is why the citation rate is typically highest 1-2 years after publication. Published articles usually get very few, if any, citations in the early months immediately after publication, in particular in lower impact factor / immediacy index journals (many of these citations are “insider” citations from the authors themselves or others in their “inner circle” who have seen the manuscript before it was published). It is a notable omission from the manuscript that they never actually say what the crude, mean citation count in both groups actually is – reviewers and readers are deliberately left in the dark as to what I think are mean citation counts of less than on average 1 per article. How “significant” is a lack of difference at this point?
What surprised (and bothered) me especially is that Davis et al. cites my PNAS data [3] to justify his short observation period, which is ironic for a number of reasons. While I indeed found already a statistically significant difference after only 4-10 months (1.2 citations in the nOA group versus 1.5 citations in the OA group), most citations appeared after 10-16 months (4.5 versus 6.4) and 16-22 months (8.9 vs 13.1). Finding a significant difference after a short period of time certainly justifies reporting such differences. But does finding no differences in an even shorter period of time (9-12 months) justify a publication blaring out negative results implying that previous research reported artifacts? What Davis et al. fails to acknowledge is the fact that PNAS has a much higher impact factor than any of the APS journals (by the factor of 3), hence a higher citation rate, and also a high immediacy index, i.e. more citations in the same year an article has been published. This alone (leaving aside any other possible differences due to the different nature of the disciplines) is reason enough to question their comparison with PNAS and argument that “our time-frame is more than sufficient to detect a citation advantage, if one exists”. Secondly, as Davis himself notes, the PNAS study was an observational study (a prospective cohort study – not retrospective as misleadingly implied by Davis et al.), and although I statistically adjusted for known confounders (including differences in author prestige between Open Access and non-Open Access articles groups), and even tried to gather information on whether authors selectively submitted their “best” work as Open Access using an author survey (a factor Davis et al. refers to as “unobservable”), residual confounding and selection bias cannot be ruled out in any observational cohort study, including my own study (and I have never done so. In fact, I was always very careful with my conclusions and never talked about any “causal” relationship. All I said was that “Open Access status” was a strong independent predictor in my PNAS sample for citations – alongside with other factors – even when adjusted for known confounders). The OA effect observed in my study may indeed be the combined effect of selection bias/residual confounding and possibly additional increased citations due to higher access rates, but it is a surprising argument for Davis et al to justify a short observation period by citing a study he criticizes as being biased.
My final concern is the issue of contamination. The more articles in the control group are (self-archived) Open Access (i.e. “contaminate” the control group), the more difficult it will be to show a difference between the groups. My article cohort is from 2004, Davis article cohort is from 2007. Considerable headway has been made on the self-archiving front in the past couple of years, with several funding agencies mandating or strongly encouraging self-archiving, so one might assume that there are different “contamination” (i.e. self-archiving) rates in the control “non-OA” group, with higher contamination rates in the Davis study. However, Davis’ data in that respect are surprising. Davis says there were only 20 self-archived articles in his total sample, which is a suspiciously low self-archiving rate of only 1.2% (with an unreported contamination rate in his control group), while my PNAS sample had 10.6% of all articles in the control group self-archived. What Davis et al. unfortunately fail to report is when were the searches for self-archiving done? The low self-archiving rate suggests to me that this was perhaps only tested once right after publication, rather than continuously after publication? Presumably the self-archiving rate increases over time, and it is not clear to me how this was handled or controlled for.
So, what is the bottom-line of the Davis study? The access data of the RCT are certainly consistent with other postulated components of the “open access advantage” [4]. As I (and others) have argued previously, the OA advantage goes beyond citations as a crude measure for uptake within the “inner circle” of the scientific community, but includes an “end user uptake advantage” and a “cross-discipline fertilization advantage”, i.e. knowledge uptake by others than the “inner circle” of scientists working in a given discipline. Looking at Esposito’s nautilus model [5], one could argue that the open access advantage in fact happens primarily in the more “peripheral” layers of the nautilus, rather than the “innermost” circle of researchers.
But it is those individuals in the outer layers who are more likely to cite an article later than those “insiders”. And for the public or other knowledge endusers (policy makers, physicians), “citations” are an unsuitable metric anyways.

Figure: Esposito's Nautilus model [5]

In summary, I applaud the study, but I am stunned by the decision of the BMJ to allow these authors to discredit the “citation advantage” as an artifact, while in reality the word on the longer-term citation advantage is still out. Would I be a reviewer or editor of this article, I would certainly have recommended to publish the access data now, but to hold off the publication of the citation data until the study is “completed”. Personally, I would be surprised if the higher access statistics would not translate into higher citation rates down the line. This would have been an important and much more credible paper if it would have been published in 2-3 years as opposed to a salami approach. And consider this: Imagine Davis’ et al would find – in their follow-up study – a significant difference between citation counts of OA articles and non-OA articles (or again no difference). Who can then rule out (and will we hear that argument from OA critics) that these additional citations aren’t intentionally produced by Open Access advocates, who now deliberately start citing preferentially those green-lock-marked articles from Davis’ dataset (or the other way around)? Sometimes it is better – as painful as it might be – to keep your preliminary study results under wraps, especially if premature publication destroys the integrity of your data.

1. Davis P et al. Open access publishing, article downloads, and citations: randomised controlled trial BMJ 2008;337:a568. DOI: 10.1136/bmj.a568
2. Laine et al. Clinical Trial Registration. BMJ 2007;334:1177-1178 DOI:10.1136/bmj.39233.510810.80
3. Eysenbach G (2006) Citation Advantage of Open Access Articles. PLoS Biology 4(5) e157 DOI: 10.1371/journal.pbio.0040157
4. Eysenbach G. The Open Access Advantage. J Med Internet Res 2006;8(2):e8 URL: DOI: 10.2196/jmir.8.2.e8
5. Joseph J. Esposito. Open Access 2.0: Access to Scholarly Publications Moves to a New Phase Ann Arbor, MI: Scholarly Publishing Office, University of Michigan, University Library vol. 11, no. 2, Spring 2008 URL:

Questions for the authors (Davis et al.) - added 01/08/2008:
1. What were the mean citation counts in both groups (crude, unadjusted - e.g. "0.5 versus 0.6 citations)? Interestingly, these crucial data were omitted. What was the absolute difference in citations between the groups?
2. When exactly was the test/were the tests for self-archiving conducted (it would need to be conducted continuously because author can self-archive their manuscript at any time)? How do you explain the low self-archiving rate? What was the self-archiving rate in the control group? (i.e. contamination rate - self-archiving in the control group reduces the power to detect differences)?
3. How do you explain your results that being on the cover page has no statistically significant impact on the number of citations (your Table 3), and even significantly REDUCES the odds of being cited (your supplementary file)? Wouldn't one expect that editors select higher impact / quality papers for the title page, and that those are more often noticed and cited?
4. How do you reconcile your results that papers covered in a press-release have no citation advantage (Table 3) with previous research stating the opposite? What is the odds ratio for that variable? (the odds ratio from the supplementary file is missing) And why didn't you stress that your paper appears to challenge not only the "dogma" that open access articles are cited more often than non-Open Access, but also some other "dogmas", for example that papers covered on the title page and in press releases are cited more often?
5. How do you explain that you can't even reproduce the bias you are talking about in the introduction in your own data set? I think we all (except perhaps Stevan Harnard) agree that self-archived articles are biased towards higher quality (as are gold-OA articles), for the reasons discussed in your introduction and previously also in my PLoS paper [1]. But if we accept this, then why do you fail to show an increased citation effect of self-archived articles in your sample (see Table 3 and supplementary file)? While gold-OA was randomized, green-OA was not. According to your data, self-archived articles are even less cited than non-self-archived articles (though not statistically significant). Doesn't this finding directly contradict your conclusion that previous studies have found a citation advantage due to a quality differential/self-selection bias? If self-archived studies are the "better" studies which have been shown to be cited more often (any discussion about the "cause" aside), wouldn't it be necessary to see "self-archiving" status to be a strong independent predictor for citations?
6. You write "Previous studies have relied on retrospective and uncontrolled methods to study the effects of open access.". How do you call the 2006 PloS study? [it's a rhetorical question, of course. The PloS study was a in fact a prospective cohort study. In any cohort study, you have cases ("exposed") and controls. In fact, a RCT is also a cohort study, with the only difference that exposure status is assigned randomly so that unobservable confounders are distributed equally. But a prospective cohort study is neither "retrospective" nor "uncontrolled". I also take exception to the remark that previous studies have confused cause and effect. There is no discussion about "cause" in the PLoS study. What was said is that OA status remained an independent predictor, even when we control for known confounders.]
7. What is the trial registration number / where is the trial protocol? What timeframe was originally defined as the primary endpoint?
8. You argue that “our time-frame is more than sufficient to detect a citation advantage, if one exists”, citing the 2006 PLoS study, which in fact found an early citation differential between OA and nOA PNAS articles. However, PNAS has a very high impact factor of >10 (i.e. a very high citation rate). What are the impact factors of the journals you included and how would the different citation rates affect the type II error?

Citation suggestion:
Eysenbach, Gunther. Phil Davis: Open access publishing, article downloads, and citations: The word is still out. Gunther Eysenbach Random Research Rants Blog. created 2008-07-31, updated 2008-08-01. URL: Accessed: 2008-08-01. (Archived by WebCite® at

1 comment:

Joseph J. Esposito said...

I can't comment on the methodological questions debated here, but I wish to state that all talk of an "open access advantage" is based upon taking assessments too early in the game. We are far from having any idea as to what OA will mean for scholarly communications. My own view is that searching for data for a future event is not and cannot be scientific. Thus we are left to our hunches, informed or otherwise. Mine is that OA won't mean much much because there is always competition for attention and only the strongest nodes on a network receive much attention at all, whether they are OA or toll access.

Joe Esposito