Annika Kangas , Teppo Hujala

Challenges in publishing: producing, assuring and communicating quality

Kangas A., Hujala T. (2015). Challenges in publishing: producing, assuring and communicating quality. Silva Fennica vol. 49 no. 4 article id 1304. https://doi.org/10.14214/sf.1304

Abstract

This paper is based on a session “How to make forest science available for all? Publishers’, editors’, and authors’ challenges” at the IUFRO XXIV world conference, organized by Pekka Nygren and Eeva Korpilahti from the Finnish Society of Forest Science. The presenters dealt with the topical problems of publishing scientific knowledge from different perspectives. The talks covered the development of journals, publications and submissions, benefits and drawbacks of open access publishing as well as electronic and traditional publishing, and possibilities to promote interesting papers either from the journal’s or from the author’s perspective, and the problems of disseminating the scientific results to the end users. In this paper, a few prevalent viewpoints, inspired by the session, are raised and discussed with some suggestions included.

Keywords
peer review; open access; altmetrics; citation index

Received 21 January 2015 Accepted 23 June 2015 Published 29 June 2015

Views 33621

Available at https://doi.org/10.14214/sf.1304 | Download PDF

1 Exponential growth of scientific publishing

The first peer-reviewed articles were published in 1665 in journals Philosophical Transactions and Le Journal des Sçavans and in 2006 there were already 23 750 titles (Jinha 2010). The number of submissions has grown exponentially during the last 10 years, and the number of journals and published papers has doubled in that time (Dreyer 2014). Jinha (2010) estimated that 2009 the number of published peer-reviewed articles exceeded 50 million. On the other hand, number of forestry journals has not increased that fast. Number of papers dealing with forests during 2002–2011 was about 20 000, but only 21% of them were published in forestry journals. That may also mean that the hottest topics are published elsewhere, in series where the impact factors are higher, which may be a threat to the future of forestry journals in general (Dreyer 2014).

2 Is open access a problem or a solution?

In recent years, open access (OA) publishing has been recommended by many research organizations as well as funding agencies (Tutkimuksen avoimuudella... 2014). Open access publications are considered especially important in the developing countries, where they may be the most prevalent mean to get access to scientific articles. When OA articles have been compared to non-OA articles in the same journal, the OA papers have a huge advantage in citations: Eysenbach (2006) noted that after 4–10 months the probability of not getting any citations was 49% for non-OA and 37% for OA articles, and also the average number of citations was higher for OA articles (1.5 versus 1.2). There is therefore great incentive for a researcher to publish in OA journals, if they have funds for paying the publishing fees. It is therefore reasonable to reserve funds in project budgets for publishing (also) on OA journals; the drawback is that since no respective savings are expected from journal subscriptions, the increasing OA publishing expenses will decrease the actual research money.

There are, however, also a number of “predatory” open access journals, quality of which is not adequate and which do not even follow reasonable ethical standards. Beall (http://scholarlyoa.com/publishers/) lists 825 OA journals that have a dubious quality. The predatory journal listed may, for instance show feign impact factors, publish already published papers, publish non-scientific texts, use non-qualified reviewers or fake reviews and so on (Beall 2012). In the worst case, these journals are published in fake institutions by fake editors (see http://scholarlyoa.com/2014/01/09/questionable-oa-publisher-launches-with-a-clever-website-and-52-new-journals/).

3 Is peer-review still the best quality assurance?

Having the paper peer-reviewed is not necessarily a guarantee of good quality, even though it has been shown that peer-review process improves the quality (Armstrong 1997). Severe criticism has sometimes been presented. For instance, Richard Horton, editor of The Lancet, has written (2000): “We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.” Yet, peer-review has been deemed as the best quality assurance available.

Increasing number of journals and submissions means that the number of review reports needed has increased exponentially. The rejection rates have also increased with the result that same papers are reviewed several times in several journals: with 50% rejection rate each published paper may have been reviewed in 2–3 journals and got 5–10 assessments before publishing (Hochberg 2010). That is a burden to the whole peer-review concept, and might reduce the average quality of the review reports.

As a possible solution to this problem, a so-called open peer-review has been suggested. For instance, all articles submitted could be published immediately and the review process could take place afterwards. Any reviewer wishing to do so could comment and the reviews could be published alongside the manuscript. The authors could still withdraw the published paper and revise it according to the comments received. Thus the work is public immediately and the papers, which the reviewers find as good or bad ones are easily distinguished. Having the reviews published would remove the possibility to submit poor-quality manuscripts to different journals until favorable reviewers are found. On the other hand, it would also remove the possibility of reviewers to stop a good-quality manuscript from being published if they dislike it. The open peer-review process could possibly both reduce the number of submissions and improve their quality. It might also improve the quality of the whole review process, when also the reviewers get scientific merit on writing good reviews, and have to publish their names. On the other hand, it might reduce the number of researchers willing to make reviews.

Another fairly recently established effort of renewing the peer-review concept involves submitting manuscripts to a qualified researchers’ community rather than to a single journal. The author will then get voluntary review by non-associated researchers. Then the author has an opportunity to submit the manuscript to a journal of his/her choice after the favorable review reports have been obtained. This type of activity is organized by Peerage of Science Ltd (http://www.peerageofscience.org/). The promise of the peer-reviewers’ community is to assess, show and increase the value of peer-reviews and speed up the process of matching a paper with a relevant journal. Papers are given credit via quality indices, which measure the quality of the reviews assessed by other reviewers, the number of reviews received, and the review-quality-weighted average of scores given to seven categories of the article, including breadth, impact, originality, methods, data, inference, and literature. Both open peer-review and Peerage of Science include features that may be considered when refining the whole peer-reviewing system.

4 Are the best publications those that are cited most?

As the number of journals and published papers is exponentially increasing, there is a danger that the availability of the publications will be more important factor in determining, which publications are actually read and cited than the quality of the research (Hairiah 2014). This is especially important if the open access journals have (on average) lower quality than the journals requiring subscription.

To improve the visibility of good science, it would be important that the best papers are open access (Hairiah 2014) or at least that a summary of such papers, preferably with non-technical language are available to everyone (Way 2014). It will be increasingly important to publish also short syntheses of the publications based on reliable science to reach the end users (Stelzer 2014). To improve the impact of science in general, we would need to be able to better separate good papers from poor ones. It could be both a responsibility and an asset for publishers to distinguish the very best articles and provide OA for those papers even without author fees.

In recent years, the researchers and journals have been measured with metrics based on impact factors. However, 80% of the impact of a journal is attributable to 20% of the papers (Neylon and Wu 2009). Furthermore, variation in the volume of research between subject fields affects the general level of impact factors. For this reason, publishers have started to provide several modified citation indices that, i.e., normalize the impact factor with the total number of citations in the subject area. Even then, the quality of single articles cannot be measured accurately with the impact factors of journals. In fact, since the articles have gone online, the importance of the impact factors has reduced: the proportion of highly cited articles in high-impact journals has reduced and that of less highly ranked journals improved after the 1990’s (Lozano et al. 2012). This may be due to authors increasingly relying on the results of search engines. With this development, good selection of keywords may markedly increase the visibility of the article.

5 How to better measure the impact of articles?

One obvious metrics for measuring the quality of single articles is the number of times the article is cited by others. However, the number of citations accumulates slowly during years and cannot be used to evaluate the quality of recent papers. It is also noteworthy that different types of articles may have different impact life-cycles, e.g. highly relevant for a shorter time versus moderately relevant for a longer time. Thus, one should be careful for not placing unintended valuations on different article types when applying citation indices over time.

For article level impact measurement new measures (the so-called Altmetrics) have been introduced (Thelwall et al. 2013). These include number of sights, saves, mentions in Twitter or Facebook or blogs (Wennström 2014). The Altmetrics may also measure the importance of the research from the society perspective rather than just scientific perspective. They may help the researchers and end-users to find the papers that are most relevant in their own work (Neylon and Wu 2009), but they also introduce the risk of reading the most popular rather than most important papers. Thelwall et al. (2013) found that the altmetrics moderately correlated with the citations of papers published in Nature and Science, but the best alternative metrics was the recommendation at F1000 (http://f1000.com/prime/about/whatis/how), which is a recommendation of peer-nominated faculty members for the best articles they have read in their fields. Such recommendation system (with names of the people giving the recommendations) might best describe the quality of single articles.

Both, journals and authors, can use these new measures in actively promoting new articles. The authors may promote their work for instance in social media such as Facebook and Twitter. It has already been noted that the promoted works do get more downloads and citations than average papers (Wennström 2014). This increases the visibility of science, but there is also a danger that what gets cited are not the best articles but the most actively promoted research. Journals may promote the papers the editors find the most important or the best quality for instance by asking reviewers to develop their reviews to commentaries that are also reviewed and published with open access (Way 2014). Such summary of the important aspects of the papers will help the researchers to notice publications of importance to them. Journals could promote the articles also by publishing other supporting material such as audio-supported slide-show presentations alongside the article (e.g. http://dx.doi.org/10.1016/j.jenvman.2014.05.029).

Reviewers could also provide standardized labels that highlight the particularly interesting, relevant or strong features in that article. Scientific quality goes beyond scientific soundness (which is a threshold level for a proper article) and is a product of several properties, originality, theory, methodology, increment to knowledge, etc. It would be helpful for readers to know for what reason an article has received a high score. Various score decompositions are currently being used in journals’ review sheets. However, at present they appear to be asked for the purposes of editorial decisions only. In the future, the most relevant relative strengths of articles as seen by the reviewers and the editorial board could also be published.

High-ranking journals do not necessarily accept topics that are important at national level (Hairiah 2014). As researchers strive for publishing in the high-ranked journals to get merits, it may also be difficult to keep journals concentrating on national level questions or applied research running (Moser 2014). If the quality of the research were not solely evaluated on a basis of the journal impact factor but also on article level measures, it would be easier to persuade researchers to publish their work also in national languages and on national topics, and possibly increase the societal impact of the research.

6 What can single researchers do?

Universities and research institutes as well as commercial publishers hold a steering role in developing the operational environment of scientific publishing. In any case, the body of research publications grows rapidly, predator journals violate research ethics, open access publishing continuously competes with subscribe publishing, peer-reviewing faces overhaul challenges, and single article quality receives growing attention.

Individual researchers can manage the situation via paying attention to research ethics. One should only publish on forums that follow good scientific practices. For checking this, it is advised to learn about predator publishers and carefully study the processes of the publisher and the journal of interest. An ethically reliable journal has a rigorous and transparent publishing process and publications listings at major indexing services (Web of Knowledge, Scopus). Besides, a researcher is advised to think through his/her publication strategy, i.e. deliberate choices why to publish, therefore to whom to publish and where and how to reach the target audiences. Choosing one’s scientific profile, in terms of how to weight pursuing scientific merits versus societal impacts, informs not only doing research but also selecting journals to submit and advertising the published papers in various channels. Typically, an individual’s publishing strategy may mean balancing between traditional citation indices and societal effectiveness.

For doctoral students, however, hard scientific competition usually offers limited opportunities to select where to publish. Choosing a low-quality journal may reduce career opportunities, while established high-class journals may easily reject younglings’ papers. A solution may then be to identify an ethically working evolving journal, which has either article-quality orientation or up-to-date advertising features. It is important to remember that at the end a good-quality paper will find its readers and citers (nearly) regardless of the actual journal. Both doctoral students and other researchers will increasingly need clarifying information on publishing alternatives and their potential impacts on their scientific profiles and careers.

7 Conclusions

In recent years, the citation indices of journals, authors and papers have been used more and more in evaluation of them but also in evaluation of universities and even fields of science. At the same time, all these measures have also been highly debated, as they might be misleading for many different reasons. Therefore, we need to keep in mind that the measures do not define good articles, journals or authors, but they may be helpful in finding them. Open access also may have both dubious and good features. Yet, as wide access as possible to the best papers is the benefit of both authors and journals, but most of all science.

References

Armstrong J.S. (1997). Peer review for journals: evidence on quality control, fairness and innovation. Science and Engineering Ethics 3: 63–84. http://dx.doi.org/10.1007/s11948-997-0017-3.

Beall J. (2012). Criteria for determining predatory open-access publishers (2nd edition). http://scholarlyoa.com/2012/11/30/criteria-for-determining-predatory-open-access-publishers-2nd-edition/. [Cited 20 November 2014].

Beall J. (2012). Predatory publishers are corrupting open access. Nature 489. p. 179. http://dx.doi.org/10.1038/489179a.

Eysenbach G. (2006). Citation advantage of open access articles. PLoS Biol 4(5): e157. doi:10.1371/journal.pbio.0040157. http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.0040157.

Hochberg M.E. (2000). Youth and the tragedy of the reviewer commons. Ideas in ecology and evolution 3: 8–10. http://dx.doi.org/10.4033/iee.2010.3.2.c.

Horton R. (2000). Genetically modified food: consternation, confusion, and crack-up. MJA 172: 148–9.

Jinha A.E. (2010). Article 50 million: an estimate of the number of scholarly articles in existence. Learned Publishing 23: 258–263. http://dx.doi.org/10.1087/20100308.

Neylon, C & Wu S. (2009). Article-level metrics and the evolution of scientific impact. PLoS Biol 7(11): e1000242. http://dx.doi.org/10.1371/journal.pbio.1000242.

Lozano GA, Larivière V, Gingras Y. (2012). The weakening relationship between the impact factor and papers’ citations in the digital age. Journal of the American Society for Information Science and Technology 63: 2140–2145. http://dx.doi.org/10.1002/asi.22731.

Thelwall M., Haustein S., Larivière V., Sugimoto C.R. (2013). Do altmetrics work? Twitter and ten other social web services. PLoS ONE 8(5): e64841. http://dx.doi.org/10.1371/journal.pone.0064841.

Tutkimuksen avoimuudella yllättäviä löytöjä ja luovaa oivaltamista. Avoimen tieteen ja tutkimuksen tiekartta 2014–2017. (2014). Opetus- ja kulttuuriministeriön julkaisuja 2014:20. [In Finnish].

Total of 11 references

Original presentations

The original presentations (available at http://www.iufro.org/download/file/16684/4139/iwc14-abstracts_pdf, p. 108–109):