Section 3 Incentives

Aligning the incentives of authors, publishers & reviewers to favor good science

3.1 The Old Funding models

This section is a bit polemical and here mostly for those outside of academia who need the context for the system I’m hoping to disrupt

3.1.1 Pathologies of the Subscription Model in the Internet Era

For those unfamiliar with the current problems in academic publishing

The classic funding model for an academic journal was the subscription model where a university library, and/or department held a subscription to the journals that the university’s staff wanted to read. These subscription fees paid for the administration and printing of the journal. In the era of physical publication large scale systematic infringement on a journal’s copyright would be expensive, requiring a printing press and relatively easy to enforce. With the advent of the digital age journals ran into the now all too familiar problems of business models that relied on the scarcity of a physical medium that is now practically infinitely replicable at negligible cost.

Now anyone with an internet connection can have a distribution network approximately equal to that of the official source and copyright is unenforceable in practice with a lot of unpleasant side effects arising from attempts to make it so. Like most businesses they engaged in the attempt keep the same business model by making an infinitely replicable digital artifact artificially scarce with pay walls as well as other rather futile attempts to deny the new reality. Now rendered effectively both non-excludable and non-rivalrous electronic academic papers became a true economic public good in a way that physical copies were not. All attempts to deny this development and return to the undesirable condition of scarcity have been technically and/or politically dangerous. New business models which do not depend on the scarcity of access to papers must replace the old way of doing things, if we are to get back to a reasonable academic publishing market.

The loss of profitability of smaller journals resulted in increased consolidation of the publishing market into a small number of large concerns. These larger publishing houses had some economies of scale, and usually some prestige publications still with print versions and which commanded a premium subscription. These large publishing houses made the most of the shield provided against real competition by the grant of ‘limited’ monopoly afforded to them by nation states in the form copyright law and the threat of it’s enforcement, a threat which has claimed at least one life Aaron Swartz. Note that this is a copyright held by the journal on content which they have only a minor role in creating. It is scientists, usually publicly funded, that write and review these articles. The journal provides basic quality pre-screening, matchmaking between authors and reviewers, and if you are lucky copy editing. Placed in a position of oligopoly power the publishing houses were, and still are to a significant degree, able to dictate terms to the university libraries which purchase subscriptions. This has opened the door to classic oligopolisitic practices such as bundling high quality journals in with poor quality ones, a pattern that will be familiar to anyone unfortunate enough to have had to engage with a company selling what the Americans refer to as ‘cable TV’. The management of these complex subscription packages has been outsourced within arcane academic bureaucracies the absurdity and complexity of which put to shame the worst imaginings of Kafka and Heller. Consequently, the majority of academics who actually use these journals have little to no information about how this works and little hope of providing feedback about the quality of service to their librarians that will have a meaningful impact on purchasing decisions. This decoupling of purchasing decisions from the users of the service contributes to the ability of the publisher to continue to deliver worse services at higher prices.

3.1.2 Open Access Won’t Save Us

To squash the hopes and dreams of those familiar with the problems of the subscription model but who think open access is a panacea

Don’t get me wrong open access is unequivocally a step in the right direction with respect to making publicly funded research available to the public. Open Accesss 2020 & Plan S go a long way to addressing this issue they are, however, far from enough to fix the malaise in academic publishing. Alas open access as currently conceived allows the publishers to give up on an expensive losing fight with ‘pirates’, (hats off to captain Alexandra Elbakyan of the good ship sci-hub), and still get paid. Unfortunately the publishers saw this coming, it’s been in the works of years and they’ve had time to devise a cunning plan. The pay to publish model has many of the same problems as the old subscription model. In some cases it makes the actual price paid to publish more transparent to the academics paying it from their grants, but this can quickly fade into the background cost of doing business along with all the other overpriced things we spend grant money on. If deals with publishers to bundle x number of publications from affiliated researchers of the grant awarding body or university in the set of Y journals are not already in the works I’ll eat my hat. Scratch that I looked it up and it’s already happening - I promise I predicted it before I checked, see: Projekt DEAL This provides a convenient return the status quo of extracting large fees from research grants to pay for publishing just by a slightly different bureaucratic channel and with little to no improvement in the incentive to provide a quality service. If anything publishers will gain increased revenue from this having successfully spun it as a concession on their part to move to open access, so that they can charge as much or more in publication fees as they charged in subscriptions in the name of a ‘sustainable’ business model. Because anything less that multi-billion dollar annual profits is clearly unsustainable.

3.2 What is the function we want from the publishing industry?

What do we need from a journal and how can we incentivise that and only that reviewer / author matching prioritisation / curation type setting, proofing plagarism, hosting

In the current publishing ecosystem incentives are often actively bad and when they are not actively bad they are often pretty neutral with respect to the key outcome of good quality scientific research.

pit of sucess

The Pit of Success: in stark contrast to a summit, a peak, or a journey across a desert to find victory through many trials and surprises, we want our customers to simply fall into winning practices by using our platform and frameworks. To the extent that we make it easy to get into trouble we fail.

The problem with Altmetrics: goodhart’s law issue with metrics/impact - resolve through alignment

3.3 The Review bounty

A new funding model for academic publishing

I propose a new system for funding academic publications which I call review bounties. If you want to publish a manuscript you don’t pay a publication fee you put up a bounty for it’s review. If you wish to be a publisher you don’t put up a pay-wall and charge an access subscription you take a cut of the review bounty. As an author you take what you want to publish to a pre-print and review server you state how much you are willing to pay to have it reviewed and published and you make ranked choice list of venues you would be interested in publishing your manuscript. Publications then have the option to take you up on overseeing the review and possible publication of your manuscript. They contact reviewers who agree to review the manuscript in exchange for an agreed cut of the review bounty. Reviews are open, not anonymous, as are all payment amounts. This openness is essential to prevent grift as if a journal permits you to simply pay some friends for an easy review their reputation as a venue for rigorous review must suffer.

There are a number of interesting variations you can make to this basic model. consider the ‘bug bounty’ in which a portion of the review bounty is held in reserve and anyone who finds an error in the work that is of material relevance to the conclusions of the work can claim the bounty. This can be decided by the senior author, a majority of the authors or a consensus of the publisher and original reviewers. The bug bounty accrues back to the authors, publisher and reviewers over time, rewarding them for making non-obvious errors and incentivizing bug hunters to find errors quickly and thus get larger rewards.

In “An Incentive Solution to the Peer Review Problem” (Hauser and Fehr 2007) Marc Hauser & Ernst Fehr propose that reviewers who fail to deliver their reviews on time be penalized by having any papers they subsequently seek to have published at the journal for which their review was late be held up twice as long as they made other authors wait. This proposal may incentivize timely review but punishing tardy reviewers in this way has serious negative externalities, it delays the publication of potentially important work and will disincentivize anyone who has ever been late with a review from publishing in that journal again. This is not likely to be convenient for the journal or the reviewer. Overall this does not seem like good game theory outside of a very narrow scope, when designing incentive systems we must keep in mind that the end goal is better science we can’t optimize solely for punctual reviewers. A solution to this problem in the review bounty system is that a reviewer forfeits the bounty if a review is not provided by the deadline. Then another reviewer can claim it, or the original reviewer can reclaim part of it but only if they now provide their review up-front prior to another reviewer claiming the bounty. The longer an author is kept waiting beyond the deadline the more of their bounty they recoup with the cost coming partly from the publisher to incentivise speedy organisation of the review.

Anonymity and secrecy in peer review is in my opinion thoroughly over-rated, it can conceal abuses of power like holding up someone else’s paper so you or your mate can publish first as much or more than it provides scope for junior people to critique the work of senior people. Openness increases the reputational risk to senior people of petty behaviour like denying someone a job or grant application because they canned your paper. Make a habit of that sort of thing and people will now notice and hold you to account. If you can’t both respectfully take and dish out an intellectually rigorous critique what are you doing in this line of work?

If perspective authors cannot afford a review bounty could also have a croud funding (not just public but potentially grant awarding bodies, foundations etc.) method built into the platform to allow the sponsorship of pre-prints to afford enough of a bounty to get reviewed and published. Option a portion of bounty only payable if the rest of the minimum value is met by others and reviewers secured.

3.3.1 poor incentives for quality publishing

• formatting

  • general friction around formatting file types and styling
  • catering to pagination and the dead tree
  • figure panels
  • lack of useful animation/interactivity

• declining readability

  • Scientific manuscripts are getting harder to read (???)

• Citation Dilution (the bigger a paper the less specific you are being when you reference it) ~ credit dilution / attribution?

• https://www.pnas.org/content/118/41/e2021636118

• scope beyond reviewer competence

  • particularly in biology the are massive papers published whith 10 years ago would have taken 10 years to complete can now be a sinlge complec paper

• narative seeking / lack of pre-registration/distinction between the predicted and the post hoc

• commitment to metaresearch to improve research publications

• A public benefit corporation like model or equivolent corporate structure to help prevent a return to the current rachet

review bounties as a key component of the attracting quality reviewer time away from established publishers

physics archive - decoupling of publishing from archiv, pre-prints don’t solve the problem need a new hybrid solution

Some have made the case that pre-prints are all we need and we can simply cut out the middle man of publishers altogether. Making the case that we can and rely on post-publication review for quality assurance and potentially on algorithmic methods for curation. Most, however see that the journals do serve some curatorial and quality filtering roles, as well as providing an organizing function for arranging that peer-review actually take place. The problem is that under the current economic model the big publishing houses are in a position to act extractively and to seek rents on their oligopoly of prestige publishing, whilst providing a pretty poor quality of service to the academics who are ostensibly their customers. The value that they provide is substantially offset by this behaviour which arises from some poorly aligned incentives. This misalignment is costing billions often in public research money that could be much better allocated if their were a healthier market in publication outlets.

3.4 x

At the moment it is reasonably expensive in both time and money to get a publication in a high prestige journal. Getting such a publication constitutes a reasonably strong and reliable signal that you have put in quite a lot of work to the process or trying to get your paper published. This however is not perfectly

costly signaling theory in publishing - jumping through arbitrary hoops to signal prestige? make it harder to fake signals, gaming metrics/acceptance with low cost noise that sounds good but lacks substance, stronger openness norms and smaller units give this bad quality signal less space to hide

providing your analysis as a literate programming artifact is a fairly high cost signal that is hard to fake, it also has much lower verification costs for 3rd parties. I can just go and look at the code underpinning your paper and see if what you did is good before I would have to ask you, coming from someone you don’t know a request to look over your exact methods

? croud funded news and views bounties? pol.is and quadratic voting for review/synthesis subject selection

such a platform has the potential to create a transparent competative market for publication services especially if aided by regulatory or other institutional interventions that penalise the predatory business practices of current industry players

The authors of the The tragedy of the reviewer commons (https://doi.org/10.1111/j.1461-0248.2008.01276.x) (Hochberg et al. 2009) identify as an issue: “attempting to publish the smallest acceptable unit” this appears to conflict with proposals for ‘micropublications’ or other smaller units of publication but I would argue that the underlying concerns that lead to calling this an issue arise from conceiving as a ‘conventional’ paper as the minimum viable unit of unit publication.

steven novella - minimum publishable unit vs minimal unit of publication, emphasis publishable several crappy experiments and a story, if experiment have to stand on their own they will need to be of higher quality not obfucated by being strung together in a narrative.

see also sciencematters.io - ‘single observation publishing’

see the science breaker (universite de geneve) lay summaries / news and views commision (bounties from anyone?)

reducing the size of the unit of publication from a full paper to a single experiment, dataset, protocol description, experimental design, theoretical model etc. This makes it easier to do your best work because the scope of the task is smaller, it also make it easier to review effectively

If we conceive of the minimum unit of scientific publication as what has become a conventional paper with several often complex experiments and an overarching narrative then an underdeveloped paper can indeed be burdensome to review. We have a compounding problem of increasing technical complexity of papers due to in part rising expectations in the amount of work going into a paper to make it worthy of a ‘top tier’ journal and increasing volume of articles.

The authors of ‘The tragedy of the reviewer commons’ (Hochberg et al. 2009) pre-review by colleagues - do this on platform and don’t waste time circulating a bunch of copies by email and reconciling the resulting comments, have and invited reviewers feature that lets you invite input on your work prior to submitting it for formal review

reviewing a modern paper is a daunting task - uncertainty about the expertise of the other reviewers what is the subset of things in the paper which you are expected to the expert reviewer on are the areas outside your scope adaquately covered by other reviewers? - I full expect people often tacitly assume the editors have handled this adequately as it theoretically their responsibility and just review what they feel comfortable reviewing.

medium/form mismatch - way to much for the short format paper

References

Hauser, Marc, and Ernst Fehr. 2007. “An Incentive Solution to the Peer Review Problem.” PLoS Biology 5 (4): e107. https://doi.org/10.1371/journal.pbio.0050107.

Hochberg, Michael E., Jonathan M. Chase, Nicholas J. Gotelli, Alan Hastings, and Shahid Naeem. 2009. “The Tragedy of the Reviewer Commons*.” Ecology Letters 12 (1): 2–4. https://doi.org/10.1111/j.1461-0248.2008.01276.x.