Decentralizing Journals and Peer Review DAOs

Peer Review Journals and Peer Plugs

After years of begging my physician friends to send me articles in certain journals and using Sci-Hub (if I’m not on a work computer), I am sick of it!!!! Journals you have drawn my ire and shall feel the wrath of a newsletter writer. I’m sure you’re quaking in your inconveniently paywalled boots.

I think it’s worth talking about how peer review and journals work today because COVID actually showed us how important both speed and quality of papers need to be in the era of the internet and fast-changing information. It’s even more interesting when in parallel we see the rise of novel community structures like Decentralized Autonomous Organizations which have features that are well-suited to rethink journals and peer review.

The OG Peer Review and Journals As A Business

Getting published in journals matters for anyone wanting street cred, especially in healthcare. The same way tech people think you’re legit if you’re on a numbered list of people below a certain age cutoff, healthcare looks to your publication history in journals to assess your social capital. Once you get 3 papers in Nature I think you get named president of healthcare or something, or at the very least you have to tell aspiring and current med students how publishing is a requirement to get into the programs they want. 

To get published, there are many steps you have to go through after you finish your experiment/analysis.

First, you create a manuscript of your paper and submit it to the publications that might be relevant. In most cases you have to affirm that the paper isn’t under review anywhere else - having to wait for the full review process to conclude before submitting somewhere else significantly slows the process down. 

Editors of the publication assess if it’s a good fit for their publication. Each of the publications will have different rules about what they care about, which are some combination of whether it’s original, whether it’s of importance/furthering the field, readability for the audience, conflict of interest issues, and more. 

The editor will send the manuscript to a network of peers in the same field, aka the peer review process. Apparently some people call this getting “refereed” which I assume is like a “football vs. soccer” thing in academia idk. 

Reviewers are doing the reviews pretty altruistically, to stay at the forefront of research, and some social pressure from others in the field. Reviewers can be new to the field or very experienced. It’s really just about who’s free and if they have the right expertise to review.

The number of reviewers will change depending on the journal and nicheness of the topic, but it’ll usually be at least two. This process can either be single blind where the reviewer is anonymous, double blind where both reviewer and author are anonymous, or open where they know each other. From here suggestions might be made on how to improve the manuscript or it’s rejected/accepted.

At this point it gets published in the journal, the author is showered with praise and treats themselves to a massage, and they get paid lots of money from the journal who is made more valuable thanks to their contribution.

Wait, sorry, I’m being told the authors actually don’t get paid. Wait, the peer reviewers don’t get paid either? But how are the journals selling 7 figure subscription contracts to institutions when the institutions are essentially providing free labor for peer review? Uhhhhh just one journal makes more than $2B in operating profits with a 35%+ margin?????

The system of peer-review and publishing has worked for a long time when we were actually dependent on them for things like physical distribution of magazines, manuscripts, etc. Today, journals use the industry's reliance on peer-reviewed publications as a proxy for quality in order to coordinate a lot of unpaid labor. Authors are somewhat dependent on these journals for reputation points in their field. This is still viewed as very important in getting certain positions/fellowships, access to grants and funding, etc.

The current process has lots of issues:

• Peer-review as a process takes a long time. PNAS is a highly selective publication and says from submission to publication can take on average 6.4 months. It takes 6 of those months just to stop laughing at the name of the publication (sorry, I'm a child).
• Papers are at the whims of the editors, who can tend to be more conservative in the kinds of ideas they approve. This can easily lead to bias based on what the editors think are “important” issues, especially when it comes to issues that affect marginalized/lower socioeconomic groups. Plus it means papers will also rarely get published if the finding isn’t interesting even if it’s societally beneficial (e.g. meta-analyses of other studies, studies with negative findings, etc.). 
• Your paper is subject to whoever you happen to get as peer reviewers of your paper, aka 2+ people at random in your field. There isn’t a ton of training on how to be a good peer-reviewer, which can lead to variability in comments/approval. They might just disagree with the general point philosophically, they might have some different forms of bias based on who the author is, etc.

• As a result of peer review being largely altruistic/reputation within your department, the reviewers will typically skew towards other academics instead of people actually working in industry who might look at research from a more practical lens vs. theoretical. 
• Reviewers have day jobs and other things they are doing as well. Review fatigue is becoming more real as more papers get constantly submitted at a higher ratio than new reviewers enter. This is compounded by the fact that journals are happy to spin-off new journals that are increasingly more niche. There’s a whole journal just dedicated to the mitochondria, where authors are accepted based on how many times they use the phrase “the powerhouse of the cell”. More journals = more subscriptions institutions have to pay for = more work for reviewers.
• There seems to be an open debate on whether the current system of peer-review actually weeds out papers correctly. This experiment re-submitted 12 already approved papers and 89% of the peer reviewers said the paper shouldn’t be published. More recently hundreds of sham papers managed to get through the process as scammers posed as guest editors.
• Many argue that having this kind of information behind a paywall is bad for society, and even the institutions who pay these large subscriptions are getting sick of them. Elsevier has been having very public battles with several universities like University of California who feel the journals are charging more without adding anything of value knowing the universities essentially have to pay. 
  

As a whole, people in the field generally believe that peer-review is important as a concept and have pretty good experiences with peer-review. This is from one of the more thorough and large surveys of people that have participated in peer-review. 

However, in the age of the internet, maybe there are new ways to do peer review and disseminate information. In fact, it’s happening either way.

The Rise Open-Access Journals, Preprint papers, and Sci-Hub

In the last decade the internet has basically removed the need to print media to be the means of distribution, and that includes journals. This has led to the rise of a few different phenomena. 

First is Sci-Hub. In 2011 a 22-year old grad student in Kazakhstan named Alexandra Elbakyan launched sci-hub, a way to pirate papers behind a paywall. Elsevier got pissed, sued Sci-Hub, and now Alexandra has to keep moving her current location and the servers for Sci-Hub are constantly changing. This sort of reminds me of record labels vs. Napster moment in the music industry - eventually a better business model was found but right now it’s antagonistic.

Second has been the rise of open-access journals. These are journals that allow readers to access the published articles for free and instead derive most of their revenue from charging the authors Article Processing Charges. Some of the name brand journals also have a hybrid model where authors can pay to make their research open-access. And the prices can be steep - Nature for example charges $5500+. The pay-to-play nature of open-access has raised questions of whether these journals are going through as strict of a standard for peer-review or instead just focusing on volume of manuscripts published. PLOS seems to be the most well-known/respected of the open-access journals. For federally funded research, there has to be an open-access version within a year after being published in a journal - which might be the best of both worlds?

The obvious benefit of open-access journals is that anyone can access them, which in theory also means that anyone can comment on them post-publication as well. This is great for the distribution of ideas, but also, any issues are more likely to be noticed with more eyeballs. If you take this strain of thought to the logical conclusion, it might suggest that we should just publish things publicly even before peer-review and instead the peer-review should happen in the public sphere.

This is the theory behind preprint papers, which you can see in repositories like bioRxiv and arXiv. Why go through peer-review before publishing at all if you can just...publish it straight to the web? “I’ve been researching this topic for months and they just...tweeted it out…”

Preprint papers have seen a surge in popularity even pre-COVID, with many authors published to the web and getting comments from the Twittersphere/internet before the actual submission to peer-review journals. This analysis of preprints suggests that about 40% of bioRxiv papers end up getting published in peer-reviewed journals.

Why should peer-review happen amongst a small group of people when it can happen in the public eye? People can give comments, other people can give comments about those comments, etc. Ideas can get published faster, iterated on quicker, and become more of a group commentary vs. 6 month waits. 

This was especially necessary during COVID, when the speed of information dissemination was key. In the early days of COVID, many of the experimental treatments that were being tested around the globe would quickly make it to preprint papers where other institutions could implement interventions that seemed to work. 

However, this also highlighted the issues in making these papers extremely accessible - groups would weaponize these papers to fit their own agendas. In fact, the early preprint around myocarditis risk from the COVID vaccine that ended up being the talking point for anti-vax groups was eventually retracted because they messed the denominator up by 25 orders of magnitude. However the paper had spread like wildfire by that point. It’s like when you accidentally add two 0’s to the end of a Venmo request and you and your friend laugh about the mistake, except this time the mistake is confusing the entire scientific community and spreading misinformation about vaccine risks during a global pandemic.

So clearly there are pros and cons to each of the different models between closed journals, open access journals, and preprints. In the same way the web was an unlock for some of these models, maybe new technologies will create other novel models.

Peer Review DAOs?

My two roommates are self-proclaimed “crypto degenerates”. I’m learning a few things from them.

• You can 8000x your money in a week if you get the right tip from a twitter account with an anime character profile
• It is easier to sell a pixelated jpeg for $3M than it is to raise $3M for a chronic disease management company
• Our landlord does not accept any shiba inu related currencies for rent

But they also talk about more interesting stuff. In particular is the idea of decentralized autonomous organizations (DAOs) that are being experimented with in the crypto community. While very nascent, the general idea is whether you can set up systems of trust and incentives so that a community of people that don’t necessarily know each other can coordinate to get something done. 

Currently for journals this is done through top down, centralized decision making. The CEO of a journal is deciding the direction of the company, the rules are disseminated through the ranks of the company, and the authors/peer reviewers don’t have a ton of say or ownership in the direction of the journal. DAOs raise the question of whether you can do this without any singular entity making these decisions, and instead, the decisions are discussed and guided by participants in the DAO.

There are lots of good reads of DAOs like Mario Gabriele’s post and Linda Xie’s post. In my head the main things DAOs need are:

1. There’s some means of voting and governance on issues and participants have some form of weighted voting. The weight can be on the reputation of the person voting, how much ownership they have of the network via tokens, and more.
2. There’s some means of discussion where members can chat about things the DAO wants to do. A lot of this happens in Discord, Twitter, etc. today.
3. Membership to the DAO can be gated or not. You might need to purchase tokens to participate, be admitted by existing members, etc. Or it could be totally open.
4. There is some form of reputation scoring. Reddit for example has Reddit gold, and badges, etc. which travels with a person’s avatar. In the DAO version of this, those rewards can travel with a person who goes to any other DAO built on top of the same protocol (aka. Interoperability, a word you all know!)
5. DAOs have systems of organizing tasks that need to get done along with incentives for said tasks. This can be monetary incentives like bounties or reputational incentives to get other benefits in the DAO like more voting power.
6. There are rules in place for how things are executed and smart contracts which make them actually happen. There are basically no intermediaries to setting up and executing these (which can also make decisions permanent, posing other issues).

DAOs today are used in many different ways. They might be focused on making upgrades to blockchain protocols, making decisions about where the community should choose to spend money, pool capital together to buy things, and more. Below is a cool example of a vote for SushiSwap, which at a very high level makes it easier to trade different cryptocurrencies. Members of the community proposed to have analytics and dashboards built to create reports on activity across the different blockchain protocols. There’s a budget asked, a discussion forum on how it would be used, and a vote to secure it (along with a history on which wallets voted for the proposal).

As I learn more about DAOs, it strikes me that this would potentially be a great model for peer-review. Journals today act as coordination + reputation, but they charge an insane amount to do so. Maybe DAOs can do it better, especially since the main things that people involved in peer-review want more of is recognition, grants, transparency, and interoperable peer review contributions.

I am way out of my comfort zone here, but I wonder what a journal DAO might actually look like.

• A journal DAO exists that’s focused on a specific topic and only lets people in based on certain inclusion criteria (e.g. demonstrate they’re part of a university, have previously published research, pass a test to demonstrate their knowledge, idk). A little friction here will keep out trolls from being able to game the system.
• Papers get submitted by authors, who have a reputation score already based on their previous contributions to peer reviews. 
• Any reviewer can leave a review, but maybe high reputation authors offer some of their reputation currency to invite specific reviewers to take a look.
• Reviewers are rewarded with some form of non-monetary reputation token that can travel with them between different journal DAOs. The reviewers get reputation tokens based on the quality of the review, potentially decided by upvotes (maybe higher reputation people can give more upvotes?) and/or people in the community get a fixed number of badges to signify a specific thing (e.g. badge for timeliness of review, or thoroughness).
• Reviews are public so that commentary is transparent, though potentially the reviewers themselves are blinded, yet still able to receive reputation scoring.
• Reputation tokens give more weight for votes to decide different actions of the journal going forward that are put to a community to vote.
• Maybe you can’t even submit a manuscript for peer review until you’ve contributed some reviews/thoughts yourself.
• Maybe there’s some sort of central grant system for people to apply to which uses the reputation system. So even though it’s not directly getting paid for reviewing papers, you can build your reputation to make it easier to get grants for your own projects.

This is just me riffing and clearly not super thought out, but the idea is that members of the community should have a means of building reputation and using that reputation to have input in what the rest of the community does.

An open question is also how much information within the journal DAO should be easily accessible to the public? Maybe after a certain amount of reputation, authors can directly publish for anyone to read; otherwise, papers can be released once a certain number of people have reviewed it?

The key reason I think a DAO here could be useful is because so much labor is already working for free in the existing journal system while the journal charges an insane take rate + the reviewers aren’t getting much credit. A system like this might unlock some of that take rate to be redeployed into funding more research, would create a system where potentially more “out there” ideas could be funded if someone with a high reputation suggests them, and would generally make the peer review system more transparent since the reviews themselves are auditable.

Of course, this only works if the academics collectively agree that this Journal DAO’s system of ascribing reputation is legit. Right now I think there’s still a lot of respect towards the big name journals, but maybe in the future that might change especially if a well respected researcher were to be one of the people starting this DAO. This could also be a product the preprint repositories themselves roll out, or a well-known grant making entity (e.g. Bill & Melinda Gates Foundation, NIH, etc.) powers themselves to make grant allocations easier? I think it’ll be hard for existing  journals to move into this since it’s so antithetical to their business model.

We’ll see - I think there’s an idea there that can be fleshed out.

Thinkboi out,

Nikhil aka. “DAOn bad”

Thanks to Bea Capistrant, Dr. Jenine John, and Apoorva Srinivasan for reading drafts of this

{{sub-form}}

‍

‍If you’re enjoying the newsletter, do me a solid and shoot this over to a friend or healthcare slack channel and tell them to sign up. The line between unemployment and founder of a startup is traction and whether your parents believe you have a job.