Category: Peer Review

The Other Academy Awards

Two related problems have been bedeviling the current discussion of new modes of academic work. First, it remains unclear to many academics how we can effectively assess digital scholarship, given its many shapes and sizes and often complex, collaborative production. This problem is receiving so much attention right now that we devoted our entire last issue of the Journal of Digital Humanities to it.

Second, given that many of these digital genres—multimedia scholarly sites, sophisticated digital collections, long-form academic blogs, and the like—are published directly to the web, a need has arisen for post-publication, rather than traditional pre-publication, peer review. Last week I was on a panel at the 2013 American Historical Association annual meeting on the future of history journals and peer review, and many in the audience were journal editors who were skeptical about the notion of post-publication peer review. Indeed, for many history and humanities scholars, “post-publication peer review” is an oxymoron; the only true form of peer review is the one that occurs before publication, and that helps to determine in a binary way whether an article or book is published in the first place.

Yet there is an obvious form of post-publication peer review already in wide use—awards—and I would like to suggest that we use them even more widely to help solve the problem of how to assess digital scholarship. As they currently stand, academic awards are icing on the cake. The prizes that the AHA gives every year—mostly books, with a few additional categories for articles, films, reference resources, and lifetime work—are wonderful signifiers of highly distinguished work. To receive one of these prizes is a major career achievement. One of the goals of the Roy Rosenzweig Prize for Innovation in Digital History, awarded by the AHA since 2009, was to validate the very top work in this new field.

One award isn’t nearly enough for our field, however, or for others that increasingly involve digital work. We need to recognize a broad swath of scholarship and innovation using awards, since it’s an effective way to signal creative, good work. Awards can be a clear form of professional validation for digital scholarship that is understandable to everyone in academia (including those outside of digital humanities), and that doesn’t rely on more controversial forms of post-publication peer review such as open review or crowdsourcing. How do we know that a professor’s blog is worthy of significant credit, that it is more than just musings and is having an impact in a field? A certifying scholarly organization or review panel has deemed it so, from a crowded field.

Furthermore, like the Grammys, we need to give awards not just for Record of the Year but for Best Jazz Instrumental and Achievement in Sound Engineering. Knowledgeable review panels with a deep understanding of the composition of digital work should be able to give out both general awards for digital projects and distinct awards, for instance, for outstanding work in user interfaces for digital archives.

Thankfully, there are already initiatives on this front. A new slate of annual Digital Humanities Awards has just launched, with an international review committee. (Nominations due today!) Now we need scholarly societies to back both interdisciplinary and disciplinary awards with their imprimatur. From conversations I’ve had recently, I sense that is likely to happen in the near future.

Of course, we need to be aware of the rather valid objection that awards can be overdone. We should avoid the digital equivalent of an award for Best Zydeco/Metal Duet. (Actually, that sounds incredible.) Last year the Recording Academy sensibly lowered the number of Grammy categories from 109 to 78. Furthermore, it’s important for awards to have some minimum number of entries or nominations every year, and as with book awards, peer review panels must retain the option of giving no award in a year if none of the options are deemed worthy. Awards must be meaningful.

Right now, however, we should think more broadly, rather than narrowly, about giving awards for digital work. There are precious few opportunities for digital projects to receive external validation. I continue to believe that other forms of post-publication peer review are needed as well (especially for developmental editing rather than vetting), but let’s at least start with a larger slate of rigorously determined awards and some (virtual) gold statues.

What Will Happen to Developmental Editing?

My colleague Zach Schrag wrote a guest post on Mike O’Malley’s blog two weeks ago with some significant criticisms of what we are trying to do with PressForward. He expressed a general worry that we were out to destroy a proven system of scholarly review, and a particular worry that we were casting off what is often called “developmental editing,” or the sharp eye of a savvy editor making suggestions for improvement. It’s a serious and important point: few of us can produce flawless arguments and prose from scratch, and can use the help of others to sharpen our writing and ideas.

As I wrote in a quick comment on Zach’s piece, I do not disagree that good editors can be crucial to the advancement of scholarship. It’s just that I do not believe Zach’s wonderful personal experience with an editor is very representative of the experience of scholars in 2011, or presents an accurate and whole picture of the cost, labor, and landscape of scholarly communication.

Here’s Barbara Fister with a recent report on what those at university presses have to say about the state of developmental editing:

I assumed that editorial work was a massive time commitment for university press editors, but the people I talked to said manuscripts need to be very nearly ready for publication these days; most editors don’t have the time for developmental or line editing. Authors increasingly need to get that work done themselves, either through writing groups or by hiring their own editors. Authors may also have to pitch in to pay for indexing, an important feature of scholarly monographs. Publishers at our discussion were not convinced that copy editing was worth the cost; the more ready a book is to go to print, the better. Design was once a standard function, but increasingly designs are templates that can be applied to any number of books. In general, work done on books once acquired seems to play a much smaller role than identifying authors to publish and then helping an audience discover the published book.

This jibes with my view of the situation: the world of fussy, behind-the-scenes editing that Zach treasures is in decline because of its costs, which were once masked by less-lean library purchasing budgets that created surpluses for presses which could be devoted to greater fussing. (Not worth getting into here, but it’s been many years since I experienced any decent developmental editing with my books or articles at presses or journals—please agree or contradict me by adding your experiences in the comments.) Worse, with additional cost-cutting on the horizon, I suspect that Zach’s ideal form of a paid, dedicated editor is unsustainable. (The sciences seem to have already figured this out; the most successful recent publications are venues like PLoS ONE and its clones from commercial publishers, which merely check for technical competency rather than content quality, and rely on the community of scientists to determine that quality.)

But let me agree with Zach that developmental editing is useful in history and the humanities. Where will it come from in the future? Zach and others believe that the only possible system is the system we know, with a dedicated editor paid for by publication gating fees. Here is where we diverge. If we look at the total picture of peer view and scholarly communication—not just in these sad days of recession and cost-cutting, but in prior generations as well—most of the developmental editing has actually come from unpaid colleagues and peers in our discipline, who are willing to give our drafts a read, or listen to us give early versions of our ideas at conferences or over coffee. Developmental editing has always largely resided in the gift economy of the scholarly community. Indeed, Zach runs our Levine Seminar series at Mason, where faculty present drafts of articles or book chapters to each other, receiving helpful criticism.

Surfacing, supporting, and expanding that gift economy is one of the goals of PressForward. Although those in the digital humanities often point to big experiments in open review—Jack Dougherty and Kristen Nawrotzki’s Writing History in a Digital Age, for instance, recently received hundreds of high-quality comments—it’s also important to recognize the increasing frequency of more modest experiments on the web.

For instance, this summer, while working on an article on a fourteenth-century motet, the Oxford musicologist Elizabeth Eva Leach posted a draft to her blog for comment. She didn’t receive hundreds of comments, but some helpful colleagues interested in the subject matter read the draft carefully and wrote in suggestions for improvement. Those little moments happen every day on the open web, and I suppose where Zach and I disagree is in their value. I’ve seen some extraordinarily extensive comments that easily equal the comments of a dedicated editor, whereas Zach worries that without that editor’s dedication, some scholars will receive no feedback.

With PressForward, we are not only trying to aggregate and curate high-quality, vetted scholarly content; we are trying to aggregate the attention of scholars so we can point to pieces like Leach’s, which in turn will receive more in-depth commentary. My view, perhaps colored by six years of blogging, is that there are many intelligent voices out there prepared to provide criticism. And the more commenters, the wider the range of views and suggestions, as opposed to the voice of a lone editor.

In short, far from destroying what is good and true, open publication with a layer of review seems like an obvious and effective way to retain some measure of developmental editing in a changing world of scholarly communication.

The Ivory Tower and the Open Web: Burritos, Browsers, and Books

In the summer of 2007, Nate Silver decided to conduct a rigorous assessment of the inexpensive Mexican restaurants in his neighborhood, Chicago’s Wicker Park. Figuring that others might be interested in the results of his study, and that he might be able to use some feedback from an audience, he took his project online.

Silver had no prior experience in such an endeavor. By day he worked as a statistician and writer at Baseball Prospectus—an innovator, to be sure, having created a clever new standard for empirically measuring the value of players, an advanced form of the “sabermetrics” vividly described by Michael Lewis in Moneyball. ((Nate Silver, “Introducing PECOTA,” in Gary Huckabay, Chris Kahrl, Dave Pease et al., eds., Baseball Prospectus 2003 (Dulles, VA: Brassey’s Publishers, 2003): 507-514. Michael Lewis, Moneyball: The Art of Winning an Unfair Game (New York: W. W. Norton & Company, 2004).)) But Silver had no experience as a food critic, nor as a web developer.

In time, his appetite took care of the former and the open web took care of the latter. Silver knit together a variety of free services as the tapestry for his culinary project. He set up a blog, The Burrito Bracket, using Google’s free Blogger web application. Weekly posts consisted of his visits to local restaurants, and the scores (in jalapeños) he awarded in twelve categories.

Home page of Nate Silver’s Burrito Bracket
Ranking system (upper left quadrant)

Being a sports geek, he organized the posts as a series of contests between two restaurants. Satisfying his urge to replicate March Madness, he modified another free application from Google, generally intended to create financial or data spreadsheets, to produce the “bracket” of the blog’s title.

Google Spreadsheets used to create the competition bracket

Like many of the savviest users of the web, Silver started small and improved the site as he went along. For instance, he had started to keep a photographic record of his restaurant visits and decided to share this documentary evidence. So he enlisted the photo-sharing site Flickr, creating an off-the-rack archive to accompany his textual descriptions and numerical scores. On August 15, 2007, he added a map to the site, geolocating each restaurant as he went along and color-coding the winners and losers.

Flickr photo archive for The Burrito Bracket (flickr.com)
Silver’s Google Map of Chicago’s Wicker Park (shaded in purple) with the location of each Mexican restaurant pinpointed

Even with its do-it-yourself enthusiasm and the allure of carne asada, Silver had trouble attracting an audience. He took to Yelp, a popular site for reviewing restaurants to plug The Burrito Bracket, and even thought about creating a Super Burrito Bracket, to cover all of Chicago. ((Frequently Asked Questions, The Burrito Bracket, http://burritobracket.blogspot.com/2007/07/faq.html)) But eventually he abandoned the site following the climactic “Burrito Bowl I.”

With his web skills improved and a presidential election year approaching, Silver decided to try his mathematical approach on that subject instead—”an opportunity for a sort of Moneyball approach to politics,” as he would later put it. ((http://www.journalism.columbia.edu/system/documents/477/original/nate_silver.pdf)) Initially, and with a nod to his obsession with Mexican food, he posted his empirical analyses of politics under the chili-pepper pseudonym “Poblano,” on the liberal website Daily Kos, which hosts blogs for its engaged readers.

Then, in March 2008, Silver registered his own web domain, with a title that was simultaneously and appropriately mathematical and political: fivethirtyeight.com, a reference to the total number of electors in the United States electoral college. He launched the site with a slight one-paragraph post on a recent poll from South Dakota and a summary of other recent polling from around the nation. As with The Burrito Bracket it was a modest start, but one that was modular and extensible. Silver soon added maps and charts to bolster his text.

FiveThirtyEight two months after launch, in May 2008

Nate Silver’s real name and FiveThiryEight didn’t remain obscure for long. His mathematical modeling of the competition between Barack Obama and Hillary Clinton for the Democratic presidential nomination proved strikingly, almost creepily, accurate. Clear-eyed, well-written, statistically rigorous posts began to be passed from browsers to BlackBerries, from bloggers to political junkies to Beltway insiders. From those wired early subscribers to his site, Silver found an increasingly large audience of those looking for data-driven, deeply researched analysis rather than the conventional reporting that presented political forecasting as more art than science.

FiveThiryEight went from just 800 visitors a day in its first month to a daily audience of 600,000 by October 2008. ((Adam Sternbergh, The Spreadsheet Psychic, New York, Oct 12, 2008, http://nymag.com/news/features/51170/)) On election day, FiveThiryEight received a remarkable 3 
million 
visitors, more than most daily newspapers
. ((http://www.journalism.columbia.edu/system/documents/477/original/nate_silver.pdf))

All of this attention for a site that most media coverage still called, with a hint of deprecation, a “blog,” or “aggregator” of polls, despite Silver’s rather obvious, if latent, journalistic skills. (Indeed, one of his roads not taken had been an offer, straight out of college, to become an assistant at The Washington Post. ((http://www.journalism.columbia.edu/system/documents/477/original/nate_silver.pdf)) ) An article in the Colorado Daily on the emergent genre represented by FiveThirtyEight led with Ken Bickers, professor and chair of the political science department at the University of Colorado, saying that such sites were a new form of “quality blogs” (rather than, evidently, the uniformly second-rate blogs that had previously existed). The article then swerved into much more ominous territory, asking whether reading FiveThirtyEight and similar blogs was potentially dangerous, especially compared to the safe environs of the traditional newspaper. Surely these sites were superficial, and they very well might have a negative effect on their audience:

Mary Coussons-Read, a professor of psychology at CU Denver, says today’s quick turnaround of information helps to make it more compelling.

“Information travels so much more quickly,” she says. “(We expect) instant gratification. If people have a question, they want an answer.”

That real-time quality can bring with it the illusion that it’s possible to perceive a whole reality by accessing various bits of information.

“There’s this immediacy of the transfer of information that leads people to believe they’re seeing everything … and that they have an understanding of the meaning of it all,” she says.

And, Coussons-Read adds, there is pleasure in processing information.

“I sometimes feel like it’s almost a recreational activity and less of an information-gathering activity,” she says.

Is it addiction?

[Michele] Wolf says there is something addicting about all that data.

“I do feel some kind of high getting new information and being able to process it,” she says. “I’m also a rock climber. I think there are some characteristics that are shared. My addiction just happens to be information.”

While there’s no such mental-health diagnosis as political addiction, Jeanne White, chemical dependency counselor at Centennial Peaks Hospital in Louisville, says political information seeking could be considered an addictive process if it reaches an extreme. ((Cindy Sutter, “Hooked on information: Can political news really be addicting?” The Colorado Daily, November 3, 2008, http://www.coloradodaily.com/ci_13105998))

This stereotype of blogs as the locus of “information” rather than knowledge, of “recreation” rather than education, was—and is—a common one, despite the wide variety of blogs, including many with long-form, erudite writing. Perhaps in 2008 such a characterization of FiveThirtyEight was unsurprising given that Silver’s only other credits to date were the Player Empirical Comparison and Optimization Test Algorithm (PECOTA) and The Burrito Bracket. Clearly, however, here was an intelligent researcher who had set his mind on a new topic to write about, with a fresh, insightful approach to the material. All he needed was a way to disseminate his findings. His audience appreciated his extraordinarily clever methods—at heart, academic techniques—for cutting through the mythologies and inadequacies of standard political commentary. All they needed was a web browser to find him.

A few journalists saw past the prevailing bias against non-traditional outlets like FiveThirtyEight. In the spring of 2010, Nate Silver bumped into Gerald Marzorati, the editor of the New York Times Magazine, on a train platform in Boston. They struck up a conversation, which eventually turned into a discussion about how FiveThirtyEight might fit into the universe of the Times, which ultimately recognized the excellence of his work and wanted FiveThirtyEight to enhance their political reporting and commentary. That summer, a little more than two years after he had started FiveThirtyEight, Silver’s “blog” merged into the Times under a licensing deal. ((Nate Silver, “FiveThirtyEight to Partner with New York Times, http://www.fivethirtyeight.com/2010/06/fivethirtyeight-to-partner-with-new.html)) In less time than it takes for most students to earn a journalism degree, Silver had willed himself into writing for one of the world’s premier news outlets, taking a seat in the top tier of political analysis. A radically democratic medium had enabled him to do all of this, without the permission of any gatekeeper.

FiveThirtyEight on the New York Times website, 2010

* * *

The story of Nate Silver and FiveThirtyEight has many important lessons for academia, all stemming from the affordances of the open web. His efforts show the do-it-yourself nature of much of the most innovative work on the web, and how one can iterate toward perfection rather than publishing works in fully polished states. His tale underlines the principle that good is good, and that the web is extraordinarily proficient at finding and disseminating the best work, often through continual, post-publication, recursive review. FiveThirtyEight also shows the power of openness to foster that dissemination and the dialogue between author and audience. Finally, the open web enables and rewards unexpected uses and genres.

Undoubtedly it is true that the path from The Burrito Bracket to The New York Times may only be navigated by an exceptionally capable and smart individual. But the tools for replicating Silver’s work are just as open to anyone, and just as powerful. It was with that belief, and the desire to encourage other academics to take advantage of the open web, that Roy Rosenzweig and I wrote Digital History: A Guide to Gathering, Preserving, and Presenting the Past on the Web. ((Daniel J. Cohen and Roy Rosenzweig, Digital History: A Guide to Gathering, Preserving, and Presenting the Past on the Web (University of Pennsylvania Press, 2006).)) We knew that the web, although fifteen years old at the time, was still somewhat alien to many professors, graduate students, and even undergraduates (who might be proficient at texting but know nothing about HTML), and we wanted to make the medium more familiar and approachable.

What we did not anticipate was another kind of resistance to the web, based not on an unfamiliarity with the digital realm or on Luddism but on the remarkable inertia of traditional academic methods and genres—the more subtle and widespread biases that hinder the academy’s adoption of new media. These prejudices are less comical, and more deep-seated, than newspapers’ penchant for tales of internet addiction. This resistance has less to do with the tools of the web and more to do with the web’s culture. It was not enough for us to conclude Digital History by saying how wonderful the openness of the web was; for many academics, this openness was part of the problem, a sign that it might be like “playing tennis with the net down,” as my graduate school mentor worriedly wrote to me. ((http://www.dancohen.org/2010/11/11/frank-turner-on-the-future-of-peer-review/))

In some respects, this opposition to the maximal use of the web is understandable. Almost by definition, academics have gotten to where they are by playing a highly scripted game extremely well. That means understanding and following self-reinforcing rules for success. For instance, in history and the humanities at most universities in the United States, there is a vertically integrated industry of monographs, beginning with the dissertation in graduate school—a proto-monograph—followed by the revisions to that work and the publication of it as a book to get tenure, followed by a second book to reach full professor status. Although we are beginning to see a slight liberalization of rules surrounding dissertations—in some places dissertations could be a series of essays or have digital components—graduate students infer that they would best be served on the job market by a traditional, analog monograph.

We thus find ourselves in a situation, now more than two decades into the era of the web, where the use of the medium in academia is modest, at best. Most academic journals have moved online but simply mimic their print editions, providing PDF facsimiles for download and having none of the functionality common to websites, such as venues for discussion. They are also largely gated, resistant not only to access by the general public but also to the coin of the web realm: the link. Similarly, when the Association of American University Presses recently asked its members about their digital publishing strategies, the presses tellingly remained steadfast in their fixation on the monograph. All of the top responses were about print-on-demand and the electronic distribution and discovery of their list, with a mere footnote for a smattering of efforts to host “databases, wikis, or blogs.” ((Association of American University Presses, “Digital Publishing in the AAUP Community; Survey Report: Winter 2009-2010,” http://aaupnet.org/resources/reports/0910digitalsurvey.pdf, p. 2)) In other words, the AAUP members see themselves almost exclusively as book publishers, not as publishers of academic work in whatever form that may take. Surveys of faculty show comfort with decades-old software like word processors but an aversion to recent digital tools and methods. ((See, for example, Robert B. Townsend, “How Is New Media Reshaping the Work of Historians?”, Perspectives on History, November 2010, http://www.historians.org/Perspectives/issues/2010/1011/1011pro2.cfm)) The professoriate may be more liberal politically than the most latte-filled ZIP code in San Francisco, but we are an extraordinarily conservative bunch when in comes to the progression and presentation of our own work. We have done far less than we should have by this point in imagining and enacting what academic work and communication might look like if it was digital first.

To be sure, as William Gibson has famously proclaimed, “The future is already here—it’s just not very evenly distributed.” ((National Public Radio, “Talk of the Nation” radio program, 30 November 1999, timecode 11:55, http://discover.npr.org/features/feature.jhtml?wfId=1067220)) Almost immediately following the advent of the web, which came out of the realm of physics, physicists began using the Los Alamos National Laboratory preprint server (later renamed ArXiv and moved to arXiv.org) to distribute scholarship directly to each other. Blogging has taken hold in some precincts of the academy, such as law and economics, and many in those disciplines rely on web-only outlets such as the Social Science Research Network. The future has had more trouble reaching the humanities, and perhaps this book is aimed slightly more at that side of campus than the science quad. But even among the early adopters, a conservatism reigns. For instance, one of the most prominent academic bloggers, the economist Tyler Cowen, still recommends to students a very traditional path for their own work. ((“Tyler Cowen: Academic Publishing,” remarks at the Institute for Humane Studies Summer Research Fellowship weekend seminar, May 2011, http://vimeo.com/24124436)) And far from being preferred by a large majority of faculty, quests to open scholarship to the general public often meet with skepticism. ((Open access mandates have been tough sells on many campuses, passing only by slight majorities or failing entirely. For instance, such a mandate was voted down at the University of Maryland, with evidence of confusion and ambivalence. http://scholarlykitchen.sspnet.org/2009/04/28/umaryland-faculty-vote-no-oa/))

If Digital History was about the mechanisms for moving academic work online, this book is about how the digital-first culture of the web might become more widespread and acceptable to the professoriate and their students. It is, by necessity, slightly more polemical than Digital History, since it takes direct aim at the conservatism of the academy that twenty years of the web have laid bare. But the web and the academy are not doomed to an inevitable clash of cultures. Viewed properly, the open web is perfectly in line with the fundamental academic goals of research, sharing of knowledge, and meritocracy. This book—and it is a book rather than a blog or stream of tweets because pragmatically that is the best way to reach its intended audience of the hesitant rather than preaching to the online choir—looks at several core academic values and asks how we can best pursue them in a digital age.

First, it points to the critical academic ability to look at any genre without bias and asks whether we might be violating that principle with respect to the web. Upon reflection many of the best things we discover in scholarship are found by disregarding popularity and packaging, by approaching creative works without prejudice. We wouldn’t think much of the meandering novel Moby-Dick if Carl Van Doren hadn’t looked past decades of mixed reviews to find the genius in Melville’s writing. Art historians have similarly unearthed talented artists who did their work outside of the royal academies and the prominent schools of practice. As the unpretentious wine writer Alexis Lichine shrewdly said in the face of fancy labels and appeals to mythical “terroir”: “There is no substitute for pulling corks.” ((Quoted in Frank J. Prial, “Wine Talk,” New York Times, 17 August 1994, http://www.nytimes.com/1994/08/17/garden/wine-talk-983519.html.))

Good is good, no matter the venue of publication or what the crowd thinks. Scholars surely understand that on a deep level, yet many persist in the valuing venue and medium over the content itself. This is especially true at crucial moments, such as promotion and tenure. Surely we can reorient ourselves to our true core value—to honor creativity and quality—which will still guide us to many traditionally published works but will also allow us to consider works in some nontraditional venues such as new open access journals or articles written and posted on a personal website or institutional repository, or digital projects.

The genre of the blog has been especially cursed by this lack of open-mindedness from the academy. Chapter 1, “What is a Blog?”, looks at the history of the blog and blogging, the anatomy and culture of a genre that is in many ways most representative of the open web. Saddled with an early characterization as being the locus of inane, narcissistic writing, the blog has had trouble making real inroads in academia, even though it is an extraordinarily flexible form and the perfect venue for a great deal of academic work. The chapter highlights some of the best examples of academic blogging and how they shape and advance arguments in a field. We can be more creative in thinking about the role of the blog within the academy, as a venue for communicating our work to colleagues as well as to a lay audience beyond the ivory tower.

This academic prejudice against the blog extends to other genres that have proliferated on the open web. Chapter 2, “Genres and the Open Web,” examines the incredible variety of those new forms, and how, with a careful eye, we might be able to import some of them profitably into the academy. Some of these genres, like the wiki, are well-known (thanks to Wikipedia, which academics have come to accept begrudgingly in the last five years). Other genres are rarer but take maximal advantage of the latitude of the open web: its malleability and interactivity. Rather than imposing the genres we know on the web—as we do when we post PDFs of print-first journal articles—we would do well to understand and adopt the web’s native genres, where helpful to scholarly pursuits.

But what of our academic interest in validity and excellence, enshrined in our peer review system? Chapter 3, “Good is Good,” examines the fundamental requirements of any such system: the necessity of highlighting only a minority of the total scholarly output, based on community standards, and of disseminating that minority of work to communities of thought and practice. The chapter compares print-age forms of vetting with native web forms of assessment and review, and proposes ways that digital methods can supplement—or even replace—our traditional modes of peer review.

“The Value, and Values, of Openness,” Chapter 4, broadly examines the nature of the web’s openness. Oddly, this openness is both the easiest trait of the web to understand and its most complex, once one begins to dig deeper. The web’s radical openness not only has led to calls for open access to academic work, which has complicated the traditional models of scholarly publishers and societies; it has also challenged our academic predisposition toward perfectionism—the desire to only publish in a “final” format, purged (as much as possible) of error. Critically, openness has also engendered unexpected uses of online materials—for instance, when Nate Silver refactored poll numbers from the raw data polling agencies posted.

Ultimately, openness is at the core of any academic model that can operate effectively on the web: it provides a way to disseminate our work easily, to assess what has been published, and to point to what’s good and valuable. Openness can naturally lead—indeed, is leading—to a fully functional shadow academic system for scholarly research and communication that exists beyond the more restrictive and inflexible structures of the past.

Video: The Ivory Tower and the Open Web

Here’s the video of my plenary talk “The Ivory Tower and the Open Web,” given at the Coalition for Networked Information meeting in Washington in December, 2010. A general description of the talk:

The web is now over twenty years old, and there is no doubt that the academy has taken advantage of its tremendous potential for disseminating resources and scholarship. But a full accounting of the academic approach to the web shows that compared to the innovative vernacular forms that have flourished over the past two decades, we have been relatively meek in our use of the medium, often preferring to impose traditional ivory tower genres on the web rather than import the open web’s most successful models. For instance, we would rather digitize the journal we know than explore how blogs and social media might supplement or change our scholarly research and communication. What might happen if we reversed that flow and more wholeheartedly embraced the genres of the open web?

I hope the audience for this blog finds it worthy viewing. I enjoyed talking about burrito websites, Layer Tennis, aggregation and curation services, blog networks, Aaron Sorkin’s touchiness, scholarly uses of Twitter, and many other high- and low-brow topics all in one hour. (For some details in the images I put up on the screen, you might want to follow along with this PDF of the slides.) I’ll be expanding on the ideas in this talk in an upcoming book with the same title.

[youtube=http://www.youtube.com/watch?v=yeNjiuw-6gQ&w=480&h=385]

Frank Turner on the Future of Peer Review

As I mentioned in my memorial post for my mentor Frank Turner, we were having a deep discussion of the future of peer review when he suddenly passed away. I wish we could have finished this discussion; as with so many other things, he brought tremendous insight to the topic. Much of the discussion was about personal experiences with peer review that I can’t recount in this public space, but we also got into “strategic planning” for changing the peer review system.

Here are the powerful last few email messages I received from Frank, redacted of personal matters and some touchy subjects. I think all of us trying to reform the academy through digital means should heed his words.

On the practice of peer review (I limit my thoughts to the Humanities) I have several different and conflicting opinions. Numerous journals are really quite well edited in my experience…In theory and often in practice peer review is a good thing.

But the problems of peer review are also longstanding…Other journals have in the past and still do remain the proprietary reserves of academic cliques or worse. One of the problems of which peer review in the humanities is only one facet is the absence of any agreed-upon and widely accepted understanding of the professional procedures and expectations. (Some would say a lack of “professional ethics.”) This has been exacerbated by the vast expansion of the academy in the second half of the last century; the often undue research expectations put on faculty in institutions that cannot financially support significant research; the necessity of editors sending out all manuscripts no matter how clearly mediocre and or undeveloped and hence expanding peer review expectations…

As you and others think through the peer review process, I would hope that you would keep several things in mind. First, you will need to avoid the appearance of playing tennis with the net down. Groups of friends or overly like-minded folks producing journals or collections of essays may disperse various views but do not necessarily make for tough-minded scholarship. Second, the kind of new reviewing processes you and others are suggesting could provide the opportunity really to establish widely accepted understandings of procedures and expectations. Such would be a major new departure, and it could benefit from the input of the editors of genuinely respected journals. Third, and I will return to this point below on another topic, as journals come to be published online (and I think within five years or less entirely on-line), they should make available to readers the possibility of commenting on articles. Again, there would need to be some kind of template so the comments are not like those on Amazon. But what would emerge would be a kind of scholarly community of commentary, revision, and correction. Fourth, at the end of the day, however, a new, open, collective peer review process will still need to indicate that some work is stronger, more deeply researched, and more profoundly analyzed than other work. I happen to think one of the benefits of studying the various areas of the humanities is achieving the capacity to make judgments. The peer review of humanities scholarship should avoid at all costs the appearance and the reality of not being able to make judgments regarding quality…

Let me expand the purview of what you and others are seeking to accomplish.  The realm of peer reviewing of articles is really quite strong when compared to that of book reviewing in journals. Books reviews are published with essentially no peer review, little or no concern or indication of actual or potential conflict of interest, and little or no concern for factual correctness. Such reviews are then used across the country in promotion processes. Scholarly book reviewing stands in a near scandalous situation. Most people review books in order not to purchase them. Reviews tend to be quite brief and as I have indicated are generally unedited except for style. Many reviewers simply rehash the dust jacket. Your group could again add to your agenda the establishment of professional procedures and expectations regarding book reviewing. These would include all reviewers indicating any conflict of interests, e.g., having taught the author, friendship with the author, residing in the same academic department or institution with the author, having written or edited a similar or competing book, having published with the same press, or having some political, religious, or ideological point of view that informs their thinking. Furthermore, again with the establishment of almost entirely on-line journal publication, all authors could be permitted to comment on and correct reviews and other scholars could similarly comment on the review or the book reviewed…

Most of you who are looking toward new ways of peer reviewing are young or at the entry level of the profession. All of you have a clear interest in reforming the existing reviewing process. I hope you will add that to your agenda as well as the peer reviewing of journals.

Peer Review and the Most Influential Publications

Thanks to Josh Greenberg, I’ve been mulling over this fascinating paper I missed from last winter about the relative impact of science articles published in three different ways in the Proceedings of the National Academy of Sciences (PNAS). It speaks to the question of how important traditional peer review is, and how we might introduce other modes of scholarly communication and review.

PNAS now allows for three very different modes of article submission:

The majority of papers published in PNAS are submitted directly to the journal and follow the standard peer review process. The editorial board appoints an editor for each Direct submission, who then solicits reviewers. During the review process the authors are blinded to the identities of both the editor and the referees. PNAS refers to this publication method as “Track II”. In addition to the direct submission track, members of the National Academy of Sciences (NAS) are allowed to “Communicate” up to two papers per year for other authors. Here, authors send their paper to the NAS member, who then procures reviews from at least two other researchers and submits the paper and reviews to the PNAS editorial board for approval. As with Direct submissions, authors of Communicated papers are at least in theory blinded to the identity of their reviewers, but not to the identity of the editor. PNAS refers to this publication method as “Track I”. Lastly, NAS members are allowed to “Contribute” as many of their own papers per year as they wish. Here, NAS members choose their own referees, collect at least two reviews, and submit their paper along with the reviews to the PNAS editorial board. Peer review is no longer blind, as the authoring NAS member selects his or her own reviewers. PNAS refers to this publication method as “Track III”… Examining papers published in PNAS provides an opportunity to evaluate how these differences in the submission and peer review process within the same journal affect the impact of the papers finally published. The possibility that impact varies systematically across track has received a great deal of recent attention, particularly in light of the decision by PNAS to discontinue Track I. The citation analysis we now present provides a quantitative treatment of the quality of papers published through each track, a discussion which as hitherto been largely anecdotal in nature.

Here’s the eye-opening conclusion:

The analysis presented here clearly demonstrates variation in impact among papers published using different review processes at PNAS. We find that overall, papers authored by NAS member and Contributed to PNAS are cited significantly less than papers which are Direct submissions. Strikingly, however, we find that the 10% most cited Contributed papers receive significantly more citations than the 10% most cited Direct submissions. Thus the Contributed track seems to yield less influential papers on average, but is more likely produce truly exceptional papers. [emphasis mine]

I suspect this will hold true for many new kinds of scholarly communication that are liberated from traditional peer review. Due to their more open and freewheeling nature, these genres, like blogging, will undoubtedly contain much dreck, and thus be negatively stereotyped by many in the professoriate, who (as I have noted in this space) are inordinately conservative when in comes to scholarly communication. But in that sea of nontraditionally reviewed material will be many of the most creative and influential publications. I’m willing to bet this pattern will be even more pronounced in the humanities, where traditional peer review is particularly adept at homogenizing scholarly work.

Just a thought for Open Access Week.