This week we read several guidelines about how to evaluate digital scholarship projects. I would distill those into my own three primary criteria:
Content
First and foremost, the content must include sound scholarship and be up-to-date. Otherwise, it is merely wizbangery, not scholarship. All the snazzy design and multimedia features won’t make up for lousy content.
This is not meant to privilege new research findings over synthesis – digital collections that bring together resources in a single repository can also be valued as contributed to the field by making knowledge and information resources more accessible.
I would actually put design within this category, but not because I value form over content. To me design is a manifestation of information architecture and good content is well-structured. Each section builds on a previous section, knowledge builds on prior knowledge. Can the user follow the argument and themes and navigate easily through the site? If the site is difficult to get around, the thesis or interpretative point of view can be lost as well. When a portion of the site does not function as intended, it is like missing a transition sentence or paragraph, a footnote, or tabular reference.
Good content also considers the intended audience. This can be where digital humanities projects can struggle because they try to be all things to all people. This is especially true when the sponsoring organization is a museum or other institution that engages the public directly. They want to demonstrate deep learning, while at the same time attracting the public to the subject matter. Worse, they add a game or some other “fun” learning resource that is of questionable utility in attracting young people to the content. When the hosting organization is an academic institution they can more easily leave it to educators to turn your scholarship into learning material.
Appropriate Use of Technology
To some extent, appropriate use of technology is related to audience. Can you reach the intended audience through this medium? Is the project something that could not be done with other media – print, image, video? Does the use of visualizations improve the method of communication or merely look good? Is there too much text, where an image would do better? Where images are primary, does it use good quality images?
Are the methods appropriate to the question asked? When you have a hammer, everything looks like a nail. Just because your source material includes place names doesn’t mean you have to make a map. If what you are really after is how the places relate to one another, then a network analysis might be more appropriate.
This criteria also includes following web-accessibility guidelines and making resources reusable and interoperable. It would also include an appropriate sustainability plan. Digital resources are more ephemeral than we think. The digital-only periodical may lose funding and the site goes down, but a printed book remains on the library shelf. If the scholarship is to have long-term value, it should be built for the long-term.
Transparency
Digital projects almost always have multiple contributors. Are all contributors properly credited? Is metadata and copyright information included with all source material? Is the underlying data available and reusable? Does the metadata work well with others? Are the technology and design choices documented along the way?
What does this mean for traditional peer review?
Traditional peer review must change to meet these criteria. Dissertation committees and peered review journals will need to have representatives across disciplines to evaluate different pieces of the project. The peer review itself becomes a collaborative project – one of the values that Lisa Spiro put forth for digital humanities community. This might mean the breakdown or weakening of academic silos and create both challenges and opportunities for departmental funding allocations.
Whether producing digital scholarship or not, the academy needs to value collaboration and transparency more than it appears to. Digital tools make it easier to attribute contributions. Researchers can benefit from the transparency that comes with collaboration so that they can get credit for their contributions when reviewed for tenure or promotion.
Traditional peer review and fear of bucking the system are standing in the way of more digital scholarship.