Experiments with Speculative Fiction in HathiTrust

sf

I just shared a Jupyter notebook for working with three thousand speculative fiction novels using HathiTrust Research Center (HTRC) Analytics. The notebook uses “Extracted Features” rather than the full text of the novels: a data format devised by HathiTrust in order to enable text analysis on post-1926 books still under copyright protection.

Beginning with a print book that looks like this…

A page from the print edition of H. G. Wells's The First Men in the Moon.

…then scanning and OCRing it to grab its text…

THE FIRST MEN IN 
 THE MOON 

 MR. BEDFORD MEETS MR. CAVOR AT LYMPNE 

 As I sit down to write here amidst the 
 shadows of vine-leaves under the blue sky of 
 southern Italy, it comes to me with a certain 
 quality of astonishment that my participation 
 in these amazing adventures of Mr. Cavor 
 was, after all, the outcome of the purest acci- 
 dent. It might have been any one. I fell 
 into these things at a time when I thought 
 myself removed from the slightest possibility 
 of disturbing experiences. I had gone to 
 Lympne because I had imagined it the most 
 uneventful place in the world. " Here, at any 
 rate," said I, " I shall find peace and a chance 
 to work ! " 
 ' And this book is the sequel. So utterly at

…HTRC finally transforms that text into Extracted Features: a compressed .json file no longer readable by human eyes (“consumptive” reading), yet containing “quantitative abstractions of a book’s written content” that we can explore through text analysis (“non-consumptive” reading):

":1,"l":1,"r":1,"o":1},"tokenPosCount":{"rate":
{"NN":1},"accident":{"NN":1},"IN":{"IN":1},
"astonishment":{"NN":1},"down":{"RB":1},
"slightest":{"JJS":1},"quality":{"NN":1},"find":
{"VB":1},"disturbing":{"JJ":1},"AT":{"IN":1},
"vine-leaves":{"NNS":1},"any":{"DT":2},"southern":
{"JJ":1},"myself":{"PRP":1},"have":{"VB":1},"is":
{"VBZ":1},"MOON":{"NN":1},"said":{"VBD":1},
"Lympne":{"NNP":1},"sit":{"VBP":1},"thought":
{"VBD":1},".":{".":5},"adventures":{"NNS":1},
"blue":{"JJ":1},"THE":{"DT":2},"world":{"NN":1},
"fell":{"VBD":1},"CAVOR":{"NNP":1},"all":{"DT":1},
"book":{"NN":1},"had":{"VBD":2},"imagined":
{"VBN":1},"it":{"PRP":2},"!":{".":1},"A":{"DT":1},
"a":{"DT":3},"And":{"CC":1},"utterly":{"RB":1},
"sky":{"NN":1},"shadows":{"NNS":1},"outcome":
{"NN":1},"Here":{"RB":1},"because":{"IN":1},"Mr.":
{"NNP":1},"purest":{"JJS":1},"removed":{"VBD":1},
"certain":{"JJ":1},"comes":{"VBZ":1},"MEN":
{"NNP":1},"I":{"PRP":7},"LYMPNE":{"NNP":1},"work":
{"VB":1},"that":{"WDT":1},"possibility":{"NN":1},
"to":{"TO":4},"participation":{"NN":1},"MEETS":
{"VBZ":1},",":{",":6},"most":{"RBS":1},"here":
{"RB":1},"these":{"DT":2},"was":{"VBD":1},"at":
{"IN":3},"been":{"VBN":1},"FIRST":{"NNP":1},"'":
{"''":1},"my":{"PRP$":1}

Each element you see in the .json sample above is a feature, a “quantifiable marker of something measurable, a datum,” as Peter Organisciak and Boris Capitanu put it in their Programming Historian tutorial on text mining with HTRC. They continue:

A computer cannot understand the meaning of a sentence implicitly, but it can understand the counts of various words and word forms, or the presence or absence of stylistic markers, from which it can be trained to better understand text. Many text features are non-consumptive in that they don’t retain enough information to reconstruct the book text.

Extracted Features files allow researchers not only to count “tokens” (words) in each “volume” (published book), but also to filter by parts of speech, browse extensive bibliographic metadata, view quantitative information about each printed page in the dataset, use named entity recognition (NER) to identify people, places, or organizations in the text, graph these elements, and more.

You can follow the instructions in the Jupyter notebook over at my GitHub repo: Experiments with 20th-Century Speculative Fiction in HathiTrust. ,

—Princeton, August 2021