Old texts in a new world: Meaning production in the digital medium

Paper presented at Materiality of Knowledge in Chinese Thought: Past and Present, Yuelu Academy


Throughout history, technical innovations in the production and transmission of written materials have often had far-reaching long-term consequences for knowledge production – from the standardization of writing forms, to the development of dictionaries and encyclopedias, to the availability and spread of printing and copying technologies. In this paper, I focus on the ongoing impact of the most recent such development: digitization and increasing use of digital modes of interaction with premodern textual materials.

Since premodern Chinese documents first became available to scholars in digital form, the existence of digital texts has caused gradual but significant changes in mainstream scholarly workflows and expectations. Full-text repositories and digital libraries now make available in seconds to anyone on the planet premodern materials on a scale once impossible for anyone other than a determined emperor to obtain, while making similarly fantastic reductions in time and effort required to retrieve certain types of information. At the same time, even more dramatic changes have begun to take place as a consequence of digitization together with the ever-increasing sophistication and power of digital systems. Faced with larger volumes of material than any individual could ever expect to read – let alone claim detailed knowledge of – text mining and distant reading approaches offer the promise of gleaning useful information from exhaustive statistical analyses at scales not achievable through traditional means. Data-driven approaches – already well developed in other disciplines – similarly enable digital approaches to historical studies in which evidence can be systematically assembled at large enough scales to solidly ground statistical claims about broad historical and societal changes over time. This paper explores the development of these approaches, and the consequences for knowledge production in the digital age.

Posted in Chinese, Talks and conference papers | Comments Off

Chinese Text Project: a dynamic digital library of premodern Chinese

Paper published in Digital Scholarship in the Humanities


This article presents technical approaches and innovations in digital library design developed during the design and implementation of the Chinese Text Project, a widely-used, large-scale full-text digital library of premodern Chinese writing. By leveraging a combination of domain-optimized Optical Character Recognition, a purpose-designed crowdsourcing system, and an Application Programming Interface (API), this project simultaneously provides a sustainable transcription system, search interface and reading environment, as well as an extensible platform for transcribing and working with premodern Chinese textual materials. By means of the API, intentionally loosely integrated text mining tools are used to extend the platform, while also being reusable independently with materials from other sources and in other languages.

Full text on publisher site

Posted in Chinese, Digital Humanities | Comments Off

Digital Approaches to Text Reuse in the Early Chinese Corpus

Published in Journal of Chinese Literature and Culture 2018, 5(2) [Full paper]

Observed textual similarities between different pieces of writing are frequently cited by textual scholars as grounds for interpretative stances about the meaning of a passage and its authorship, authenticity, and accuracy. Historically, identifying occurrences of such similarities has been a matter of extensive knowledge and recall of the content and locations of passages contained within certain texts, together with painstaking manual comparison by examining printed copies, use of concordances, or more recently, appropriate use of full-text searchable database systems. The development of increasingly comprehensive and accurate digital corpora of early Chinese transmitted writing raises many opportunities to study these phenomena using more systematic digital techniques. These offer the promise of not only vast savings in time and labor but also new insights made possible only through exhaustive comparisons of types that would be entirely impractical without the use of computational methods.

This article investigates and contrasts unsupervised techniques for the identification of textual similarities in premodern Chinese works in general, and the classical corpus in particular, taking the text of the Mozi 墨子 as a concrete example. While specific examples are presented in detail to concretely demonstrate the utility and potential of the techniques discussed, all of the methods described are generally applicable to a wide range of materials. With this in mind, this article also introduces an open-access platform designed to help researchers quickly and easily explore these phenomena within those materials most relevant to their own work.

Posted in Chinese, Digital Humanities | Comments Off

Accessible Text Mining with Text Tools and the Chinese Text Project


  • Create a free account on ctext.org and log in.
  • Make sure to validate your e-mail address by opening the link the system sent you (if not, the link above will display a warning/reminder in red to do so).
  • Enter the API key “aas2019″ (without quotes) in the box labeled “API key”, and click “Save”.
  • [Optional] Install the “Text Tools” plugin into your ctext account.

Some parts of the “Practical introduction to ctext.org” and “Text Tools for ctext.org” will be demonstrated – please refer to the tutorials for step-by-step instructions.

Direct link to Text Tools: http://ctext.org/plugins/texttools/#help

Other suggested examples

In addition to the examples shown in the tutorials:

  • Try comparing the aggregate vocabulary of two texts (e.g. the 墨子 and 呂氏春秋) using the “Vectors” tab. Click “Toggle values” to display the heatmap, and try inspecting some of the comparisons.
  • Try the “Run PCA” link with these or other texts.
  • Try creating vectors that model only a specifically selected subset of vocabulary use. To do this, start by entering multiple search terms in the Regex tool (one per line) – one example would be grammatical particles such as 而, 也, 以, 乎, 之, 矣, 亦. From the “Summary” tab, click “Create vectors”, and then from the output choose “Run PCA”.
  • Using the regex tool with “Group rows by” set to “None” and “Extract groups” checked, try extracting data about biographies in the 宋史. You may want to start by using a small part of the text, e.g. 列傳第二十一 (ctp:ws281485). Example regex: (\w+),字(\w+),(\w+)人。
  • A few additional examples and instructions for using materials not written in classical Chinese are available on the SUTD workshop page.
Posted in Chinese, Digital Humanities, Talks and conference papers | Comments Off

Text Transformation API

Draft – This is a preliminary draft specification. Please note that some implementation details will change before publication. Last updated: 22 March 2019.


Transformations of textual data are important processes in many natural language processing and text analysis workflows. Examples include tokenization, lemmatization, and appending of part of speech tags, as well as many other (often language-specific) procedures. In this specification, a text transformation is any operation which takes as input a sequence of Unicode characters, and produces as output a sequence of Unicode characters. The Text Transformation API (TTA) defines a simple specification for how to negotiate, request, and deliver text transformations over HTTP.

A TTA server is a system which both: 1) publishes a TTA service manifest, and 2) provides or references at least one TTA transformation service endpoint.

Service manifest

A service manifest is a valid JSON file containing a list of transformation services. Each service is described using the following key-value pairs:

Key Value
endpoint The URL of the transformation service endpoint described by this entry.
languages A list of ISO 639-1 language codes to which the endpoint is relevant or recommended.
title A human readable description of the service the endpoint describes.

Transformation service endpoint

A transformation endpoint is a HTTP or HTTPS URL which accepts a string of text sent to it via the HTTP POST method using the “application/x-www-form-urlencoded” content type. The content of the string must be supplied in the “data” parameter of the request in UTF-8 encoding.

The response to any valid request must be a JSON file containing exactly one of the following key value pairs:

Key Value
output The contents of the “data” parameter transformed according to the service provided by the requested endpoint.
error A string explaining why the request failed.

Transformation client

A transformation client is any software which 1) requests TTA service manifests, specified by their URL; 2) provides a user with a means of viewing the “title” descriptions of the endpoints from any conformant TTA manifest, and 3) provides a user with a means of transforming texts using any conformant endpoint.


A non-normative example of a TTA service manifest (containing references to example TTA service endpoints) is: https://txt.ctext.org/services.pl

A non-normative example of a TTA client is accessible here.

Posted in Digital Humanities | Comments Off

SUTD Workshop

Materials from a workshop held as part of Working with different kinds of ‘text’ in the Digital Humanities at the Singapore University of Technology and Design.


We will follow parts of the “Practical introduction to ctext.org” and “Text Tools for ctext.org” tutorials with a few changes to use English language texts as well as Chinese ones, and a few new features of the beta version not yet included in the tutorials.

Link to Text Tools (beta version): http://ctext.org/plugins/texttoolsbeta/#help

Important note/reminder: For tools which have this option, we will use “Tokenize by character” set to “On” for the Chinese materials, and “Off” for the English ones.

Other suggested examples

As well as the examples shown in the tutorials:

  • Try some regexes on the English examples. A useful expression is likely to be “\w+” – any sequence of non-punctuation characters (intuitively, a word). Try as an example “the \w+”.
  • Using the “English_wordlist.txt” file as a list of regexes (just paste the contents of the file into the “Regex” box), generate vectors for the two Wizard of Oz stories. Run PCA on the results – you should see interesting differences between the two. Also try preprocessing the data by tokenizing and lowercasing the texts.
  • Try tokenizing one or modern Chinese documents [example].
  • Using the regex tool with “Group rows by” set to “None” and “Extract groups” checked, try extracting data about biographies in the 宋史. You may want to start by using a small part of the text, e.g. ctp:ws55241. Example regex: (\w+),字(\w+),(\w+)人。
Posted in Chinese, Digital Humanities | Comments Off

Large-scale Optical Character Recognition of Pre-modern Chinese Texts

This paper appears in International Journal of Buddhist Thought and Culture 28(2) (December 2018). [Full paper]


Optical character recognition (OCR) – the fully automated transcription of text appearing in a digitized image – offers transformative opportunities for the scholarly study of written materials produced prior to the digital age. Digitization, in the sense of photographic reproduction, is a largely straightforward, mechanical process, and one with significant value in its own right for purposes of preservation as well as access to rare materials. As a result, hundreds of millions of pages of pre-modern Chinese works have been digitized by libraries and academic institutions around the world – a significant portion of this increasingly being made freely available online.

To make use of this material efficiently, transcriptions of the textual content of these images are needed. Given the enormous volume of image data in existence – and its continual production as digitization continues – this task is only feasible if it can be fully automated: performed by software without manual intervention. Individually, reliable transcriptions produced by OCR offer enormous time savings to researchers, making it possible to efficiently navigate materials in ways not possible without digital transcription. In aggregate, however, these transcriptions make possible entirely new ways of exploring historical materials – making it possible to rapidly identify material that one suspects may exist somewhere, without knowing in advance where that might actually be. It is also a prerequisite also to virtually any type of statistical analysis of these materials – the potential utility of which continues to increase as a larger and larger proportion of the extant corpus is transcribed.

This paper introduces a procedure for OCR of pre-modern Chinese written materials, both printed and handwritten, describing the complete process from digitized image through to automated transcription and manual correction of remaining errors, with particular attention to issues arising in this domain. The process described has been applied to over 25 million pages of pre-modern Chinese works, and the paper also introduces the Chinese Text Project platform used to both make these results available to scholars as well as provide a distributed, crowdsourced mechanism for facilitating manual corrections at scale as well as further analysis of these materials.

Noise removal

Character pitch identification

Seal isolation

Posted in Chinese, Digital Humanities | Comments Off

EASTD 135: Text and Data in the Humanities

This course introduces students to key concepts and techniques fundamental to applying digital methods to the study of textual materials and other types of data in humanities subjects. The core topics covered are digital representations of data, ways of structuring and managing data, extracting data from textual materials, and data visualization and analysis. Concepts introduced in lecture sessions will be reinforced and applied concretely in particular contexts during corresponding practical sessions and take-home assignments.

No background in digital methods is assumed, however students are expected to have basic computing skills and access to a suitable laptop. Examples will be selected from a variety of subject domains within the humanities with the primary focus being on textual materials.


Week 1 (Jan 28, 30) – Introduction and motivation

  • Data and digital techniques in the humanities
  • Examples of data-driven approaches in humanities scholarship

Week 2 (Feb 4, 6) – Representation I

  • Fundamentals of digital representation of information
  • Basic types of data and their digital representations

Week 3 (Feb 11, 13) – Data and ontologies I

Week 4 (Feb 20) – Representation II

  • Research data management

Week 5 (Feb 25, 27) – Data and ontologies II

  • Databases and structured data

Week 6 (Mar 4, 6) – Data and ontologies III

  • Linked Open Data in the humanities

Week 7 (Mar 11, 13) – From text to data I

Week 8 (Mar 18, 20) – Visualization I

  • Charts and diagrams

Week 9 (Mar 25, 27) – Visualization II

  • Graphs, maps, and trees

Week 10 (Apr 1, 3) – From text to data II

  • Topic modeling

Week 11 (Apr 8, 10) – From text to data III

  • Part of speech tagging and parsing of natural languages

Week 12 (Apr 15, 17) – From text to data IV

  • Markup and annotation systems

Week 13 (Apr 22, 24) – Review

  • Review and discussion of project work

Week 14 (Apr 29, May 1) – Project presentations

  • Student projects presented in class

Posted in Courses, Digital Humanities | Comments Off

Networks of Text Reuse in Early Chinese Literature

Poster presented at Connected Past 2018.


The phenomenon of text reuse – syntactically and semantically similar fragments of text repeated apparently independently in multiple pieces of writing, and often in works purporting to be composed by entirely different authors – is extremely widespread in early Chinese literature. Such reuse is typically unattributed, and its existence is often revealed only through painstaking comparison with other pieces of potentially related writing. Computational methods have for the first time made feasible the comprehensive identification of such reuse throughout large corpora of material, and have thus made practical studies based on patterns of reuse which emerge at much larger scales than had previously been possible to consider.

This work uses network analysis to investigate patterns of text reuse in the early Chinese corpus and the relationship between these patterns and difficult questions of authorship attribution within these texts. Using detailed data on individual instances of text reuse created through an exhaustive automated study of the entire transmitted corpus of Chinese from the earliest transmitted works through to those dating prior to the end of the Han dynasty (220 AD), this study demonstrates the utility of network visualization and analysis in identifying and exploring patterns of text reuse which shed light on the authorship of these early materials.

Posted in Chinese, Digital Humanities, Talks and conference papers | Comments Off

Accessible digital text analysis for classical Chinese

Paper presented at Future Philologies: Digital Directions in Ancient World Text, Institute for the Study of the Ancient World, New York University, April 20 2018.


Despite a growing interest in digital humanities as a field of study and focus of specialization, significant barriers to the adoption of digital techniques remain within research and teaching in practice in many humanities disciplines. While an increasing number of humanities scholars have demonstrated willingness to invest time and effort in cultivating necessary technical skills, in practice many more are prevented from experimenting with digital methods due to perceived high barriers to entry. One approach to accelerate the adoption of digital techniques is to attempt to reduce the prerequisite technical skills required to apply techniques to research data in practice through the creation of platforms and tools able to bridge technical gaps for some of the most powerful and generally applicable use cases.

With this goal in mind, this talk introduces a suite of browser-based text analysis tools designed for pre-modern Chinese materials and intended to easily integrate into scholarly workflows, including in particular those common in Chinese literature, philosophy, and history departments. Major goals include accessibility of the tools themselves, as well as transparency of their working and ability to introspect the mechanisms underwriting the results and visualizations produced. By enabling rapid exploration of arbitrarily chosen textual materials while also providing insight into the algorithms used, these tools have pedagogical applications in addition to research uses, and are already in use teaching at several institutions.

Posted in Chinese, Digital Humanities, Talks and conference papers | Comments Off