Prepare data

Import text

library(readr)
crowdata <- read_delim("crowdata.csv", ";", 
                       escape_double = FALSE, trim_ws = TRUE)

datatable(crowdata)


Focus on 1935 (702 periodicals) and retain only the relevant variables (skip the variables we won’t use for interpretation):

crow35 <- crowdata %>% filter(Year == "1935") %>% 
  select(Title_zh, Title_eng, Periodicity, Language, City_zh, City_py, Province, Established)
datatable(crow35)


We will use the “Title_zh” (title of periodical in Chinese) as the text variable for segmentation and further manipulations. So first we need to remove all “NAs” in the “Title_zh” variable, referring to foreign periodicals with no Chinese title:

# alternatively: 
crow35_zh <- crow35[complete.cases(crow35[ , 1]),]
crow35_zh <- crow35 %>% drop_na(Title_zh)


The dataset now contains 636 unique periodicals with Chinese titles.

Segmenting Chinese Text

Words and terms are the basic units of many computational text analysis methods, however Chinese characters are not “naturally” divided by whitespaces like some other languages such as English. A number of methods are developed to segment Chinese characters - here we try the widely used “jieba” segmenter on our sample texts.

Load the “jiebaR” and “stringR” (or tidyverse) packages:

library(jiebaR)
library(stringr)


Initialize an engine for word segmentation, use all the default settings, and try it on a simple sentence.

# initialize jiebaR worker
cutter <- worker()

# test the worker
cutter["今天的天氣真好"]
## [1] "今天" "的"   "天氣" "真好"


We then define a function called seg_x by which we segment the texts stored in the “Title_zh” variable of the data crow35 and save them as a new variable of crow35 called “Title_zh.seg”:

# define the function of segmenting
seg_x <- function(x) {str_c(cutter[x], collapse = " ")}

# apply the function to each document (row of Title_zh)
x.out <- sapply(crow35_zh$Title_zh, seg_x, USE.NAMES = FALSE)

# attach the segmented text back to the data frame
crow35_zh$Title_zh.seg <- x.out

# view the first 6 rows of the data frame

crow35_zh %>% 
  select(Title_zh, Title_zh.seg) %>% 
  head()


The dataset now contains an additional column with the segmented text. Note that the segmentation is neither perfect nor consistent (for instance, 申報 and 南中報 are maintained as compounds, whereas 新聞 報 is properly segmented). The tokenizer does not consider 報 as a segmenting unit. In order to improve the tokenizer, it will be necessary to identify all problematic cases and create a customized list of characters to be segmented on the basis of this list.

Create corpus and document-term/feature-matrix

With the texts segmented by whitespaces, we can move on to create corpus and document-term/feature-matrix (DTM/DFM) that are often used for further text analysis. Here we use functions of the quanteda package to create corpus and DFMs, so does to explore and visualize the texts. quanteda is an R package for managing and analyzing text data; it provides tools for corpus management, natural language processing, document-feature-matrix analysis and more.

library(quanteda)

We create a corpus from the texts stored in the Title_zh.seg variable using the corpus() function. We also tokenize the texts using tokens() and construct a document-feature-matrix using dfm(). Note “fasterword” is specified so that the texts are tokenized by whitespaces. We can then view the most frequent terms/features in this set of texts using topfeatures(). The quanteda package also offers a function textplot_wordcloud() by which you can easily plot a wordcloud from DFMs.

# create corpus
crowzhcorpus <- corpus(crow35_zh$Title_zh.seg)
# summarize the corpus object
summary(crowzhcorpus, showmeta = TRUE, 5)
# see the text in the five first documents of lcorpus
texts(crowzhcorpus)[1:5]
##       text1       text2       text3       text4       text5 
##   "新聞 報"      "申報" "時事 新報"      "時報"   "益世 報"
# create dfm with "terms/features" spliting by whitespaces;
# ie, preserve what has done for segmenting by jiebaR

# tokenize:"tokens" from doc 1, split by whitespaces
tokens(crowzhcorpus, what = "fasterword")[1:5]
## Tokens consisting of 5 documents.
## text1 :
## [1] "新聞" "報"  
## 
## text2 :
## [1] "申報"
## 
## text3 :
## [1] "時事" "新報"
## 
## text4 :
## [1] "時報"
## 
## text5 :
## [1] "益世" "報"
# tokenize and create document-feature-matrix
ltokens <- tokens(crow35_zh$Title_zh.seg, what = "fasterword")
ldfm <- dfm(ltokens)

# a dfm with 636 documents and 517 features
ldfm
## Document-feature matrix of: 636 documents, 517 features (99.6% sparse).
##        features
## docs    新聞 報 申報 時事 新報 時報 益世 南中報 電聲 電影
##   text1    1  1    0    0    0    0    0      0    0    0
##   text2    0  0    1    0    0    0    0      0    0    0
##   text3    0  0    0    1    1    0    0      0    0    0
##   text4    0  0    0    0    0    1    0      0    0    0
##   text5    0  1    0    0    0    0    1      0    0    0
##   text6    0  0    0    0    0    0    0      1    0    0
## [ reached max_ndoc ... 630 more documents, reached max_nfeat ... 507 more features ]
# list top 20 features
topfeatures(ldfm, 20)
##     日報       報       新 民國日報     晚報     商報     新聞     民報 
##      161       83       28       27       20       18       17       17 
##     週報     月刊     工商     時報     晨報     新報     天津     週刊 
##       14       14       13       12       12       11       11       10 
##     畫報     上海     雜誌       靑 
##       10       10        9        9
# plot wordcloud
par(family='Kaiti TC') # set Chinese font on Mac; you may not need to set font on Windows
textplot_wordcloud(ldfm, min_size = 1.5, min.freq=2, random.order=FALSE,
colors = RColorBrewer::brewer.pal(8,"Dark2"))

Frequency

Create feature frequency tab

# tabulate feature frequency
dfmtab <- textstat_frequency(ldfm)
head(dfmtab)


Sometimes you only care about, say, longer features/terms, use dfm_select() to choose those meet certain conditions, e.g., terms contain two or more words:

# select 2+ word features
ldfm2 <- dfm_select(ldfm, min_nchar = 2)
topfeatures(ldfm2, 20)
##     日報 民國日報     晚報     商報     新聞     民報     週報     月刊 
##      161       27       20       18       17       17       14       14 
##     工商     時報     晨報     新報     天津     週刊     畫報     上海 
##       13       12       12       11       11       10       10       10 
##     雜誌     國民     新民     中華 
##        9        8        8        7


Plot a wordcloud graph from the DFM containing those 2-or-more-word terms. Here we select terms appearing at least twive to plot and set 200 as the maximum number of terms to be included:

# plot wordcloud
par(family='Kaiti TC')
textplot_wordcloud(ldfm2, min_size = 1.5, min.freq=2, random.order=FALSE, max.words = 200,
rot.per = .25,
colors = RColorBrewer::brewer.pal(8,"Dark2"))
## Warning: min.freqcolorsmax.wordsrandom.orderrot.per is deprecated; use
## min_countcolormax_wordsrandom_orderrotation instead

Alternative Visualization

The function textstat_frequency() can tabulate all feature frequencies like we did above - we can also limit the frequencies to be tabulated and plot these selected features using ggplot() of the ggplot2 package:

# tabulate the top 10 features
textstat_frequency(ldfm2, n=10)
# plot freq. by rank of the most frequent 50 features
library(ggplot2)
theme_set(theme_minimal())
textstat_frequency(ldfm2, n = 50) %>%
ggplot(aes(x = rank, y = frequency)) +
geom_point() +
labs(x = "Frequency rank", y = "Term frequency")


We can also use dfm_weight to create a DFM representing weighted frequencies of the terms, for instance, a DFM with relative feature/term frequencies, i.e., the proportion of the feature counts of total feature counts:

# create dfm with relative term frequencies
dfmpct <- dfm_weight(ldfm2)

# plot relative term frequencies
textstat_frequency(dfmpct, n = 10) %>%
ggplot(aes(x = reorder(feature, -rank), y = frequency)) +
geom_bar(stat = "identity") + coord_flip() +
labs(x = "", y = "Relative Term Frequency") +
theme(text = element_text(family = 'STKaiti'))

Stopwords and dictionary

Concluding remarks