Some corpus linguistic !

Some corpus linguistic !

Tonight! Lets make some corpus linguistic !

Corpus sources

Once again, my years of geekiness and friendlyness helps! From Wiktionary contributions > Source > Authors blog > polite message > I got a link to a recent study and its gathered data. They already extracted the content from OpenSubtitles.org, watch yourself :

http://opus.lingfil.uu.se/OpenSubtitles_v2.php [1]

dig in it, its simply great.

Counting and sorting

From there on I needed some guiding for treatments. After Edouard Lopez gave me some directions and keywords : AWK, SHELL rather than pure REGEX, I went ahead to write down clear, concise, focused questions on StackOverflow[2][3]

Given a multilingual .txt files such as:

But where is Esope the holly Bastard! But where is ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Console for Awk and Shell

For speed reasons, we will work in console. The first thing to do is thus to tell the console in which folder are your files :

cd /my/folder/withfiles/

Characters frequency (+sorted!)

$ grep -o S myfile.txt | awk {a[$1]++}END{for(k in a)print a[k],k} | sort > myoutput.txt

Space-separated Words frequency (+sorted!):

$ grep -o w* myfile.txt | awk {a[$1]++}END{for(k in a)print a[k],k} | sort > myoutput.txt

or all awk:

$ awk {a[$1]++}END{for(k in a)print a[k],k} RS=" | " myfile.txt | sort > myfileout.txt

On all .txt of a folder and its subfolders

find -iname *.txt -exec cat {} ; | grep -o w* | awk {a[$1]++}END{for(k in a)print a[k],k} | sort

Done! Big thanks to @Sudo_O !

Result

Looks like (for words):

Word frequency
1	?
1	?
1	Bastard
1	Esope
1	holly
1	the
1	?
1	?
1	?
1	?
2	?
2	But
2	is
2	where
2	?
2	?
3	?
4	?
4	?
5	?

References

• [1]: J�rg Tiedemann, 2009, News from OPUS - A Collection of Multilingual Parallel Corpora with Tools and Interfaces. In N. Nicolov and K. Bontcheva and G. Angelova and R. Mitkov (eds.) Recent Advances in Natural Language Processing (vol V), pages 237-248, John Benjamins, Amsterdam/Philadelphia
• [2]: Awk: Characters-frequency from one text file?
• [3]: Awk: How to work on multiple files.txt in folder and subfolders?

download file now