Some Shell Sed file treatments !

Some Shell Sed file treatments !

Preparation

Lets connect to Manus server.

ssh yug@[my ip] -p 822
passphrase : [press enter]
password : [ask me!]
cd ~/wikidata-scrapper/
tree

Lets merges the kids into a Big Brother file:

cat ./worker-00/zh_output_all.txt ./worker-01/zh_output_all.txt ./worker-02/zh_output_all.txt ./worker-03/zh_output_all.txt >> all.txt

How to clean up my .txt ?

Given lines more or less like :

< lexeme item="????" ar="" cs="" ja="" ko="" vi="" zh="" zh-hans="" zh-hant="" fr="" fralias=""> {Requests:288139, FR: got:50943, fail:237196, pc:0.17680008607}

Lets delete lines with specific string marking empty french result :

sed -i /fr=""/d all-test.txt

or (not tested):

A=fr="" sed -i "/$A/" all-test.txt

My file thus get 20% of its initial size ! Then, delete all the meta data :

sed -i s/{.*}// all.txt

Its now ~12% of the initial size, or 39MB. Replace or delete the unwanted sections :

#sed -i string type | target | replacement | global file.ext
sed -i s|ar="[^"]*" cs="[^"]*"| |g all-test.txt

How to download from private ssh ?

# telecharge [Host1:File1] [landing_place]
scp -P 822 yug@88.190.......:/home/yug/wikidata-scrapper/all.txt ~/home/yug/Documents/Temp/

download file now