원문정보
초록
영어
As non-formal language, texts containing dirty words are widespread in Web reviews. Due to their bad effects on users of communication, it is essential to perform automatic analysis on Chinese texts containing dirty word. In this paper, we first crawled over millions of evaluating sentences which contain a lot of dirty words from the Web. Second, we manually annotated 40 typical dirty words with weights. And then proposed a machine learning-based approach for collecting dirty word texts corpus. Overall, more than 6000 sentences were collected from the huge amount of Web reviews to form a corpus on Chinese texts containing dirty words. With the corpus, we present SVM (Support Vector Machine) and ME (Maximum Entropy) classifiers to automatic detect Chinese texts containing dirty words. Empirical studies demonstrate that the SVM and ME classifiers are both effective for this task and the recall and precision are both over 97%.
목차
1. Introduction
2. Related Work
3. Constructing the Corpus on Dirty Word Texts
3.1. Dirty Word Texts
3.2. Collecting Review Texts
3.3. Solution
3.4. A High-Precision Classifier
3.5. Preprocessing
3.6. Features Recommendation
3.7. Corpus Constructing
4. Automatic Detecting Chinese Dirty Word Texts
4.1. SVM and ME Classifiers
4.2. Feature Selection
4.3. Experiments
5 Conclusion
References