You are on page 1of 2

God Does Not Play Dice with Social Sentiments

WALID S. SABA

I previously wrote an article [1] expressing my opinion on the many claims that have been made in

recent years by a flood of so-called “Sentiment Analysis” software products. My main argument was that sentiment analysis is even a much more difficult and challenging problem than that of building computer programs that can understand ordinary spoken language. Since we do not, as of yet, have a solution to the latter, we certainly do not have anything that can decipher sentiment from ordinary spoken language, and what these system do is just slightly better than flipping a coin. I outlined in that article that the state-of-the-art in natural language processing (NLP) is still not at the point where it can deal with metonymy, metaphor, sarcasm, irony, non-literal meaning, etc. The article received some good comments and some that essentially questioned the frequency of such phenomena in ordinary spoken language. In a follow-up article [2] I replied to these concerns showing some data that illustrates that metonymy, metaphor, sarcasm, irony, non-literal meaning, etc. are not rare in ordinary language use, but are the norm. Given that these phenomena constitute more than 50% of language use, and given that the best accuracy any statistical approach has claimed is around 80%, then the accuracy of any sentiment analysis software is less than that of flipping a coin, as has also been reported by a formal study [3].

I can understand the excitement

and eagerness of some that want to achieve this monumental challenge, but things have gone out of hand. We now have many with little or no experience or even knowledge of the many challenges of building computational models for language analysis making claims that are beyond any credibility. In the long run, this is very damaging to the whole enterprise of NLP. When the truth comes out about the vacuous claims of these systems, it will negatively taint the entire NLP community and this will hinder progress in real NLP work that is based on solid scientific and logical foundations.

that is based on solid scientific and logical foundations. I do not have the space nor

I do not have the space nor the opportunity here to make a scientific argument on the vacuous and

false claims of such software systems (this will appear shortly elsewhere!). I will however briefly present here an “existential proof” of my claim. I have conducted a thorough experiment using several of the leading sentiment analysis software systems, using text written by both, those that vehemently advocate and defend the validity of such products, as well as those that question the entire enterprise.

Paradoxically, when presented with articles written by those that are very excited about automatic sentiment analysis, “sentiment analysis software” was assigned a negative sentiment, even when the input was content written by the vendors of such products! What is more bizarre, is that “sentiment analysis software” received a positive sentiment when presented with text that is very critical of such products (e.g., this article!) Negatives and positives were almost assigned at a whim, and many

software products did not agree on sentiment. The results were really what I expected: it was almost a flip of a coin.

I am not going to report here on the software products I used in my experiment, nor on the articles and authors that very strongly advocate such software products. I could if I need to, but I invite the defenders of such software products to try their articles on the software of their choice!

Little Knowledge is Dangerous

When I was a graduate student working on my PhD in AI/NLP, I was fortunate to have been seated in the reception dinner of ACL-95 (which was at held at MIT) on the same table that one of the most prominent researchers in AI/NLP was seated. He was at Bell Labs, back then, and he is now at the [one of?] largest software companies on the globe. I quietly and discretely asked:

“I have looked at this year’s conference proceedings, and I do not see any real “language understanding” work – no new semantic formalisms, theories, or models. All I see are Markov models, the infamous Bayesian formula, and large data/corpus analysis experiments with some tabulated results with good (of course!) precision and recall numbers at the end. What’s going on?” The answer to my long question was honest and clear: “We all gave up! We cannot figure out what the semantic rules of ordinary spoken language are, not to mention our complete ignorance of the pragmatic, epistemic and cognitive aspects of ordinary spoken language. So, we now crunch data and hope to discover some patterns!”

we now crunch data and hope to discover some patterns!” I can understand the frustration of

I can understand the frustration of researchers that constantly needed to produce publishable results. I can even understand the over eagerness of some hackers who clearly underestimate the real challenges in computational language understanding, but we are now at point where almost every day there is someone that claims they have some software that can quantify opinions, attitudes, feelings, and sentiments expressed in ordinary spoken language. This is damaging for all of us who “really” work in NLP.

1. Walid Saba (2012), Henry Kissinger vs. Sentiment Analysis, SEOJournal.

2. Walid Saba (2012), Henry Kissinger’s Sentiments are not an Exception, SEOJournal.

3. Brian Tarran (2010), Automated Sentiment Analysis Gives Poor Showing in Accuracy Test, research- live.com.