Comparing Sentiment Analysis REST API's

Skyttle Blog - A tool for evaluating Sentiment Analysis REST APIs

There is a growing number of Sentiment Analysis REST APIs out there, and the potential user is faced with a lot of choice. Accuracy of analysis is the most important factor, and the best way to see if an analyzer will perform well in the intended task, is to run different analyzers on a sample of your data, and compare their output with manually assigned sentiment labels.

To make it easier for potential users to run such experiments, we’ve released a small open-source project. The project implements clients to several Sentiment Analyzers: Alchemy, Bitext, Chatterbox, Datumbox, Repustate, Semantria, Skyttle, and Viralheat. As input, it takes a text file with short texts, each annotated as positive, negative or neutral, and outputs a spreadsheet where responses of each API are recorded, as well as an accuracy rate and an error rate calculated against the manual labels.

The project is available on github: https://github.com/skyttle/sentiment-evaluation. Once you clone/unpack it, you will need to install requirements:

...

SemantAPI - Semantapi.Robot

SemantAPI is a free, open source toolkit intended for a quick and easy comparison of the most popular NLP and sentiment analysis solutions on the market. The toolkit offers 2 independent analysis applications: SemantAPI.Robot and SemantAPI.Human. Both applications are written in C# and based on Microsoft’s .Net framework 3.5 platform.

Redistributable package of SemantAPI toolkit can be downloaded here.
The source code is available on GitHub here.

SemantAPI.Robot is an application that takes the specified source file and runs an analysis of every line therein, using the selected services.

The results are generated in a regular CSV file, with two columns per selected service:

• The “sentiment score” column contains float sentiment values provided by the target service, which can be used for precise sentiment analysis.
• The “sentiment polarity” value contains a verbal representation of the sentiment score, making it easy to read and understand at a glance.

The current version of the SemantAPI.Robot application supports the following NLP solutions:

• Semantria. Modern, fast-growing NLP solution based on Lexalytics’ Salience engine.
• AlchemyAPI. One of the world’s most popular NLP solutions.
• Chatterbox. Social technology engine that uses machine learning for sentiment analysis.
• Viralheat. Social media monitoring solution that offers a sentiment analysis API for 3rd-party integrators.
• Bitext. Semantic technologies solution with a sentiment analysis API that claims to have the highest accuracy on the market.

This is a day job kind of thing, one that I'm seeing more chatter and discussion about. In house we've licensed one library, and I've built a couple Proof of Concept app's with it. But I was doing so, kind of in a vacuum, not being able to compare the results against another platform. We, with this, I now can!

That and I just like the idea of these service and having the C# to access them all... :)