Getting started with IBM Watson

Cognitive computing is a new type of software that combines machine learning and natural language processing techniques in order to solve complex problems. IBM Watson is a revolutionary system implementing the cognitive computing paradigm. In this tutorial we will show how to create a “Question and Answer” demo Java web App that uses the IBM Watson API. The first section introduces IBM Watson and the technologies behind it, the next one explains the architecture of the system and the last section contains the implementation details of the demo application.

IBM Watson

You may already know that in 2011 IBM Watson competed on Jeopardy! against two former winners and won the first place. This was a great achievement in the field of artificial intelligence and a validation that cognitive computing is ready to be used in real life applications. Since then the system evolved continually and is now available also for developers to integrate it in third party Apps. On IBM Bluemix there are a series of services which can be used to connect to Watson Developer Colud. Here are a few examples:

Concept Expansion service – analyses text and interprets its meaning based on usage in other similar contexts. For example, it could interpret “The Big Apple” as meaning “New York City”. It can be used to create a dictionary of related words and concepts so that euphemisms, colloquialisms, or otherwise unclear phrases can be better understood and analyzed. Live demo
Language Identification service – detects the language in which text is written. This helps inform next steps such as translation, voice to text, or direct analysis. The service can be used in tandem with the Machine Translation service. Today, the service can identify 25 languages – Arabic; Chinese (Simplified); Chinese (Traditional); Cyrillic; Danish; Dutch; English; Farsi; Finnish; French; German; Greek; Hebrew; Hindi; Icelandic; Italian; Japanese; Korean; Norwegian (Bokmal); Norwegian (Nynorsk); Portuguese; Spanish; Swedish; Turkish; Urdu. Live demo
Machine Translation service – converts text input in one language into a destination language for the end user. Translation is available among English, Brazilian Portuguese, Spanish, French, and Arabic. Live demo
Message Resonance service – analyzes draft content and scores how well it is likely to be received by a specific target audience. This analysis is based on content that’s been written by the target audience itself such as fans of a specific sports team or new parents. Today, analysis can be done against people active in cloud computing or discussions but future versions will let users provide their own community data. Live demo
Relationship Extraction service – parses sentences into their various components and detects relationships between the components. It can process new terms (like people’s names in a news feed) it has never analyzed before through contextual analysis. Sentence components include parts of speech (noun, verb, adjective, conjunction, etc.) and functions (subjects, objects, predicates, etc.). The service maps the relationships between the components so that users or analytics engines can more easily understand the meaning of individual sentences and documents. Live demo
User Modeling service – uses linguistic analytics to extract a set of personality and social traits from the way a person communicates. The service can analyze any communication the user makes available such as their text messages, tweets, posts, email, and more. Users of the service can understand, connect, and communicate with people on a more personally tailored level by analyzing personality and social traits. Live demo
Question and Answer service – interprets and answers user questions directly based on primary data sources (brochures, web pages, manuals, records, etc.) that have been selected and gathered into a body of data or ‘corpus’. The service returns candidate responses with associated confidence levels and links to supporting evidence. The current data corpora on IBM Bluemix focuses on the Travel and Healthcare industries.

In this tutorial we will try the “Question and Answer” service. As previously mentioned the system was trained using sample data from Travel and Healthcare, so our demo will be able to answer questions expressed in natural language from this two fields. To have Watson learn a custom field you need access to Watson Developer Cloud Ent erprise (PDF file) or to the Watson Ecosystem program. Here you will be using a web tool called Watson Experience Manager to ingest training data to the system.

Under the hood, IBM Watson uses IBM’s DeepQA software, the Apache UIMA (Unstructured Information Management Architecture) framework, Apache OpenNLP for natural language processing and OpenCyc for the conceptual logic reasoning.

If you need to know a little more about Apache OpenNLP you can read our article on this subject: Getting started with Apache OpenNLP.

“Question and Answer” demo application architecture

To use IBM Watson we will create a Java web application which will be the frontend for the Watson API calls. Between our application and IBM Watson we will have a RESTfull Web Service. The application and the service will be hosted on IBM Bluemix.

The source code will be hosted on the Jazz Hub server. We will need the project on the development machine in order to submit the demo application code. The submission of the App to IBM Bluemix will be done using the Cloud Foundry command line tools.

The client can use a web browser to access IBM Watson “Question and Answer” demo application. After receiving a plain text question, the system will semantically analyze its content, search the most relevant answers in the corpus and return the them to the user.

Application architecture

Demo Java web App

Prerequisites:

Linux machine
IBM Bluemix account
Apache ANT, used to compile the Web App. For installation see Installing Apache Ant
Cloud Foundry command line tool, used to submit the Web App to IBM Bluemix. For installation see Installing CF command line tool
Git command line tool, used to manage the project code. For installation see Installing Git command line tool

Steps needed to create your own instance of IBM Watson:

1. Login to your IBM Bluemix console.

2. Create a liberty profile: go to Dashboard, click Create an App. Under Runtimes select Liberty for Java, enter the App name and click Create. You should now have a running instance of a Java Liberty profile available at the address specified during the creation phase. In our case it was http://technobium.mybluemix.net.

3. Add a service: click Add a Service under the newly created Java Liberty profile. From the Watson section, choose “Question and Answer”. In the pop-up enter the name of the new service. In our case “technobiumQAService”. After this pop-pup a new one will ask if you want to re-stage the App in order to use the new service. Click OK and the App will be restarted.

4. Clone the project to your computer: click Add Git to add this project to a GIT repository on Jazz Hub. If you don’t have an account, you will be assisted to create one. Accept and continue when you are asked to “Populate the repository with the starter application package and enable build and deploy.” At the end you should have a Git URL. Use the “git clone” command and this URL in order to clone the project on the local machine. Delete all the files in this folder – you will copy a sample Java project here as you will see in the next step.

5. Download the sample Java project qa-java-sample.zip, which calls the “Question and Answer” service. Extract the archive, copy the entire content inside the qa-java.git folder into the project you cloned and cleaned in the previous step. Edit the manifest.yml file and replace the name of the service and the name of the App to reflect the your service and App. In our example the file looks like this:

applications:
- services:
  - technobiumQAService
  name: technobium
  path: webApp.war
  memory: 512M

applications:

- services:

- technobiumQAService

name: technobium

path: webApp.war

memory: 512M

6. Build the Java web App project using the following Ant commands:

ant clean
ant build

1 2	ant clean ant build

7. Connect to IBM Bluemix API and submit the web App:

cf api https://api.ng.bluemix.net
cf login -u your-bluemix-username
cf push your-App-name

cf api https://api.ng.bluemix.net

cf login -u your-bluemix-username

cf push your-App-name

8. You can start using your App. Our Demo App for instance has the following URL: http://technobium.mybluemix.net

The Java demo web App contains a Servlet which handles the actual call to the “Question and Answer” service:

protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
		logger.info("doPost");

		String question = req.getParameter("questionText");
		String dataset = req.getParameter("dataset");

		//create the { 'question' : {
		//	'questionText:'...',
		//  'evidenceRequest': { 'items': 5} } json as requested by the service
		JSONObject questionJson = new JSONObject();
		questionJson.put("questionText",question);
		JSONObject evidenceRequest = new JSONObject();
		evidenceRequest.put("items",5);
		questionJson.put("evidenceRequest",evidenceRequest);

		JSONObject postData = new JSONObject();
    	postData.put("question",questionJson);

    	try {
    		Executor executor = Executor.newInstance().auth(username, password);
    		URI serviceURI = new URI(baseURL+ "/v1/question/"+dataset).normalize();
    		String answersJson = executor.execute(Request.Post(serviceURI)
			    .addHeader("Accept", "application/json")
			    .addHeader("X-SyncTimeout", "30")
			    .bodyString(postData.toString(), ContentType.APPLICATION_JSON)
			    ).returnContent().asString();

			List<Map<String, String>> answers = formatAnswers(answersJson);

			//Send question and answers to index.jsp
			req.setAttribute("answers", answers);
			req.setAttribute("questionText", question);
			req.setAttribute("dataset", dataset);

		} catch (Exception e) {
			// Log something and return an error message
			logger.log(Level.SEVERE, "got error: "+e.getMessage(), e);
			req.setAttribute("error", e.getMessage());
		}

    	req.getRequestDispatcher("/index.jsp").forward(req, resp);
	}

protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {

logger.info("doPost");

String question = req.getParameter("questionText");

String dataset = req.getParameter("dataset");

//create the { 'question' : {

// 'questionText:'...',

// 'evidenceRequest': { 'items': 5} } json as requested by the service

JSONObject questionJson = new JSONObject();

questionJson.put("questionText",question);

JSONObject evidenceRequest = new JSONObject();

evidenceRequest.put("items",5);

questionJson.put("evidenceRequest",evidenceRequest);

JSONObject postData = new JSONObject();

postData.put("question",questionJson);

try {

Executor executor = Executor.newInstance().auth(username, password);

URI serviceURI = new URI(baseURL+ "/v1/question/"+dataset).normalize();

String answersJson = executor.execute(Request.Post(serviceURI)

.addHeader("Accept", "application/json")

.addHeader("X-SyncTimeout", "30")

.bodyString(postData.toString(), ContentType.APPLICATION_JSON)

).returnContent().asString();

List<Map<String, String>> answers = formatAnswers(answersJson);

//Send question and answers to index.jsp

req.setAttribute("answers", answers);

req.setAttribute("questionText", question);

req.setAttribute("dataset", dataset);

} catch (Exception e) {

// Log something and return an error message

logger.log(Level.SEVERE, "got error: "+e.getMessage(), e);

req.setAttribute("error", e.getMessage());

}

req.getRequestDispatcher("/index.jsp").forward(req, resp);

}

Conclusion

“Question and Answer” demo is a simple and interesting way to gain a first impression about the power of IBM Watson. You can take a look also at the other cognitive computing services offered on the IBM Bluemix platform. Get to know them and use them to create innovative applications.