You are on page 1of 1

Only Apache Spark cannot help you to build a chat bot end to end !

Apache Spark is mainly used for data processing over huge data sets and also widely
used in executing ML models. There are various sub frameworks in spark such as Spark
ML, Spark Streaming etc.

Now a chat bot essentially consists of a few components if you are building it from
scratch,

1. a UI,
2. a back end which handles the requests from the UI,
3. a data store which stores contents based on which the bot responds. the
backend generally does real time intent extraction from the chat which the user
provides as input and looks up and creates an appropriate response either by
looking up a knowledge base (which generally is pre populated) or creates an
aggregation of contents from the various knowledge bases.
4. The model building, the knowledge creation and knowledge extractions can
use some parts of spark streaming, but do know that the model building and
creation needs to have happened before they are pushed into production.
You may want to consider the following technologies and approaches for building your
chat bot if you are building from scratch. One important aspect is to make sure that the
chat bot is real time, so considerations on NFR is critical.

1. An UI framework such as Angular JS or React JS


2. A backend which connects the UI, usually you can have this as a web app,
running on Java or python. The backend code generally does the intent
extraction, classification, search from the knowledge base. You can think about
the following open source technologies for these ( open NLP, Lucene, Elastic
Search etc)
3. A backend DB, where the knowledge base is stored, usually a NOSQL DB
such as MongoDB.
4. You should also consider making your application event driven, so make use
of a messaging framework, such as Kafka. The data processing can happen with
Spark Streaming. Always remember, if you have a very large data set, then you
can really take advantage of Spark, but for limited data sets Spark may be an
overkill.
You can also explore several out of box solutions which provide chat bot capabilities
such as How to build a chatbot - IBM Watson

Hope this helps !

You might also like