Professional Documents
Culture Documents
Apache Spark is mainly used for data processing over huge data sets and also widely
used in executing ML models. There are various sub frameworks in spark such as Spark
ML, Spark Streaming etc.
Now a chat bot essentially consists of a few components if you are building it from
scratch,
1. a UI,
2. a back end which handles the requests from the UI,
3. a data store which stores contents based on which the bot responds. the
backend generally does real time intent extraction from the chat which the user
provides as input and looks up and creates an appropriate response either by
looking up a knowledge base (which generally is pre populated) or creates an
aggregation of contents from the various knowledge bases.
4. The model building, the knowledge creation and knowledge extractions can
use some parts of spark streaming, but do know that the model building and
creation needs to have happened before they are pushed into production.
You may want to consider the following technologies and approaches for building your
chat bot if you are building from scratch. One important aspect is to make sure that the
chat bot is real time, so considerations on NFR is critical.