1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber

s Lauber | Low Code for Data Science | Dec, 2…

Open in app Sign up Sign in

Search Write

DATA STORIES | ARTIFICIAL INTELLIGENCE | KNIME ANALYTICS PLATFORM

Creating a Local LLM Vector Store

from PDFs with KNIME and GPT4All
A fully no-code solution!

Markus Lauber · Follow

Published in Low Code for Data Science · 7 min read · Dec 1, 2023

123

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 1/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

Photo by Christopher Burns on Unsplash.

KNIME is constantly adapting and integrating AI and Large Language Models

in its software. Starting with KNIME 5.2 it is possible to use local GPT4All
LLMs to create your own vector store from your own documents (like PDFs)
and interact with them on your local machine. I will talk about some findings
at the end of the article.

You can also read my initial article about: “KNIME, AI Extension and local
Large Language Models (LLM)”

I adapted a workflow from a KNIME AI Learnathon (them using ChatGPT) to

demonstrate the possibilities of local LLMs and Vector Stores:
“Create your own LLM Vector Store (from your documents) with GPT4All local
models”:

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 2/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

KNIME Workflow to create and use a GPT4All LLM and a local Vector Store from your own Document (PDF)
(https://forum.knime.com/t/gpt4all-embeddings/75594/5?u=mlauber71).

You will have to install GPT4All and the KNIME AI Extension. If you experience
problems please also refer to the “GPT4All Installation behind a Firewall” of
this article. Also as of Q1/2024 there is a bug when using the GPT4All offline
or behind a firewall (which this is all about) — you will have to make some
adaptions in some code.

A manual for a Coffee Machine

The document to explore is the user manual for a coffee machine, a 24 page
PDF. The file will be converted into a KNIME document type for text analytics
and be ‘fed’ to a (local) GPT4All model.

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 3/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

A) Create Knowledge Base

You convert your PDF after some cleaning into a local Vector Store with the
help of the FAISS Vector Store Creator node and a GPT4All embedding
model:

Create a Vector Store (https://hub.knime.com/-/spaces/-/~RgLTaML-8RjQVBfi/current-state/).

Step 1: Download the GPT4All models you want to use

Place your GPT4All models (.gguf extension) in the ../gpt4all_models/ folder.

[Download them at: https://gpt4all.io/index.html]. Please note which ones are
OK for commercial use!

Step 2: Select your GPT4All model

Select the model (.gguf) you want to use in the component. The name will
automatically be added to the text.

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 4/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

Select your GPT4All model in the component.

Step 3: Divide PDF text into sentences

Search, drag and drop Sentence Extractor node and execute on the column
“Document” from the PDF Parser node. This will split the document cell in
multiple rows: one row for each sentence. Then use a Row Filter node to
remove all sentences below 5 terms. You can experiment with additional text
preparations.

Step 4: Create the vector store

Search, drag and drop FAISS Vector Store Creator node, connect to the
Embeddings4All Connector and your string sections output. Execute the
node on the column with the strings to create the vector store. You can either
download a suitable embedding like (currently: all-MiniLM-L6-v2-f16.gguf) or
let the node do the work.

Step 5: Save the vector store

Save the vector store by adding a Model Writer node. To save it properly, you
can use a relative path and specify the name of the vector store such as
“vector_store.model”.

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 5/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

You have now successfully created a local Vector Store.

B) Use an LLM for a Completion Task via the Knowledge Base

Now you can ask questions in a ‘batch’ mode, that is you can send a lot of
them to the model using the same basic prompt:

Place a set of questions to the model and your vector store (https://hub.knime.com/-/spaces/-/~RgLTaML-
8RjQVBfi/current-state/).

Step 1: Load the selected GPT4All model

Load the model into the GPT4All Chat Model Connector. Here you can use
the Flow Variable from the left side.

Step 2: Adopt the Knowledge Base from the Vector Store

The Model Reader node reads from the workflow data area the vector store
you previously created in part A). The Vector Store Retriever will try to find 15
relevant documents to add to your prompt you will pose to the Large
Language Model:

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 6/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

Tell the Vector Store Retriever how many documents to find for your question.

Step3: Check and if necessary edit the prompt

You might want to edit some information and instructions around the
question itself you want to ask. This might be a good place to test the effects
of different prompts.

Do some prompt engineering and also add the additional informations (adapted from:
https://hub.knime.com/-/spaces/-/~WNe6bb2w2bemYBWE/current-state/).

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 7/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

Step 4: Connect to the Vector Store Retriever and the LLM Prompter

Run the questions thru the model and the Vector Store.

Drag in the Vector Store Retriever node and the LLM Prompter node,
in-between add a String Manipulation node for Prompt Engineering.

Step 5: Export Results to an Excel File

You can save the table via an Excel Writer node. Optionally you can compare
with a Table View node the answers by the LLM and the ones we imported
with the questions for reference.

Step 6: Inspect and compare the results.

C) ‘Live’ Chat with your GPT4All model using your Vector Store
In addition to asking your questions in a batch, I created a KNIME Component
where you could ‘live’ chat with the model and the information you provided
from the coffee machine manual. The component will take your question,
select suitable documents from the vector store and then give you the
answer:

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 8/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

A live chat with the model based on your PDF’s data (https://hub.knime.com/-/spaces/-/~RgLTaML-8RjQVBfi/current-
state/).

The initial task/role is only provided once, though you can change it. You can
also edit the prompt within the component if you want.

Step 1: Load Model and Vector Store like before

Load the model into the GPT4All Chat Model Connector. Here you can use
the Flow Variable from the left side where you selected the model.

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 9/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

Step 2: Select the initial Role for the prompt and the number of documents
to be searched

Besides your precise question, there should be a role being defined and some
additional instructions.

What happens inside the component is the answers are stored in a KNIME
table and are reloaded so you have your conversation stored and shown in
your chat window.

Process the chat inside the Component (https://hub.knime.com/-/spaces/-/~RgLTaML-8RjQVBfi/current-state/).

Currently the chat does *not* refer to items that already have been discussed
(like it would be with using a live connection to ChatGPT). But the upside is
that the conversation does happen just on your own machine without the
data being sent to the internet.

Even if you cannot process very large amounts of data (depending on the
power of your machine) you might be able to test prompts and see if your
vector store might work.

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 10/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

Just start playing with LLMs and KNIME :-)

Some Observations on using the local Chat

I would also like to share some initial observations while talking to the coffee
machine manual. They might (or might not) be relevant to other such
approaches:

Domain knowledge still is a thing. It seems to work best if you already

know about coffee and coffee machines and ask specific questions

The setup still can sometimes give false answers or make things up. It
seems to help to refer to the coffee machine (although this is already done
be the ‘wrapper’ around the question). It might be good to experiment
with the prompts

What does not work well is ‘negativ’ questions or things that are not in the
manual. I asked the setup if the machine also can make tea and the
answer was somewhat evasive. You can boil water but it did not say that
this might not be the best idea. The model seems to be reluctant to say
no. Maybe something to add to the prompt

You should focus on what you expect to be in the document. I am not sure
how well it works to combine it with the general content of the model

You should experiment with limiting or expanding the number of

(additional) documents provided. Sometimes a smaller number might be
better. Also the documents it can find and link will depend on the question
you ask (if it can find texts/documents linked to your input)

When creating KNIME document types (that you use to train the Vector
store) you can add additional metadata, like author, page number, title and

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 11/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

so on. It might be useful to try and add that to answers to have some
reference

So (again) please be aware that this is a language model trained to sum up

things and to some extent also to please the user; so be aware. It is not some
sort of general intelligence.

Note: this article has been edited to describe more precisely the use of
embedding models when creating vector stores.

If you enjoyed this article you can follow me on Medium

(https://medium.com/@mlxl) and on the KNIME forum
(https://forum.knime.com/u/mlauber71/summary) and hub
(https://hub.knime.com/mlauber71).

More KNIME content on Medium can also be found here:

Low Code for Data Science

A journal of articles written by (and for) the KNIME Community
around visual programming, data science algorithms &…
medium.com

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 12/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

Gpt4all Large Language Models Vector Store Knime Editors Pick

Written by Markus Lauber Follow

166 Followers · Writer for Low Code for Data Science

Senior Data Scientist working with KNIME, Python, R and Big Data Systems in the telco
industry

More from Markus Lauber and Low Code for Data Science

Markus Lauber in Low Code for Data Science Tasmay Pankaj Tibre… in Low Code for Data Scie…

KNIME, AI Extension and local Support Vector Machines (SVM):

Large Language Models (LLM) An Intuitive Explanation
Leverage the power of LLMs without data Everything you always wanted to know about
privacy concerns this powerful supervised ML algorithm

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 13/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

7 min read · Oct 11, 2023 17 min read · Jul 1, 2023

125 1 1.1K 7

Dennis Ganzaroli in Low Code for Data Science Markus Lauber

K-AI powered Python Code Orange3 — Machine Learning

Generation in KNIME and the… comparing different algorithms,…
Evaluate different Sports Rating Systems like Besides the great KNIME there are other low-
Elo Rating and TrueSkill with the help of K-AI,… code data mining and data analytics tools ou…

12 min read · Jan 8 6 min read · Mar 25, 2023

83 4

See all from Markus Lauber See all from Low Code for Data Science

Recommended from Medium

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 14/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

Markus Lauber in Low Code for Data Science Ankush k Singal in AI Advances

KNIME, AI Extension and local Building Intelligent Systems with

Large Language Models (LLM) OpenLLM and LlamaIndex using…
Leverage the power of LLMs without data Ankush k Singal
privacy concerns

7 min read · Oct 11, 2023 · 4 min read · 6 days ago

125 1 508 5

Lists

Natural Language Processing AI Regulation

1097 stories · 562 saves 6 stories · 278 saves

The New Chatbots: ChatGPT, ChatGPT

Bard, and Beyond 23 stories · 393 saves
12 stories · 270 saves

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 15/16
1/16/24, 7:25 PM Creating a Local LLM Vector Store from PDFs with KNIME and GPT4All | by Markus Lauber | Low Code for Data Science | Dec, 2…

Ignacio de Gregorio in Towards AI Fabio Matricardi in Stackademic

Is Mamba the End of ChatGPT As Tiny-Vicuna-1B is the lightweight

We Know It? champion of the Tiny Models
The Great New Question Command and Conquer: the smallest Vicuna
flavor is the Tiny Master of Instruction,…

· 8 min read · 4 days ago · 14 min read · 5 days ago

2.6K 25 864 3

ChenDataBytes Ferry Djaja

Natural Language Processing Create a Receipt Parsing Using

(NLP) Zero to Mastery Part I:… OCR and a Large Language Model
Unlocking Ten Key Concepts for NLP In this tutorial, I will go through how I leverage
Proficiency an OCR to capture data from receipts and…

10 min read · Dec 28, 2023 · 10 min read · Sep 29, 2023

3 93 1

See more recommendations

https://medium.com/low-code-for-advanced-data-science/creating-a-local-llm-vector-store-from-pdfs-with-knime-and-gpt4all-311bf61dd20e 16/16