215124, 9:11 AM Using @ local LLM to Automate an Android device Mendhak / Code Using a local LLM to Automate an Android device While most well known Large Language Models (LLMs) are closed and behind paywalls, there exist open models such as LLaMa and its derivatives, available for free and private use. A thriving open-source community has built up around them, and projects like MLC and Ilama.cpp bring these LLMs to consumer devices such as phones and laptops. These projects have currently captured my attention; it's pretty fascinating to see an LLM running on low end hardware, and to imagine what possibilities this could open up in the future through this increased accessibility to the masses. Here's a video of llama.cpp running on my Pixel 6. Yes, it’s a hobbled model made for weaker hardware, and yes the speed isn’t great. But still It’s like having a personal, private information retrieval tool. ntpe:ticode menahak convautomate-andrld-with-ocalsnt ane215/24, 9:11 AM Using @ local LLM to Automate an Android device Llama2 running on Android Device (Pixel 6) | wanted to explore the potential of integrating an LLM into an automation workflow, just to see if it was possible. The conclusion is that it is somewhat possible, and in this example | am using it as a daily itinerary generator for the current location I'm in. hitps:icode,mendhak.comlautomate-android-with toca! 26ott aM ce ome ecg & coy caer Sen ene pol CECT The Setup On Android, the most widely-used automation frameworks are Tasker and Using @ local LLM to Automate an Android device rn eae Pea Ors Cree eM ema ta make your way to your accommodation and settle into the city life of Paris! Start Seal ye artery Pree iC u on a uci NTS Near eer nes ure eterna es Explore Montmartre & Sacré-Coeur Basilica Take a walk through the Se ee eared eee aint enc! the early 19th century! Visit the iconic Moulin Rouge cabaret or climb up to ee Reece Afterward, head over to Sacré-Coeur Cee Re teeny skyline while enjoying some peaceful RUC cu etc ae Discover Notre Dame Cathedral & Latin Ceres tues ae ere en eee Ree ea UOC an) Seren uired to make it sul Automate, both of which can work with Termux commands. This setup is highly practical and straightforward to work with. llama.cpp is a framework hitps cade mendhak.comfautomate-androldsath calm215/24, 9:11 AM Using @ local LLM to Automate an Android device to run simplified LLMs, and it can run on Android. Termux is a Linux virtual environment for Android, and that means it can execute Bash scripts. Automate, load StableBeluga) setup I'll go over how | set up llama.cpp, the Termux environment to run it, and the Automate app to invoke it. Building llama.cpp The llama.cpp README has pretty thorough instructions. Although its Android section tells you to build llama.cpp on the Android device itself, | found it easier to just build it on my computer and copy it over. Using Android Studio's SDK Tools, install the NDK and CMake. hitpssicode mendhak.comlautomate-android-with toca! 48215/24, 9:11 AM Using @ local LLM to Automate an Android device — Android Studio NDK and CMake You can then follow pretty much the same instructions as the README. Clone the llama.cpp repo, point $NDK at the NDK location, and build it: cd export hitpssicode mendhak.comlautomate-android-with toca! sn21824, 0:11 AM Using @ local LLM to Automate an Android eves This creates a main executable in the build-android/bin directory. We'll need to copy the executable over to the Android device, specifically into the Termux working space. For that we'll need to set up Termux and SSH. Termux and SSH Termux is a terminal emulator for Android, think of it as a Linux environment. This is where we'll be running the llama.cpp binary with the LLM. Start by installing Termux from F-Droid - this isn't a preference, the Google Play Store version has been deprecated. After installing Termux, | ran pkg upgrade to ensure the latest packages were available. Next is to set up an SSH server in Termux to allow connecting from your computer. This part is technically optional, but working over SSH is the easiest way to deal with lots of typing; an alternative would be to pair a Bluetooth keyboard with your Android phone but that still requires squinting and hunching. Following the steps, # In Termux: apt install openssh passwd # Change the password whoami # Make note of the username, For me it was u@_a301 sshd # Start the SSH server on port 8022 It’s a good idea to test connectivity from the computer, over its default port 8022, entering the password that was set above. This should output a list of files and exit ssh. hitpssicode mendhak.comlautomate-android-with toca! ene215/24, 2:11 AM Using @ local LLM to Automate an Android device # From computer: fest SSH Copy the binary over We can now use scp to copy the built binary over. | just copied it over to the home directory. # From computer hips cade menchak.comlautomate-androld-wthocal-tn! 76215124, 9:11 AM Using @ local LLM to Automate an Android device Download a model and run it To run main we'll need an actual LLM to interact with. LlaMa2 is well known, but | decided to go with a derivative called StableBeluga. llama.cpp requires models to be in a GGUF format, one for StableBeluga has been made available here. # From computer ssh u@_a3@1@192.168.50.66 -p 8022 # You are now in Termux # Test the binary -/main -h # Create a directory to download model files into mkdir -p models/7B/ # Install wget pkg install wget # Download the Stable Beluga 7B GGUF model into the directc wget https: //huggingface.co/TheBloke/StableBeluga-7B-GGUF/r > It takes a while to download the model, and we can now run our first test. Try some sentence completion. ./main --seed -1 --threads 4 --n_predict 3@ --model ./model > ntpe:ticode menahak convautomate-andrld-with-ocalsnt ane215/24, 9:11 AM You can also try providing a prompt and have an interactive session with the assistant. Ask it some questions, and say Goodbye to exit, or press Ctrl+C. Eire aso Nec a Rm ures maaan ras rc To return control without Er Rprnncranciertesty prorreathr rt Se a Prana com teanTes interactive chat session Set up Automate hips cade menchak.comlautomate-androld-wthocal-tn! one215724, 9:11 AM, Using @ local LLM to Automate an Android eves Automate is an automation framework app for Android; by coincidence it's published by a company called LlamaLab. Automate can interact with Termux in a few different ways but the simplest one is to use a plugin and grab an example workflow and just modify it. After installing Automate, go to Settings > Privileges, and enable the option Run commands in Termux environment. Install the Tasker plugin for Termux (Automate can work with Tasker plugins), and download the sample Run Termux Command With Tasker workflow. Automate should handle this link and the downloaded workflow becomes available in its list as Run Termux Command with Termux:Tasker Go ahead and create a test script as the sample needs, just to ensure it's working. Create the script: mkdir -p ~/.termux/tasker/ nano ~/.termux/tasker/test.sh With these contents: #!/data/data/com.termux/files/usr/bin/sh echo $1 Then save, and make it executable: chmod u+x ~/,.termux/tasker/test.sh ntpe:ticode menahak convautomate-andrld-with-ocalsnt 101821824, 0:11 AM Using @ local LLM to Automate an Android eves Finally try running the sample workflow in the Automate app, and after a moment a toast with the number ‘1000’ should appear. The Automate Flow is passing 1000 as an argument to the script which the script faithfully echoes, it's picked up by the plugin and sent back to the Flow, to be shown in a toast. Script to interact with the model The final piece is to create a script that will call the llama.cpp main binary pointing at the Stable Beluga model, and have Automate call that script in turn. Create a bash script at ~/. termux/tasker/qa. sh with the following content: #! /data/data/com.termux/files/usr/bin/sh the_args="$@" the_output=$(./main --log-disable --seed -1 --threads 4 --r You are a knowledgeable AI assistant. Respond to the user's dt User: $the_args ###t Assistant: " 2>/dev/null) echo ${the_output##*Assistant:} hitpssicode mendhak.comlautomate-android-with toca! ane21824, 0:11 AM Using a lca! LLM to Automate an Android device This uses Stable Beluga’s prompt template to ask a question, and then extracts everything after the Assistant: in the response from the LLM. That is echoed back so that Automate can pick up on it. As before, make it executable, and it's worth trying it out to make sure it's working. chmod Automate calls the script Modify the flow in Automate, instead of passing in 1000, have it pass a hardcoded question, or prompt the user for a question using the Dialog Input block (set the output variable to myvar). You can even output the result in a Dialog Message block, have its message set to the variable so which is the response sent back from the plugin block, which contains the value echoed by the script. hitps:icode,mendhak.comlautomate-android-with toca! sane215/24, 9:11 AM Using @ local LLM to Automate an Android device Pe as ® Ca) Ces aT) asdfghjki Travel Agent example A simple tweak can turn the LLM into a travel agent. Create a ~/.termux/tasker/travelagent.sh with the following contents. Note that --n_predict, the number of predicted tokens, is now set to 250, which means it'll take a little longer to produce an output. hips cade menchak.comlautomate-androld-wth4ocal-tn! ssin8215/24, 9:11 AM Using @ local LLM to Automate an Android device $the_args HHH Assistant: " 2>/dev/null) echo ${the_outputi##*Assistant:} In Automate, create a new Flow which makes an HTTP request to https: //ipinfo.io/city (which returns your city based on IP address), passing that as an argument to the script. hitpssicode mendhak.comlautomate-android-with toca! saneott aM ce ome ecg & coy caer Sen ene pol CECT Using @ local LLM to Automate an Android device rn eae Pea Ors Cree eM ema ta make your way to your accommodation and settle into the city life of Paris! Start Seal ye artery Pree iC u on a uci NTS Near eer nes ure eterna es Explore Montmartre & Sacré-Coeur Basilica Take a walk through the Se ee eared eee aint enc! the early 19th century! Visit the iconic Moulin Rouge cabaret or climb up to ee Reece Afterward, head over to Sacré-Coeur Cee Re teeny skyline while enjoying some peaceful RUC cu etc ae Discover Notre Dame Cathedral & Latin Ceres tues ae ere en eee Ree ea UOC an) Seren So this flow is: Use my IP address to get my city, then pass the city name to the LLM and ask it to generate a short itinerary, Decision making? hitps cade mendhak.comfautomate-androldsath calm215724, 9:11 AM, Using @ local LLM to Automate an Android eves With the pieces in place, it’s a matter of modifying the system prompt for the LLM to have it behave as a decision making tool. The key is to shape the output of the model to match an expected structure, and then to get Automate to parse it and ‘do something’ with it. For example, given a piece of text you can ask the model to produce positive, or negative. That output used in Automate's if block can act as a branch. It's more complicated, but it's conceivable that the LLM could be provided with tools from the specific Automate Flow, and use that to work out a decision itself. Looking at a library like Langchain, the prompt could look something like this: Answer the following questions as best you can. You have ac get_content_from_page: Useful for when you need to get the get_weather_in_location: Useful for when you need to know t¢ get_current_date: Useful for when you need to know the date Use the following format: Question: the input question you must answer Thought: you should always think about what to do Action: the action to take, should be one of [get_content_1 Action Input: the input to the action Observation: the result of the action\ .++ (this Thought/Action/Action Input/Observation can repez Thought: I now know the final answer Final Answer: the final answer to the original input questi ntpe:iicode menahak convautomate-andrld-with calm! 1618215124, 9:11 AM Using @ local LLM to Automate an Android device Begin! Question: {input} Thought: {agent_scratchpad} The main trouble here of course would be the tedious parsing required, feeding it into the right tool (branch) in an Automate Flow, and feeding the response back. This could probably be made easier if frameworks are developed around it. Termux can run Python, which means a lightweight framework to interact with LLMs might be possible. For now the simplest approach is probably to use the LLM to produce a single output and carry on, not bothering with back-and-forth conversations. Updated: 19 Sep 2023 §7 GitHus i] PHotocrapHy &Nooptes 9g GPSLoccer © Feed — © 2024 mendhak — ntpe:ticode menahak convautomate-andrld-with local! ame215/24, 9:11 AM Using @ local LLM to Automate an Android device hitpssicode.mendhak.comlautomate-android-with toca! vse