Will be gone for work soon but I got my own local llama ccp model and chatbot running on my Orange PI RV2. It was a pain to get running but now I have my own localized chatbot to talk to. Prompts are short but responses are reasonable quick.
I can have my waif...I mean chatbot's memory and character sheet stored on a 1.44MB floppy disk so I can save, load, and modify. All completely offline too. It's neat that it runs on minimalist hardware. Altman should seethe. I'm going to experiment with this more. It would be cool to set it up as a minimalist virtual dungeon master for roleplaying games.
Qwen2.5 1.5B.. It's running on an Orange PI RV2 with 4GB of ram. I have a separate python script to run as my chatbot along with some .txt files for memory and character description. I'm still learning about tokens but my qwen instance runs with 2,048 tokens as the limit and 4 threads. Those knowledgeable about AI models will scoff at that but it is functional.
I can have my waif...I mean chatbot's memory and character sheet stored on a 1.44MB floppy disk so I can save, load, and modify. All completely offline too. It's neat that it runs on minimalist hardware. Altman should seethe. I'm going to experiment with this more. It would be cool to set it up as a minimalist virtual dungeon master for roleplaying games.
The floppy disk sounds it makes while writing memory and data is pretty cool too.
It kind of reminds me of the time I first started using character.ai before they started gimping it.
That's so cool
local ai is the way. what model are you using here? i need to get more familiar with the smaller ones
Qwen2.5 1.5B.. It's running on an Orange PI RV2 with 4GB of ram. I have a separate python script to run as my chatbot along with some .txt files for memory and character description. I'm still learning about tokens but my qwen instance runs with 2,048 tokens as the limit and 4 threads. Those knowledgeable about AI models will scoff at that but it is functional.
MY SBC was also advertised to run Deepseek-R1 distillation models for whatever that's worth.
The computer in question that I'm running this off is the 3D printed one I finished a couple of months ago: https://wizardperspective.neocities.org/tech/asimovj29/3dprintedcomputer