I started experimenting with running an LLM locally.
I have a Lenovo laptop with an RTX5060 8GB NVidia graphics card, and wanted to see if i can run an LLM locally to do stuff.
Document comparison, simple application generation, running agents, and so on.
It’s been quite the ride. So many options, so many choices, so many apps that you can install on a Windows machine, depending on what you want to do with it.
But at the start of the journey you don’t know yet what to do. So what i started to do is just install a bunch of things, read up on what the use case would be for these things, and then figure out what model to run.
One immediate observation: each app that uses models will download a model into its own ‘space’. So if you have three or four different tools installed, and you want to install “model A”, it will in the end download that same model four (4) times. Sigh.
Anyways, more to come on this. I have a lot of notes from various attempts, that i want to consolidate, clean up and post here.