I wanted to run LLMs locally on my server (for better privacy), and was wondering if:
I could use Intel ARC/AMD GPUs - these are often less expensive and AMD has open source drivers, which is something I like.
If a PCIe x4 Gen 3 slot would be enough (it’s an x16 slot with x4 speeds) - this is an important consideration.
Would 8GB of RAM (in the GPU, I believe it’s called VRAM?) be enough?
I’m looking at language models to train on my Reddit and Lemmy content, in an aim to make it write like me (and maybe even better than me? Who knows). I don’t quite know which models I will train, or how I will do so (I certainly won’t be writing anything from scratch), but I was wondering; with the explosion of FOSS AI models, maybe something like this would be possible with the hardware constraints I mentioned above?
Does the speed of the connection between the GPU and the CPU really matter in such applications?
I’ve been doing a lot of research on this over the last 2 weeks. I have my machine in the mail, but have not tried anything myself on my own hardware.
For Stable Diffusion, 8GBV is usually considered absolute minimum to do very basic stuff only. 16GBV or more is the basic need for a decent workflow.
For AMD I have seen multiple sources saying to avoid it, but there are a few people that have working examples in the wild. Apparently, AMD only supports the 7k series of GPUs officially with ROCm/hips/AI stuff.
Officially with Stable Diffusion, only nvidia is supported.
I don’t know the kind of LLM I would want to run. I’m just going through some names, would you be able to recommend anything that might learn from text?
Thanks, it would seem I need to stick to Nvidia, although I don’t like the idea very much. Unfortunate
PrivateGPT is on my list to try after someone posted about it weeks ago with this how to article (that has a view limit embedded before a pay wall)/github project repo:
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
You can rent super powerful GPUs by the minute via cloud infrastructure. It’s probably the most viable way.
Sorry, but I don’t think that’s a private idea. I probably won’t be doing that
What software do you want to run?
I’ve been doing a lot of research on this over the last 2 weeks. I have my machine in the mail, but have not tried anything myself on my own hardware.
For Stable Diffusion, 8GBV is usually considered absolute minimum to do very basic stuff only. 16GBV or more is the basic need for a decent workflow.
For AMD I have seen multiple sources saying to avoid it, but there are a few people that have working examples in the wild. Apparently, AMD only supports the 7k series of GPUs officially with ROCm/hips/AI stuff.
Officially with Stable Diffusion, only nvidia is supported.
I don’t know the kind of LLM I would want to run. I’m just going through some names, would you be able to recommend anything that might learn from text?
Thanks, it would seem I need to stick to Nvidia, although I don’t like the idea very much. Unfortunate
This is a general list that was shared recently (has google analytics though):
PrivateGPT is on my list to try after someone posted about it weeks ago with this how to article (that has a view limit embedded before a pay wall)/github project repo: