Sounds like a rather frustrating journey for you.

Thanks! Glad to see the 8x7B performing not too bad - I assume that’s a Mistral model? Also, does the CPU significantly affect inference speed in such a setup, do you know?

So you access the models directly via terminal? Is that convenient? Also, do you get satisfying inference speed and quality with a 16GB card?

Why exactly are the IBM dependencies a problem for you?

I guess I just like independent, community-driven distros, since there’s less space for financially motivated enshittification. Just shortly after I decided to go with FCOS, RedHat / IBM decided to close down CentOS, for example.

I can’t really find good resources on how FCOS is working and what are the benefits. Is it updating the system/kernel automatically as well as the containers?

The system & kernel yes. The whole system is basically a read-only system “image” for which the devs make sure all the packages play nicely together. Packages are not updated individually, but whole system “image” are released periodically, which the system then downloads automatically and reboots (you decide when it actually reboots through the config). If anything goes wrong, the system is rolled back to the previous “image”.

When you go with podman, there’s a systemd service you can enable which will update the containers (i.e. pull the specified image tag). I’m not aware of a similar mechanism for Docker, which is why I use watchtower for that which has been working smoothly so far.

Edit:

And what are generally, in your opinion, the advantages of FCOS?

For me, it’s the (quite safely designed) auto-updates of the base system (I just feel like having to do less repetitive work), infrastructure-as-code aspect, and the container mindset (as I containerize everything anyways). Also I just have a weakness for new, fancy stuff.

I use Fedora CoreOS on my homeserver and a bunch of VPSs. Migrated the homeserver just recently, but I’ve migrated the first VPSs a bit more than a year ago. So far, I had no problems with it. There’s a low-traffic mailing list where the devs inform about security issues and breaking changes to the whole container stack.

I used debian before for some years, but at some point became tired of manually updating the system (which is probably one of the biggest benefit of FCOS). It takes, however, quite some time to put your first Ignition config together, and debugging is tedious as you have to redeploy to see if a bug / error is now gone (I’ve used a VM for that).

I use podman on some, Docker on other servers (you can’t use both at the same time). Both have been working well so far.

I’d recommend it, but would also recommend taking a look at Flatcar Linux which is more or less the same without the IBM dependency (which makes my stomach hurt sometimes).