P5

One other comment. I deployed on an affordable Bosgame M4 Mini PC, 32 GB DDR. Ive already hit memory limitations and slowness creating when creating vector embeddings. Since our deployment is for our 10 person organization I will need to beef up the hardware.

It depends if you’re talking about dense models or MOE models. On 70B+ MOE models, you have benchmarks above that, and you won’t get better performance with a single GPU with 24 or 32GB of VRAM and the rest in RAM. The only time you’ll do better with GPUs is if your entire model fits in VRAM, includ

For the best results you're looking for something with 32GB of RAM to share with Radeon RDNA integrated graphics. Currently I have a friend looking at the GMKtec M7 for this and similar purposes after seeing my K8 Plus. There are some cheaper options like the Bosgame P5 6800H with less features but