P5

One other comment. I deployed on an affordable Bosgame M4 Mini PC, 32 GB DDR. Ive already hit memory limitations and slowness creating when creating vector embeddings. Since our deployment is for our 10 person organization I will need to beef up the hardware.

It depends if you’re talking about dense models or MOE models. On 70B+ MOE models, you have benchmarks above that, and you won’t get better performance with a single GPU with 24 or 32GB of VRAM and the rest in RAM. The only time you’ll do better with GPUs is if your entire model fits in VRAM, includ

For the best results you're looking for something with 32GB of RAM to share with Radeon RDNA integrated graphics. Currently I have a friend looking at the GMKtec M7 for this and similar purposes after seeing my K8 Plus. There are some cheaper options like the Bosgame P5 6800H with less features but

The P3 Lite & P5 Pro are alright. Know a few BosGame P3 Lite owners who experience some thermal issues while a few P5 Pro owners experienced HDMI problems. Both apparently have the standard mPC Wi-Fi 6E performance issues from poor quality cards found in a number of laptops & mPCs these days.