Articles tagged “buying guide”

5 articles

When the Model Doesn't Fit on the GPU

A 155GB model on a 96GB card still runs, if it is a mixture-of-experts model and llama.cpp parks the experts in system RAM. We measured fourteen such splits on an RTX 5090, a Radeon AI PRO R9700, an RTX PRO 6000, two RTX 5070 Tis and an H200: what the trick saves, what it costs, why the PC behind the card is what you are waiting on, how well a napkin predicts it, and the twelve-channel EPYC build that gets around it.

Mac Studio M5 Ultra for AI: What Can You Run

We ran a 256GB M5 Ultra through the same benchmark as eight other machines: 22 models, five prompt sizes, two engines, cached follow-ups, crowds and power. It holds 300B-class models no desktop card can and writes them at 39 to 131 tokens per second, loses more speed than any PC as the chat grows, and reads twice as fast on MLX as on llama.cpp. What fits, what does not, and when a DGX Spark, an RTX PRO 6000 or an RTX 5090 is the better buy.