3AM Lab Notes

Building Labs at the Worst Times

A short history of buying the wrong hardware at exactly the wrong point in the hype cycle.

About two decades ago, virtualization was the hot topic.

The cloud was still a dirty little secret whispered by people who knew where the industry was going. Private or public, same tomato, different consumers.

Back then I ran vCloud Director on a pair of Intel i5-3570 boxes with 4x8GB DDR3, a pair of 256GB SATA SSDs, and a 1TB HDD presented to the host through FreeNAS.

Who would have thought this was going to look like Nutanix’s great-great-grandfather?

Oh, and I used that lab to POC vCloud Director for a real public cloud offering.

Those two nodes cost me an arm and a leg.

SSDs were not common yet. They were mostly seen as cache, SLOGs for ZFS, or database server toys. Only storage systems from the big boys like Dell, HP, IBM, and EMC had SSD cache offerings that felt serious.

Cache use. Not all-flash-everything.

The drives were not large enough or cheap enough yet to make what is now boringly normal: an all-flash storage array.

So it is with great dismay that I write in this climate, where DDR5 and NVMe prices are back in the same wallet-rendering condition. The AI boom is upon us, in case you have been living under a rock for the past three years. Anything used by data centers running massive AI hardware is either out of stock or priced to oblivion.

It makes a hardware nut like myself teary-eyed.

Fortunately, there is more than one way to make bad financial decisions.

LLMs can run locally if your GPU has enough RAM. Or you can build an LLM gateway and point it to your preferred provider. Personally, I prefer running models locally. I would love to cobble together three RTX 4090s, but even used, those GPUs are overpriced.

If the budget committee, also known as my wife, would approve an RTX Pro 6000 with 96GB of VRAM, no cobbling would be needed.

Until then, the lab makes do with a mix of cloud models, local models, and whatever fits without turning the monthly budget into modern art.

The hardware changes. The sickness remains.

I build labs because reading about systems is not the same as watching them misbehave.