Today I x-eeted something that got community noted. This is the first time I’ve gotten CN’d and it felt great! Free speech FTW. While I was perhaps being slightly exuberant, I still stand by my post… Let’s unpack it a bit.
Meta is training LLaMa 3 this year. I believe they will not only launch 3 this year but maybe even 4.
In the video, Zuck said Meta will have 600,000~ H100s worth of GPU compute to train open source AI. It was a natural leap for me to say they are dedicating those GPUs to train LLaMa, since that is the name of the family of models they are building toward shipping the leading open “source” (weights, more accurately) AI.
Each H100 costs around $30K~, so 600K~ is around $20B.
I find the level of conviction that Meta has in building the best open weights AI to be not only remarkable, but deeply commendable. Even though the architecture and paradigms being used to build the current generation of AI systems is so incredibly inefficient.
Full disclosure that my fund helped incorporate and is the major founding investor in Liquid AI (I am also their first board member). Liquid is building an alternative to the Transformer architecture with multi-modal capabilities offering inherent causality and expressiveness not found in any other AI architecture. We’ve already seen results that are several orders of magnitude more compute efficient vs. Transformers. If GPT-4 required 20-30K A100s (6X less powerful than H100s), and GPT-5 needs 10-15X more GPUs than that (say, 300K~ H200 equivalents)… then an AI architecture that can deliver the same cost performance but on 300X less compute means we may only need a few thousand H200s~ to build the most powerful AI within the next year. The cost savings here measure in the many many billions of dollars… and the environmental impact reduction is also pretty compelling.
Joseph Jacks’ Personal Blog 