Google's first TPU was developed a year after Tensorflow. And for that matter, Tensorflow works fine with CUDA, was originally entirely built for CUDA, and it's super weird the way it's being referenced in here.
Tensorflow lost out to Pytorch because the former is grossly complex for the same tasks, with a mountain of dependencies, as is the norm for Google projects. Using it was such a ridiculous pain compared to Pytorch.
And anyone can use a mythical TPU right now on the Google Cloud. It isn't magical, and is kind of junky compared to an H100, for instance. I mean...Google's recent AI supercomputer offerings are built around nvidia hardware.
CUDA keeps winning because everyone else has done a horrendous job competing. AMD, for instance, had the rather horrible ROCm, and then they decided that they would gate their APIs to only their "business" offerings while nvidia was happy letting it work on almost anything.
The same reason most of AMD's 'open' initiatives don't gain traction: they throw it out there and hope things will magically work out and that a/the community will embrace it as the standard. It takes more work than that. What AMD historically hasn't done is the real grunge work of addressing the limitations of their products/APIs and continuing to invest in them long term. See how the OpenCL (written by AMD) Cycles renderer for Blender worked out, for example.
Something AMD doesn't seem to understand/accept is that since they are consistently lagging nVidia on both the hardware and software front, nVidia can get away with some things AMD can't. Everyone hates nVidia for it, but unless/until AMD wises up they're going to keep losing.
Tensorflow lost out to Pytorch because the former is grossly complex for the same tasks, with a mountain of dependencies, as is the norm for Google projects. Using it was such a ridiculous pain compared to Pytorch.
And anyone can use a mythical TPU right now on the Google Cloud. It isn't magical, and is kind of junky compared to an H100, for instance. I mean...Google's recent AI supercomputer offerings are built around nvidia hardware.
CUDA keeps winning because everyone else has done a horrendous job competing. AMD, for instance, had the rather horrible ROCm, and then they decided that they would gate their APIs to only their "business" offerings while nvidia was happy letting it work on almost anything.