Well that still leaves the NSA with much more money per year than Harvard and Stanford combined, although that does not say much about the intellectual output.

It isn't really that interesting to compare the NSA budget to the Harvard or Stanford budget, because in neither case is the entire budget going to crypto research, nor can we casually assume any sort of percentage equivalency. The question is how much effort is the NSA putting into crypto research vs. the academic community as a whole, and I imagine we don't have a good way to answer that question. Even raw dollars wouldn't necessarily tell the whole story, since the NSA cryptographers probably have an easier time working together on focused tasks, which both lead to more progress on said tasks while at the same time potentially leaving entirely unresearched areas if groupthink/consensus/management says there's no point to researching something. (I would imagine management uses a relatively light touch if a researcher asserts that something needs to be dug into, but groupthink is a inevitable.) Consequently characterization of where they are ahead and potentially where they may even be behind is probably a bit difficult, since the situation is in reality probably very complex and unknowable to anyone outside the NSA.

"The question is how much effort is the NSA putting into crypto research vs. the academic community, and I imagine we don't have a good way to answer that question."

Actually we do. Network-centric warfare, financial market stability, policy planning, diplomacy and law enforcement do not rely on the academic community for the integrity and security of their systems. They do rely on the NSA. Its not unreasonable to infer the NSA access to funding and intellectual talent in the face of a credible threat or worthwhile opportunity is practically unlimited as compared to academia.

Your broader point is well taken (e.g. Harvard's FY14 budget was $4.4B), but no university relies on distributions from its endowment for most of its operating budget. The balance comes mostly from tuition and government funding, with fortunate universities like the two you mention also receiving gifts earmarked for immediate use.