How many students does Harvard graduate each year? How many employees does Google have? If one year Google hired the entire math and CS graduating class of Harvard, would it still be "pretty rare" to see one of those people at Google?

Harvard you'll see more in the Cambridge or NY office. And at Facebook (and Microsoft, but maybe not anymore) due to the founder's alumni network.

And as another mentioned, Google is so big that most hires can't be from a few small schools.