The latency issue is only present once, the initial page load. After that the resources are cached. Second to that, if you're following best practices for page speed, the user will not notice at all because a snippet of CSS that provides the initial layout and styles will be sent with the HTML body. Amongst dozens of other things you can do to make this a non-issue.
>>The latency issue is only present once, the initial page load.
The initial page load is also one of the most important things to optimize for things like, you know, conversion of visitors to paying customers. I've given up on subscribing to new products and services simply because their pages weren't performing well, and I'm sure many others here have done the same.