Caches aren't the problem (in the context of SPECTRE), branch prediction is. Getting rid of caches would be very costly performance-wise. VLIW CPUs don't predict branches themselves, but rely on the compiler to generate optimal code ahead of time. I was actually expecting an updated Itanium after the SPECTRE debacle.
AFAIK branch prediction enables the attack by enabling speculative execution, but the data is not leaked through the branch predictor, but rather through the timing differences observed due to speculative loads bringing data to cache.
I guess if you remove the branch predictor, you might avoid spectre while keeping caches, but I think you can keep the branch predictor and remove caches to also avoid spectre.
The downside I see in keeping the caches is that you keep the _source_ of the timing differences, so an attacker just needs to find a different attack vector to create a new timing attack.
If you remove the caches, you kill the source of most timing attacks.
