I guess I'm more amazed at how much energy goes into relatively minor advances or even in going backwards. Like the proliferation of frameworks, the difficulty in cross-platform development, how hard scaling remains, etc.
Better languages: {C#, C++14, Go, Rust} > Java, {ES6, TypeScript} > JavaScript, and Python keeps getting better thanks to Python 3 and an better libraries.
Scaling: OpenCL and CUDA have both taken off, delivering everything they promised. Hadoop matured and spawned variants, and now Spark is starting a new cycle. Thanks to Amazon and Google we now have cheap computing cycles on tap with APIs usable by ordinary developers.
It seems to me that only Web Development and Mobile Apps are a morass. But web development was bound to be a step backwards given its strange history, while mobile app development can only be as good as the underlying platforms, which started from really rudimentary to get where they are now.
Cross-platform development is not like the other problems (too many poor quality frameworks, unsolved problems like scaling, etc.) The main problem with cross-platform development is that the creator of the platform usually has no incentive to work with other platforms; quite the opposite. That road leads to commoditization. Users and developers want commoditization, but sellers do not.
Another problem is that often if you create a platform, it is because you want to do something different than established platforms. This inevitably makes cross-platform difficult.
Think of what EJB, EJB2 and Spring did to Java. Think of Struts. Instead of fixing the real problems of the language, we went in a lot of side paths that barely got us ahead of where we started and, if nothing else, slowed progress down. Java in 2002 vs Java in 2012: Not really that much better. Working in Java before was like working with a broken bone, and instead of fixing it, the frameworks were painkillers. So we still had a broken bone, and then we were addicted to Vicodin.
We have been in a similar boat for a while in Javascript land. A few specific calls make web development to be a worse experience than using Delphi to build native apps back in 1998. This problem of repetitive code + css + a dom has led to lots and lots of libraries and frameworks, moving constantly. But it feels like they are not really moving, as far as productivity is concerned. FP inspired tools, like Elm, or React, might finally get us out of this mess, but the jury is still out.
And let's not get started about the state of build systems in javascript land. Lots of things are being written, but it's really hard to figure out why. The rate of actual progress is pretty slow.
We are also in a similar boat in cloud and virtualization. Lots of little tools that are supposed to make our life easier, but most are extremely fragile, because we build the infrastructure and sell it to the world before we understand the problem. The entire ecosystem around docker and coreos is the wild west. And then there's the distributed databases. Dozens of options, less than a handful that handle a network split in a sensible way. Just read some of the posts in the Jepsen series.
Sometimes we solve a problem well, quickly, and it's solved for good. Other times it feels that all this effort from the community is the same as being stuck in the mud. In those cases, the tools don't feel like stepping stones, or foundations to build upon. It's a bunch of different people making infrastructure that fail to learn any lessons from their competitors.
If anything, what is amazing is that we manage to get systems that work in some fashion, given that we are using so many broken tools.
Well, I don't really agree with any of this, frankly. Every single thing you cite as slowing progress WAS progress, just not the progress you feel "should" have been. I think you hold your opinions based on a fallacy that with your hindsight now, you could solve the problems of 20 years ago.
And I certainly don't agree with your assessment that any problem has been solved "for good."
What I see happening is a natural evolutionary process informed by the independent actions of millions of people, and some hindsight-fueled speculation of what the value of central planning would be if it could be done with perfect knowledge. Basically, one works, has worked, is working, and will continue to work (and not just in software), and the other is a fantasy brought about by the human brain's biases, which has led to things like the USSR and China's current gov't, amongst other travesties.
I'll take a stab at it, at least my perspective on it from the "enterprise application" trenches.
It's amazing how complicated many multi-project, multi-levels-of-indirection rube-goldberg-machine-like solutions are to accomplish the most trivial of tasks. The ratio of actual "doing the actual required functional work" code to various "architectural best practices" and other plumbing is I'd speculate often in the neighborhood of 1:20.
Yeah. The main reason that there has been any progress at all is that countless thousands of people have been working at it, and occasionally somebody will come up with an idea that will stick. Countless other (possibly brilliant) ideas will be ignored and die.
