I remain sad that Mesos never really took off, and then k8s ate the market. It had a lot of really clever ideas and could do stuff well that k8s still can't (although it has been catching up, ten years later). In particular, the scheduler architecture allowed it to work well for both batch workloads and service-like workloads within a single resource pool.
Borg, Omega, K8s, and Nomad all have a separate scheduling pathway for batch jobs, don't they? My understanding is that this is the big complexifier for all distributed schedulers: that once you have two different service models for scheduling (placement sensitive delay insensitive, and placement insensitive delay sensitive), you now have two scheduler process, and you have to work out concurrency between their claims on resources.
The Omega design in particular is, like, ground up meant to address this problem; the whole paper is basically "why the Mesos approach is suboptimal for this problem".
It's true that Omega was designed specifically to solve this problem well (and one of the big insights from the original Borg paper is how efficient it is to have heterogenous workloads on a single cluster). But (as a non-google person with no inside info) I think the amount of omega inspiration in kubernetes is overstated.
Until a few years ago there was very limited support for pluggable schedulers in kubernetes (I think it's gotten better recently but I'm a bit out of date). The beauty of Omega and Mesos is that schedulers were a first-level concept. How Spark thinks about job scheduling will not be the same as how Presto does, or a real-time system like Flink. Mesos provided a framework where you could create custom schedulers for each class of application you run.
For example, for a query engine you might care about scheduling latency more than anything else, and you'd be willing to accept subpar resources if you're able to get them more quickly. For a Spark job, the scheduler can introspect the actual properties of the job to figure out whether a particular resource is suitable.
There was a ton of Omega inspiration in K8s, in the sense that Omega was mostly a failed software project, and while the scheduler potions ended up being backported, K8s mostly avoided doing the hard things and settled for good enough because of it.
Yeah it was neat and oh so much less complex to setup than k8s from scratch.
But k8s is not just job scheduler, it comes with (at hefty complexity cost) with all the piping and plumbing around it so it appealed to developers that could pop out a single .yaml that described their whole architecture.
