The statically linked binaries run inside the containers. Static linking gives you a certain kind of portability (no need for library dependencies on the machine.) The containers give you isolation, resource management, etc.
The way it has been explained to me is that Google's fat binaries have no dependencies beyond libc. The task scheduler would just have to deploy that one file to one or more machines to get it to run.
Containers are much more flexible that statically linked binaries. You could have multiple binaries in a container sharing a common set of dynamically linked files.
Fat binaries inside containers sounds a bit like the worst of both worlds...
If you want unrelated jobs to share library binaries, you need to stick to explicit release schedules for those libraries, which means more coordination between the various app teams and the teams that own the libraries.
When the size of the libraries (megabytes) is compared with the typical heap size of running jobs (gigabytes - a small number of large instances per job is typically a lot more efficient than a large number of small instances) the space savings of shared libraries become pretty negligible.
Back then, one of the main bottlenecks in the system was the central scheduler, in particular the amount of work it had to do tracking what binary packages were installed on each machine and what were needed for the candidate jobs it might plan to run on those machines. Having many packages per job just makes the scheduler bottleneck worse.
There were two places where it did actually make sense to share packages between jobs:
- Java Virtual Machine and supporting libraries
- libc
These were external (so changed much less frequently), quite large, and were needed by very large numbers of jobs, so the space savings of having one copy of each needed version per machine outweighed the extra scheduling load required for them.
Static binaries are congruent to containers, in a way. Both bundle an app and dependencies together. There are advantages and disadvantages to both approaches, but Docker-style containers are more elegant in some ways.
