> More so than Debian? Debian's seems pretty thorough. More so than Red Hat's, with the amount of important customers that depend on their distro?

Yes and yes. openSUSE's automated testing builds each package not just in a chroot but a new barebones VM, to ensure that depdndency definitions are complete. I've seen Fedora devs remark about how they don't have that kind of budget to throw as, e.g., COPR builds, the implication being that they have to put more engineering into their testing infrastructure. (Sorry, it was years ago and I don't have the energy or time to source it.)

Those VMs are accompanied with screen scrapers that perform OCR in order to automatically test the pre-boot environment, before a display server comes up.

It is possible that other distros are catching up here (I know nixos-tests can include virtualized, multi-machine, networked integration tests, for some packages, for instance). But this testing infrastructure for openSUSE is quite mature. See, e.g., this presentation from nearly a decade ago: https://www.youtube.com/watch?v=2K6ZmvIOtPw

As for Debian, the greater centralization of openSUSE's package sources is perhaps worth emphasizing as a major difference here. For openSUSE, package sources are all checked into special VCS system that is part of the Open Build Service (which is capable of tracking and building external packages, including packages from other distros as well). With Debian, package sources are uploaded as binary artifacts by maintainers who can all have their own standards for style/conventions, testing, and other policies. It's not as easy to get a. God's-eye-view of Debian. This blog post provides some general insight into quirks of Debian's non-standardized processes, perhaps a few years out of date now (I don't know), but I hope it makes clear how it might be plausible for a smaller distro to have more mature centralized, automated testing: https://michael.stapelberg.ch/posts/2019-03-10-debian-windin...

> Debian generally seems to prefer decentralized approaches over centralized ones. For example, individual packages are maintained in separate repositories (as opposed to in one repository), each repository can use any SCM (git and svn are common ones) or no SCM at all, and each repository can be hosted on a different site. Of course, what you do in such a repository also varies subtly from team to team, and even within teams.

> [...]

> When you want to make a package available in Debian, you upload GPG-signed files via anonymous FTP. There are several batch jobs (the queue daemon, unchecked, dinstall, possibly others) which run on fixed schedules (e.g. dinstall runs at 01:52 UTC, 07:52 UTC, 13:52 UTC and 19:52 UTC).

> Depending on timing, I estimated that you might wait for over 7 hours (!!) before your package is actually installable.

Debian is an amazing project and it's no happenstance that it has served as the base for many successful distros as well as thrived in its own right for many years. But there's some social and technical cruft there that leaves room for some competition, isn't there?

(It looks like nowadays Debian has actually adopted SUSE's openQA as part of its automated testing stack: https://openqa.debian.net/)

I have a fuzzier picture of how OBS/openQA compares to Red Hat and Fedora's package building and testing tools! You've reminded me that I'd like to learn more about that. But do dig into OBS to get a sense of how SUSE is built. It may even be useful to you: their build cluster offers lots of free compute and it'll let you build and host repos for free— even of other distros like Debian and Red Hat.

None of the tooling questions are the real stuff, though— the real stuff is the predictability and freshness of openSUSE Tumbleweed, which users of other rolling release distros tend to speak favorably about once they've tried it.

Elements of that famous 'balance' are quantifiablw. Check out how openSUSE ranks on overall package freshness: https://repology.org/repositories/statistics/pnewest

Tumbleweed is substantially more up-to-date than Fedora Rawhide, but is also markedly more stable and more committed to stability— the expectation is generally that Tumbleweed should be highly usable by and reliable for non-experts, but the expectation for Rawhide is that it exists primarily for Fedora developers and any breakage is on you.

I'd be happy to compare zypper (and dnf) to apt and all (I admit I've not used Void Linux). I've gotta go for now!

> openSUSE's automated testing builds each package not just in a chroot but a new barebones VM, to ensure that depdndency definitions are complete.

How are you sure other distros don't do that? I'm also not convinced that building in a vm instead of a chroot is any kind of advantage.

> Sorry, it was years ago

So isn't it possible they've changed things now?

> Those VMs are accompanied with screen scrapers that perform OCR in order to automatically test the pre-boot environment, before a display server comes up.

So to me, all that sounds is nifty, but not especially impressive. Any devops team could write something to do the same in a couple of hours. If other distros don't have that, it could simply be because they don't need it.

> It is possible that other distros are catching up here

This is the crux of your position, though, that all these other distros are that much more behind openSUSE. You start your reply with "yes and yes", very confident, but now here in the meat of the discussion you say you don't actually know.

I'm not trying to be combative or adversarial here, but it really seems like a lot of assumptions are being made.

> With Debian, package sources are uploaded as binary artifacts by maintainers who can all have their own standards for style/conventions, testing, and other policies.

I mean, Debian as a system has a standard for its packages, standardizing what they felt they needed to, and all packages adhere to that and the system has been working well, possibly longer than suse has even existed.

> but I hope it makes clear how it might be plausible for a smaller distro to have more mature centralized, automated testing

Well, I never doubted that was possible. I'm just skeptical OpenSuse's testing setup is/has leading/led to a system that is more stable or whatever other supposed advantages it has over say Debian and Red Hat.

> But there's some social and technical cruft there that leaves room for some competition, isn't there?

Is that cruft not just project preferences, of which each distro has it's own version?

> But do dig into OBS to get a sense of how SUSE is built. It may even be useful to you

My days of installing a distro for fun have been over for a long, long time. Now I only really like Alpine, Devuan if I need something chunkier. I appreciate the suggestions and conversation though, thanks!

> So to me, all that sounds is nifty, but not especially impressive. Any devops team could write something to do the same in a couple of hours. If other distros don't have that, it could simply be because they don't need it.

This isn't a complete description of openQA, but an illustrative snapshot. Btw, it looks like Fedora has adopted this tool as well (several years ago, hehe): https://openqa.fedoraproject.org/

And yet there is no one anywhere who claims that Fedora Rawhide is more stable than openSUSE Tumbleweed. This is in part because of a real difference of purpose and outlook; Tumbleweed is somewhat exceptional as a rolling release which explicitly emphasizes stability through automated testing as a value and goal.

> > It is possible that other distros are catching up here

> This is the crux of your position, though, that all these other distros are that much more behind openSUSE. You start your reply with "yes and yes", very confident, but now here in the meat of the discussion you say you don't actually know.

> I'm not trying to be combative or adversarial here, but it really seems like a lot of assumptions are being made.

Sure. I'm not an active contributor to openSUSE or Debian or RHEL, and I'm not currently working on a study of their production pipelines. I'm just trying to give you a clearer picture of some differentiators that I've picked up as a Linux user, developer, and ops professional with the incidental experience I've had with those distros in the past ~20 years.

But I think some of these 'assumptions' are pretty fair, e.g., the distro that created and chiefly maintains openQA has a more mature automated testing infrastructure than a distro that only adopted it a few years ago.

> My days of installing a distro for fun have been over for a long, long time. Now I only really like Alpine, Devuan if I need something chunkier. I appreciate the suggestions and conversation though, thanks!

You don't need to install (or use) openSUSE to play with OBS. Just make a free account on build.opensuse.org and start building packages for any DEB-based, RPM-based, or PKGBUILD-based distro (as well as container images incl. for Docker, AppImage, Snap, and Flatpak). No Alpine/apk support yet, sadly. Idk whether Devuan is supported as a base distro or not, but there are some repositories intended for use with Devuan hosted on build.opensuse.org, e.g.: https://build.opensuse.org/project/show/home:bgstack15

> Well, I never doubted that was possible. I'm just skeptical OpenSuse's testing setup is/has leading/led to a system that is more stable or whatever other supposed advantages it has over say Debian and Red Hat.

The only way you can really compare the stability of two distros is by actually running them. What I can offer you short of that experiential knowledge is that openSUSE has a sophisticated, mature testing infrastructure of which major components have been adopted by the other distros you cite as examples.

You will be hard pressed to find someone who can compare the testing infrastructure of even just two distros in great detail and totally current. This guy probably can, he's a major NixOS/nixpkgs contributor, a Fedora developer, and an Alpine contributor: https://github.com/fabaff . I can't personally think of anyone else, but I'm sure there are some.

I can tell you that still fairly recently, one Debian developer gave a talk about the Open Build Service with the subtitle 'fix the mess': https://www.youtube.com/watch?v=Nh0FoS_J_Gs

> Is that cruft not just project preferences, of which each distro has it's own version?

Yes? But preferences can be literally anything with widely varying impacts on social and technical processes... so to say 'everyone has them' doesn't really say anything at all.

Anyway, about the actual package managers I can be much more concrete because I've used them all much more recently.

> So, what makes the openSUSE package manager so much better than apk, apt or xbps?

Like I said, I won't speak to xbps because I've not used it. Additionally, this comes with the caveat that you may prefer package managers that make different tradeoffs than zypper and dnf do. But here are the things that openSUSE's package manager (and nowadays also Fedora's) get really right imo that apk and apt are lagging on.

This means that resolving dependencies in a way that is guaranteed to be both correct and complete requires is hard (indeed, it's NP-hard). Dependency managers that take this seriously employ SAT solvers, openSUSE's now lives in this library: https://github.com/openSUSE/libsolv. apt lagged behind zypper in this area for many, many years, and just finally got a SAT solver for dependency resolution a couple months ago: https://blog.jak-linux.org/2024/05/14/solver3/ . Apk doesn't have a backtracking dependency solver and thus will sometimes fail to identify possible solutions to dependency resolution problems, even when they exist. But for that, it gets to be fast. I believe pacman and xbps also make that same tradeoff (completeness for speed).

Apt has similarly lagged in establishing a modern command-line interface. Zypper provided a modern, unified subcommand interface for managing all packages and repositories when it came out nearly two decades ago, whereas apt has only offered a modern subcommand interface for less than a year ( https://9to5linux.com/ubuntu-24-10-and-debian-13-trixie-to-f... ), and last I used it (within a year or two) it also still lacked facilities for key management (apt-key was deprecated but there is no integrated replacement). APT is making some very nice advances, but they're still playing catch-up.

The single thing I miss the most with APT, though, is entirely missing, and it's the notion of 'vendor'. openSUSE's package manager introduced me to the concept of 'vendor stickiness' (https://en.opensuse.org/SDB:Vendor_change_update), which allows you to upgrade without changing who is providing what package for you in a natural way. (This idea has also since been adopted by Fedora in their `dnf` package manager, as well.) Zypper's facilities for creating a hierarchy of repositories, prioritizing them, and marking some equivalent to others with respect to 'vendor', are just way more flexible and powerful than anything available in APT.

> I mean, Debian as a system has a standard for its packages, standardizing what they felt they needed to, and all packages adhere to that and the system has been working well

From a packager's perspective, the tooling situation for openSUSE is also way more uniform than for Debian or downstream distros, which can make it easier to get your head around. Even a small amount of packaging experience for Debian is enough to reveal a marked enough difference here to justify a preference for some other distro.

> possibly longer than suse has even existed.

Debian is something like 6 months older than SUSE. SUSE, like Debian, is one of the oldest surviving Linux distributions.