CloudStack SBOM improvements

[ppkarwasz]

#14570 generates SBOMs for the main building blocks of CloudStack (the Java modules, the UI and Marvin). That is a good basis, and I would like to discuss which SBOMs would actually be useful to downstream users, and how to produce them.

I see two candidates:

1. A source SBOM (pre-build) that accompanies each source release. It would mainly flag the files that are not under Apache-2.0 or that come from other projects, think of it as a NOTICE file on steroids, and give an idea of the external dependencies needed to run CloudStack. This kind of SBOM is not useful for vulnerability analysis.
2. A package SBOM (build) for each .deb and .rpm we publish. Ideally it is complete: every component the package contains, whatever the language (Java, JavaScript, Python, C) and however deeply nested (a JAR shaded into another JAR, a native library inside a JAR).

On how to produce them: SBOM generators either read the build files (pom.xml, package-lock.json, debian/rules, ...) to know what should be in an artifact, or scan the artifact to find what is actually in it. Both have blind spots. For example, cloudstack.jar shades both commons-logging and jcl-over-slf4j, which contain the same class names, and no build file tells you which copy ended up in the JAR. My preference is for the project to publish build-based SBOMs, since users can run a scanner just as easily as we can.

If you use CloudStack, or package or redistribute it, it would help to hear from you:

• Do you consume SBOMs today, and with which tools? Which format (CycloneDX or SPDX) and version?
• Which artifacts matter to you: the source release, the packages, the system VM templates?
• What would you use them for: vulnerability management, license compliance, regulatory requirements (e.g. the EU Cyber Resilience Act)?

License compliance is not my area, so opinions on what the source SBOM should contain are especially welcome.

[ppkarwasz]

As a starter for a discussion, I analyzed what ends up in the Debian *.deb packages, with particular attention to what no scanner (or almost no scanner) will find on an installed system. Scanners look for manifest files, compute hashes (e.g. to find the corresponding file in Maven Central or other content-indexed registries) or look for symbols in binaries. It is easy to throw them off (intentionally or, usually, unintentionally), but a *.cdx.json file, even in the /usr/share/doc folder of the package, will allow a scanner to retrieve a (possibly partially duplicate, because already detected) set of components that are in the system.

Ghost components in cloudstack-management

I ran syft on the extracted cloudstack-management_4.23.0.0_all.deb. It reports 25 components: 9 JARs and the 16 Go modules of cmk. This is what it does not see:

Ghost	Where	Why scanners miss it
~320 third-party Maven libraries (Spring, Jetty, CXF, Jackson, Netty, AWS SDK, ...)	shaded into lib/cloudstack-4.23.0.0.jar (264 MB, 193 000 classes)	All META-INF/maven/**/pom.properties are gone and the JAR's hash matches nothing in Maven Central. syft reports a single component, cloudstack 4.23.0.0.
71 native libraries (JNA, snappy-java, Netty epoll, ...), for Linux but also Windows, macOS and Solaris	inside the same shaded JAR	They are neither Java libraries nor files on the system: nothing catalogs a .so inside a JAR inside a package.
~130 npm packages (Vue, Ant Design, axios, ...)	bundled by webpack into the 483 files of webapp/js	A bundle has no package.json and no lockfile. The names are only visible in the *.js.map files that happen to be shipped, which no scanner reads.

The 9 JARs that are found are exactly those copied as-is into lib/ (Bouncy Castle, MySQL Connector/J, Linstor, ...), next to the shaded JAR. cmk is found too, because Go binaries embed their module list. So the ghosts are not a limit of the scanners: they come from how we package.

cloudstack-common has more of them, for example code copied into the source tree rather than declared as a dependency, like noVNC and jQuery in systemvm/agent, which ends up in agent.zip and in the system VMs.

Can't the build tools tell us?

For the shaded JAR, not yet. Support for the Maven Shade Plugin in the CycloneDX Maven Plugin was requested in CycloneDX/cyclonedx-maven-plugin#472 two and a half years ago, and it is finally going forward thanks to @hboutemy in CycloneDX/cyclonedx-maven-plugin#696. That PR teaches the plugin to tell, from the packaging plugins that run, which dependencies are embedded in the artifact and which are only required (CycloneDX 1.7 isExternal). We are not there yet: the PR is still open, Shade's configuration is flexible enough to need more work, and a module that produces both a plain and a shaded JAR still gets a single SBOM.

In the meantime, I put together an experiment on top of #14570: ppkarwasz:sbom-phase1. It produces one SBOM per Maven module, which matches the JARs that debian/rules copies into each package by hash, pins the cmk download, and lists Marvin's Python requirements as external components. The plan describes how these pieces could be assembled into one SBOM per .deb.

What could make CloudStack more detectable

Some modifications of CloudStack might also be beneficial to have a more detectable set of embedded components:

• Stop shading cloudstack.jar. The management server already starts with lib/* on the classpath, so the third-party JARs could be copied unchanged, like the 9 above. Every one of them would then be identifiable by its hash, and its signature (if any) would stay valid. This would also sidestep the Shade problem above.
• Ship only the native libraries we run. Windows, macOS and Solaris binaries can't run on a Debian system, but they still show up in vulnerability reports.
• Ship an SBOM in each package, e.g. /usr/share/doc/cloudstack-management/cloudstack-management.cdx.json, so that a scanner on the installed system can fill in what it cannot discover by itself.

[ppkarwasz]

For the 71 native libraries hidden in cloudstack.jar there is also a better solution: stop shipping them! 😉

They come from a handful of JNI libraries: JNA, snappy-java, zstd-jni, lz4-java and Netty's native transports (which are based on APR, APR-util, Tomcat native, and an OpenSSL fork BoringSSL that doesn't issue CVEs). Several of these are packaged by GNU/Linux distributions, and the distribution versions are better suited to our use case. For example, on Debian 12:

Maven artifact	Debian package	Native part links against
net.java.dev.jna:jna	libjna-java + libjna-jni	libffi8
com.github.luben:zstd-jni	libzstd-jni-java	libzstd1
org.lz4:lz4-java	liblz4-java + liblz4-jni	liblz4-1
org.xerial.snappy:snappy-java	libsnappy-java + libsnappy-jni	libsnappy1v5

The Maven JARs embed a static copy of the C library for every OS and CPU. The Debian ones contain no native code: they link to the system library, so a CVE in libzstd is fixed by the usual Debian security update, without a new CloudStack release.

In short, the plan would be:

1. Stop shading. Shipping a single JAR may have been useful historically, but CloudStack binaries are now distributed as OS packages, and the management server already starts with lib/* on its classpath.
2. Swap individual JARs for distribution packages. Once every JAR is a separate file in lib/, the packaging scripts can replace, e.g., lib/jna-5.x.jar with a symlink lib/jna.jar → /usr/share/java/jna.jar and add libjna-java to the package's Depends:. In the SBOM, the embedded pkg:maven/... component becomes an external pkg:deb/... one.
3. Get security fixes from the distribution. Users would no longer need a new CloudStack release because a CVE was fixed in one of these libraries.

Of course, this makes dependency management more complex:

• The JAR and its native library must come from the same package, since the JNI code is bound to the Java classes it was built with.
• The same .deb is installed on several Debian and Ubuntu releases, each with a different version of the library, so more testing is needed.
• Distribution versions can also lag behind: Debian 12 ships snappy-java 1.1.8.3, while CloudStack uses 1.1.10.7. So this should be decided library by library, starting with JNA, which is used by the KVM agent.
• On EL and SUSE, most of these packages are not available, so the RPMs would keep the Maven JARs.

Note

Many of the JARs that contain native libraries also seem unnecessary, since CloudStack does not use them in a way that needs the native code. For example:

• waffle-jna, a Windows-only authentication library pulled in by the MariaDB driver, is the only reason JNA is in cloudstack-usage and in the system VMs;
• zstd-jni and lz4-java only come with kafka-clients, for the Kafka event bus plugin;
• the Netty kqueue and macOS DNS resolver JARs cannot even load on Linux.

Excluding them would make the SBOM slimmer, and reduce the churn caused by new CVEs in components we ship but don't use, without having to write VEX statements. It carries the same update and testing risks, though.

[ppkarwasz]

To produce SBOMs for Debian packages (and packages for other distributions too, I assume), there is a small problem to sort out in the Package URL definition first. The deb PURL type is defined as pkg:deb/<namespace>/<name> and the namespace is not strictly defined. We would need to understand whether we should use:

• pkg:deb/cloudstack.org/cloudstack-management,
• or pkg:deb/debian/cloudstack-management?repository_url=https:%2F%2F/downloads.cloudstack.org%2Fdebian.

I opened package-url/purl-spec#1059 to discuss this.