Breaking News: The Shai-Hulud npm Malware Returns With 320+ Affected Packages

CI/CD Security Can’t See What Your Pipeline Ships Without Code Changes

CI/CD Security Gaps

TL;DR

  • Your scanner checks the artifact. It doesn’t check how it was built. A base image update can change your production binary without a single code commit, and your dashboard stays green.
  • Supply chain attacks more than doubled in 2025. 22% hit CI/CD pipelines directly. Projected losses: $60 billion by year end.
  • An SBOM tells you what’s inside an artifact. A Pipeline Bill of Materials (PBOM) tells you how it got there, runner identity, build arguments, environment variables, and whether the artifact actually matches its source.
  • Mutable tags and unpinned dependencies make your builds non-deterministic. Same commit. Different binary. Different day.
  • One shared pipeline template update silently changes build logic across hundreds of services. No pull request. No code review. No trace in your repo.
  • Track execution lineage, not artifact snapshots. That’s how you go from knowing what shipped to knowing what actually ran.

At 02:13 UTC, hundreds of production services are rebuilt across multiple pipelines, even though application code remains the same. A shared pipeline template was updated earlier in the day, and a cached base image tag now resolves to a different digest during the nightly runs. This reflects a growing risk. In 2025, software supply chain attacks more than doubled globally, with 22% of incidents specifically targeting build environments.

Bar chart showing supply chain attacks by sector in 2025 with IT, technology, and finance as top targets for CI/CD and software supply chain breaches

By morning, security dashboards still show a clean state. Dependency scans complete without blocking releases, SBOMs refresh automatically, and no new alerts surface that would suggest elevated risk. From the outside, the system appears unchanged and under control. In reality, the software running in production is no longer identical to what was deployed the day before. The rebuild altered the final artifacts, yet no code review occurred, no pull request captured the change, and no security control can explain what shifted inside the build process.

Traditional CI/CD security tooling is not built to reason about this class of change. It inspects artifacts after they exist, evaluates them against static policies, and reports a pass or fail. When the pipeline itself becomes the source of drift, those tools lack attribution, context, and history. This gap is why OWASP Top 10:2025 now ranks Software Supply Chain Failures as a top-three global risk.

This article focuses on what CI/CD security misses when pipelines rebuild without code changes, why that pattern became prominent in 2025, and the specific visibility gaps that teams must close in 2026 to maintain real security at machine speed.

Why Rebuilds Without Code Changes Are Normal in Current CI/CD

Rebuilds without code changes appear suspicious only if application repositories are treated as the sole source of change. In production CI/CD systems, multiple upstream inputs shape the final artifact, and those inputs evolve.

Rebuilds Are Triggered by Inputs That Live Outside Application Repositories

In most environments, application code represents only one input into the build process. Base images, dependency registries, pipeline templates, and shared build scripts all influence the final artifact while living outside the repository boundary.

A rebuild can occur when a base image tag resolves to a different digest or when a centrally managed pipeline template modifies a build step inherited across services. The repository remains unchanged, but the resulting artifact does not.

Teams rely on these external inputs to maintain consistency across large fleets. The consequence is that rebuilds can modify software without leaving a trace in application-level version control.

Dependency Resolution Is Time-Dependent, Not Deterministic

Dependency resolution depends on an external state that shifts over time. Registry availability, cache behavior, mirror synchronization, and artifact retention policies all affect which dependencies are resolved during a build.

Two builds executed hours apart against identical sources can produce different outputs. Those differences can alter linked libraries, enabled features, or runtime behavior in ways that affect exploitability rather than simply inflating vulnerability counts.

Assuming source stability implies artifact stability is a fragile assumption in systems that rebuild frequently.

Shared Pipelines Turn Small Changes Into Fleet-Wide Events

Pipeline reuse concentrates build logic into a small number of shared definitions. A single update to a common pipeline affects every repository that consumes it the next time a build runs.

GitHub workflow diagram showing how shared pipeline templates distribute changes across repositories via fork, push, commit, and pull request cycles

This diagram shows how a shared pipeline template distributes changes across many repositories. It illustrates the risk of centralized build logic having a large impact area. 

A single change in a shared GitHub Action or GitLab Template can introduce tools or permissions that the application developer never requested and cannot see in their own repository.x

# internal-templates/go-build.yml
# A "minor" update by the Platform team to "help" with debugging
spec:
  inputs:
    stage:
      default: build
---
build-job:
  stage: $[[ inputs.stage ]]
  script:
    # This curl command was added to "improve telemetry"
    # but introduces an unverified binary into the build environment.
    - curl -sSf https://telemetry.internal.net/setup.sh | sh
    - go build -o main .

This reuse reduces maintenance overhead but increases the blast radius of pipeline changes. When rebuilds affect hundreds of services at once, the problem is no longer whether change occurred. The problem is whether existing security controls can explain what changed across the fleet and why.

What Traditional CI/CD Security Observes

Most CI/CD security tools were not built to answer that question. They evaluate artifacts in isolation, without retaining the pipeline context that produced them. As a result, they observe the output of rebuilds without understanding the cause of the change.

Artifact Snapshots Taken After the Fact

What the Scanner Sees

The scanner mounts the filesystem of your Docker image. It finds /usr/local/lib/node_modules/axios/package.json. It reads version: 1.6.0. It checks its database: “Is axios 1.6.0 vulnerable?”

  • Result: No.
  • Verdict: Pass.

What the Scanner Misses

The scanner does not know why axios is there.

  • Scenario A (Safe): A senior engineer explicitly bumped the version in package.json to fix a bug. (Intent: Deliberate).
  • Scenario B (Risky): A scheduled pipeline ran at 3:00 AM. The package.json allowed ^1.5.0. The registry served 1.6.0 because it was published 10 minutes ago. No human reviewed this update. (Intent: None/Accidental).

This visibility gap exists because traditional tools rely on a static SBOM (Software Bill of Materials). An SBOM is a post-build snapshot that tells you what is inside an artifact, but it discards the execution context required to validate integrity.

To solve this, we need a PBOM (Pipeline Bill of Materials).

SBOM vs. PBOM

While an SBOM identifies the components, a PBOM records the specific pipeline definition, runner identity, and environment variables that produced it.

The SBOM (Artifact Snapshot)

The SBOM only confirms that axios is present and its version.

To the scanner, Scenario A and Scenario B are identical. To a security engineer, Scenario B is a high-risk “Ghost Build” that introduces unverified code into production.

{
  "component": "axios",
  "version": "1.6.0",
  "license": "MIT",
  "purl": "pkg:npm/axios@1.6.0"
}

The PBOM (Pipeline Lineage)

The PBOM links that same component to the specific automated process that fetched it, revealing the “why” and “how.”

{
  "artifact": "axios@1.6.0",
  "provenance": {
    "pipeline_id": "GH_RUN_88291",
    "triggered_by": "schedule",
    "resolved_via": "npm-registry-resolver",
    "base_image_digest": "sha256:e2ddd7...",
    "build_environment": {
      "runner": "ubuntu-latest-hosted",
      "privileged_mode": false
    }
  }
}

This limitation is why OX introduced the concept of a Pipeline Bill of Materials (PBOM), which captures build lineage instead of post-build snapshots.

Pass or Fail Signals Without Causality

Security outcomes are typically reduced to binary signals. An artifact either satisfies the policy or it does not. Passing results closes the loop without examining why the artifact differs from previous builds.

If you run a build on Monday and a build on Tuesday with zero code changes, the scanner analyzes them as two independent events.

  • Monday Build: 100MB image. 10 High vulnerabilities.
  • Tuesday Build: 102MB image. 10 High vulnerabilities.

The scanner reports “No Change in Risk.” But the image grew by 2MB. Why?

  • Did a compiler flag change?
  • Did a curl command in the Dockerfile download a larger, potentially malicious binary?
  • Did the base image switch from Alpine 3.18 to 3.19?

Because the source code didn’t change, there is no git diff to warn you. Because the CVE count didn’t change, the scanner doesn’t warn you. The 2MB of mystery code goes straight to production.

Blindness to Pipeline Behavior and Identity

Traditional tools do not treat pipelines as first-class security objects. They do not track which pipeline definition ran, which identity modified build behavior, or which automation influenced dependency resolution.

This is the most dangerous gap. A brand-new backdoor is “clean” by definition because it doesn’t have a CVE ID yet.

Example: The xz backdoor (CVE-2024-3094). Before the CVE was published, every standard scanner marked the infected versions as Green/Passing.

It was because the package name was valid. The version was valid. It just happened to contain malicious code injected during the build process.

A scanner asks: “Is this component on the blacklist?” 

An integrity tool (Active ASPM) asks: “Is this component verified by the pipeline lineage?”

If the pipeline lineage shows that a build artifact contains binary blobs that do not match the source code compilation, or if dependencies appeared that were not pinned in the lockfile, the build should fail, even if the vulnerability scan is 0.

The Drift That Scanners Cannot Explain

Rebuilds introduce drift because they re-execute the build process against a moving environment. The resulting differences are not abstract. They change what runs in production, even when the source tree is identical.

Base Image Resolution Alters the Runtime Environment

Base images are resolved at build time, not at commit time. When a rebuild pulls a newer layer, system libraries change. Certificates rotate, toolchains update, and default configurations shift. From a scanner’s perspective, this appears as a modified dependency set inside the image. What the scanner cannot determine is how that change entered the build or whether it affects code paths the application actually exercises.

Docker Hub showing mutable base image tag resolving to different digests over time, illustrating non-deterministic CI/CD build and pipeline security risk

This shows how a single image tag can point to multiple different digests over time. This proves why rebuilds are often not deterministic even when the code is the same. 

A new vulnerable library may be present but unreachable. A previously safe library may now sit on a reachable execution path. Without knowing how the image was assembled, the scanner cannot distinguish between the two.

Build-Time Flags Modify Execution Without Changing Dependencies

Pipeline logic frequently controls compiler flags, runtime toggles, and build options that never surface in application repositories. These flags influence optimization levels, bounds checks, logging behavior, and memory layout.

A rebuild that changes these parameters can alter how code executes without introducing any new packages. Dependency-based scanners report no material difference, even though the runtime behavior is no longer the same. This is one of the most common sources of silent drift. The software looks identical on paper while behaving differently under load or attack.

# Non-deterministic: "latest" or "20-alpine" points to different 
# digests every few weeks.
FROM node:20-alpine 

WORKDIR /app
COPY package*.json ./

# Non-deterministic: resolves transitive dependencies 
# based on the registry state at 03:00 AM.
RUN npm install 

COPY . .
RUN npm run build

Dependency Resolution Produces Non-Identical Outputs Over Time

Dependency resolution depends on the registry state at the moment the build runs. Artifacts are cached, evicted, replaced, or mirrored asynchronously. A rebuild may resolve a different transitive dependency than an earlier build, even when version constraints remain unchanged.

Scanners capture the new dependency list but provide no explanation for why the change occurred or whether it alters reachable behavior. Teams are left comparing snapshots without understanding causality. At scale, this turns drift analysis into guesswork.

Because scanners cannot explain where drift originates, teams fall back on manual investigation to determine whether changes are safe. This approach assumes that humans can review rebuild-driven change at the rate it occurs.

Why Humans Cannot Review Rebuilds at Machine Speed

That assumption no longer holds. Rebuild frequency has increased beyond the point where human review can serve as a reliable control mechanism. We have entered the era of VibeSec: machines writing code and machines building code.

AI coding assistants generate logic at a speed no human reviewer can match. Simultaneously, automated dependency updaters flood repositories with PRs. Now add “Ghost Builds” to the mix: hundreds of pipelines executing daily without human intervention.

Why Approval Gates Fail in Rebuild-Heavy Pipelines

In a traditional model, a senior engineer approves a Pull Request. But for a rebuild triggered by a base image update, there is often no Pull Request. There is no diff to review. The pipeline simply runs.

Asking a human to verify these builds is impossible. They cannot diff the binary. They cannot audit the ephemeral build environment that existed for 3 minutes at 4:00 AM. Security must operate at machine speed.

When review depends on human interpretation, security either blocks delivery or accepts unexplained change. Neither outcome provides control in rebuild-heavy pipelines.

How Pipeline Lineage Changes the CI/CD Security Model

To operate without manual judgment, CI/CD security must retain the information humans rely on during investigation: how the build ran, which inputs changed, and which identities influenced execution.

To secure  delivery, we must distinguish between the contents of an artifact and the process that created it. An SBOM provides a static index of packages found in the final image, essentially a post-build snapshot. While useful for vulnerability matching, it discards the execution context required to validate integrity.

image5

Securing a high-velocity pipeline requires a digital chain of custody. By treating every build input, from the runner identity to the specific YAML template, as a piece of evidence, teams can verify the entire assembly process. If any “string” in this lineage is broken or unverified, the artifact cannot be trusted, regardless of what the vulnerability scan says.

A PBOM (Pipeline Bill of Materials) captures the complete execution state. It records the specific pipeline definition, the runner identity, build arguments, and environment variables active at runtime. This creates a verifiable lineage linking the final artifact back to its initialization.

Validating Drift Through Context 

In a “Ghost Build” scenario, this lineage data allows engineers to differentiate between malicious tampering and automated maintenance.

Where a standard scanner simply flags a checksum mismatch and a shift in CVE counts, a PBOM isolates the variables that altered the build. It correlates the artifact change directly to specific pipeline events, such as a base image digest update or a dynamic dependency resolution, while confirming that the application source code remained unchanged. This validates that the drift originated from a deterministic, trusted workflow rather than an unverified manual intervention.

Instead of just checking for CVEs, an Active ASPM tool performs an integrity check. If the code did not change but the binary SHA did, it flags the “Lineage Gap.”

# Conceptual CLI output for a lineage-aware security check
$ ox-cli check-integrity --pipeline-id 88291

[!] WARNING: Binary Drift Detected
    - Git Commit: 4a2f1b (No changes)
    - Previous Artifact SHA: sha256:991ab...
    - Current Artifact SHA:  sha256:cc21e...
    
[+] Lineage Analysis:
    - Base Image updated: node:20-alpine -> [Digest Changed]
    - Build Argument 'DEBUG_LEVEL' modified by 'system-bot'
    
[?] Verdict: Drift is documented via Pipeline Lineage. Proceed? [Y/n

Pipeline lineage makes rebuild-driven change explainable, but explanation alone is insufficient when pipelines run. Security posture must update automatically as lineage changes with every build.

Where Active ASPM Fits When Rebuilds Are the Norm

This is the point where CI/CD security models diverge. Once pipelines rebuild, security can no longer rely on discrete checkpoints tied to repositories, pull requests, or releases. The problem becomes maintaining an accurate, up-to-date understanding of application risk while the underlying software keeps changing.

OX approaches this problem through Active Application Security Posture Management, treating posture as a state derived from pipeline execution rather than a static assessment performed at specific lifecycle events.

Posture Derived From Pipeline Execution, Not Repository State

In rebuild-heavy environments, repository state is an incomplete signal. The same commit can produce different artifacts depending on pipeline inputs such as base image resolution, dependency behavior, and shared workflow logic at build time.

OX models posture at the pipeline and artifact level. Each build updates posture based on how the software was assembled, not just which repository it originated from. This allows security teams to reason about risk in terms of actual deployed software rather than assumed equivalence between commits and runtime behavior.

The practical outcome is that posture reflects what is running now, not what was reviewed earlier.

PBOM as the Structural Link Between Pipeline and Risk

SLSA pipeline provenance diagram showing how build templates, external parameters, resolved dependencies, and build caches link to final artifact output

This is an example of pipeline provenance data following the SLSA standard. it shows the specific details a PBOM records to verify that a build process is honest. 

OX uses a Pipeline Bill of Materials (PBOM) to connect build behavior directly to posture. Unlike an SBOM, which describes component inventory after the fact, the PBOM records the pipeline definition, resolved templates, build steps, inputs, and identities involved in producing an artifact.

When an artifact changes without a code diff, PBOM provides the explanation. Teams can see whether the change originated from a base image digest update, a pipeline modification, or dependency resolution behavior during the build.

This shifts investigation away from artifact comparison and toward lineage inspection. Instead of diffing outputs, teams inspect the process that produced them.

Reachability and Exploitability Evaluated Against the Built Artifact

OX does not treat vulnerabilities as static attributes of packages. Risk is evaluated in the context of how the artifact was built and how it executes.

When rebuilds introduce new libraries or alter runtime behavior, PBOM context allows reachability and exploitability to be assessed against the actual execution paths created by the pipeline. This reduces time spent triaging issues that exist only in unused components or unreachable code.

Security effort concentrates on changes that affect real attack surfaces rather than theoretical exposure.

Posture in Rebuild-Driven Systems

In systems where pipelines rebuild frequently, posture decays quickly if it depends on snapshots or manual review. OX keeps posture current by recalculating it as builds occur, using lineage data rather than assuming stability between events.

This model does not require slowing pipelines or inserting human judgment into the critical path. It assumes rebuilds will continue and focuses on explaining and containing their impact.

Active ASPM, as implemented by OX, fits rebuild-heavy environments because it treats pipeline execution as the source of truth. Posture stays aligned with reality not by adding more checks, but by retaining the context required to understand change as it happens.

Enterprise Case Studies: The Real-World Cost of 2025 Supply Chain Failures

To help CISOs and Platform Leads justify the shift toward Pipeline Lineage, we have compiled the most significant CI/CD and supply chain incidents from 2025. These examples show how traditional scanning failed to stop “Ghost Builds” and unverified automated changes.

Incident (2025)Vector: The “Ghost” in the MachineBusiness Impact & LossWhy Traditional Scanners Missed It
tj-actions GitHub BreachAttackers redirected the v1 tag of a popular GitHub Action to a malicious commit. Millions of workflows pulled the “new” v1 automatically.Global Secret Leak: Thousands of enterprise CI/CD secrets (AWS keys, npm tokens) exfiltrated via build logs.The scanner saw a valid version (v1). It didn’t know the lineage of the tag had been hijacked.
Salesloft Drift CompromiseMalicious code injected into a 3rd-party dependency. Stolen OAuth tokens allowed lateral movement into private repos.1.5 Billion Records Exposed: Massive data breach impacting enterprise sales and marketing platforms.The dependency was “known good” and passed static analysis. The identity of the committer was the missing link.
PhantomRaven Campaign126 malicious npm packages targeted global developers to steal GitHub credentials and CI/CD tokens.Systemic Risk: Potential for long-term “sleeper” access to internal build environments.Packages were “clean” (zero CVEs) because they contained new, unique info-stealer logic.
S1ngularity AI-Build AttackAI-powered malware injected into the Nx build system, compromising over 2,100 GitHub accounts.Cloud Infrastructure Exposure: AWS and OpenAI tokens were harvested directly from ephemeral build agents.The build system itself was poisoned. Scanners only looked at the code, not the build environment integrity.

Conclusion

When base images shift and dependencies resolve at runtime, your security posture becomes a moving target. Relying on artifact-only scanning in this environment creates a dangerous gap where you lose visibility into why your binaries are changing and whether those changes were authorized.

Traditional security models fail here because they treat every rebuild as an isolated event. To maintain control over a high-velocity CI/CD fleet, you must transition from auditing static snapshots to tracking execution lineage. By adopting a PBOM (Pipeline Bill of Materials), you move beyond listing ingredients and start verifying the entire assembly process. This context is what allows security teams to distinguish between a routine maintenance build and a supply chain compromise.

If your current tooling cannot explain why an artifact changed when your code did not, your delivery pipeline is unverified. Secure your lineage, or you are simply deploying code you haven’t truly reviewed.

Next Steps for Your Pipeline:

  • Audit your latest tags: Identify how many production Dockerfiles rely on non-deterministic base images.
  • Evaluate your drift: Compare the SHA of your last three scheduled builds; if they differ without a commit, your pipeline is non-deterministic.
  • Verify with OX: See how Active ASPM tracks PBOM lineage to automate the validation of every build.

FAQ

  1. 1. Do CI/CD rebuilds really change software without code changes?

    Yes. Rebuilds re-execute the build process against a changing environment. Base image layers, dependency resolution, and pipeline behavior can differ between runs, producing artifacts with different runtime characteristics even when source code is identical. Active ASPM with PBOM is required to explain and track these changes reliably.

  2. 2. Are SBOMs enough for CI/CD security?

    No. SBOMs describe what components exist inside an artifact, but they do not explain how those components entered the build or which pipeline behavior introduced them. When the pipeline is the source of change, SBOMs lack the context needed to assess trust or exploitability. Active ASPM fills this gap by tracking pipeline lineage.

  3. 3. Why is pipeline lineage necessary for CI/CD security?

    Pipeline lineage links artifacts to the exact build behavior that produced them. Without it, teams cannot determine why an artifact changed, whether the change was expected, or how to contain it safely. Active ASPM uses PBOM to make lineage part of the security model.

  4. 4. Can CI/CD security be automated safely?

    Yes, but only when automation is driven by build context rather than artifact snapshots alone. Safe automation depends on knowing what caused a change and how far its impact spreads. Active ASPM provides that context, allowing remediation to occur at pipeline speed without introducing uncontrolled risk.

Tags:

post banner image

Run Every Security Test Your Code Needs

Pinpoint, investigate and eliminate code-level issues across the entire SDLC.

GET A PERSONALIZED DEMO
Frame 2085668530

Subscribe to Our Newsletter

Stay updated with the latest SaaS insights, tips, and news delivered straight to your inbox.

Security Starts at the Source