Skip to content

CI-ARCHITECTURE.md

Latest commit

 

History

History
265 lines (188 loc) · 15.3 KB

CI-ARCHITECTURE.md

File metadata and controls

265 lines (188 loc) · 15.3 KB
Raw

CI Architecture

Overview of the GitHub Actions CI/CD ecosystem for Apache Camel.

Workflow Overview

PR opened/updated
       │
       ├──► pr-id.yml ──► pr-commenter.yml (welcome message)
       │
       ├──► pr-build-main.yml (Build and test)
       │        │
       │        ├── regen.sh (full build, no tests)
       │        ├── incremental-build (test affected modules)
       │        │       ├── File-path analysis
       │        │       ├── POM dependency analysis
       │        │       └── Extra modules (/component-test)
       │        │
       │        └──► pr-test-commenter.yml (post unified comment)
       │
       └──► sonar-build.yml ──► sonar-scan.yml (SonarCloud analysis)
                                    [currently disabled — INFRA-27808]

PR comment: /component-test kafka http
       │
       └──► pr-manual-component-test.yml
                │
                └── dispatches "Build and test" with extra_modules

Workflows

pr-build-main.yml — Build and test

  • Trigger: pull_request (main branch), workflow_dispatch
  • Matrix: JDK 17, 21, 25 (25 is experimental)
  • Steps:
    1. Full build via regen.sh (mvn install -DskipTests -Pregen)
    2. Check for uncommitted generated files
    3. Run incremental tests (only affected modules)
    4. Upload test comment as artifact
  • Inputs (workflow_dispatch): pr_number, pr_ref, extra_modules, skip_full_build

pr-test-commenter.yml — Post CI test comment

  • Trigger: workflow_run on "Build and test" completion
  • Purpose: Posts the unified test summary comment on the PR
  • Why separate: Uses workflow_run to run in base repo context, allowing comment posting on fork PRs (where GITHUB_TOKEN is read-only)

pr-manual-component-test.yml — /component-test handler

  • Trigger: issue_comment with /component-test prefix
  • Who: MEMBER, OWNER, or CONTRIBUTOR only
  • What: Resolves component names to module paths, dispatches the main "Build and test" workflow with extra_modules and skip_full_build=true
  • Build: Uses a quick targeted build (-Dquickly) of the requested modules and their dependencies instead of the full regen.sh build

pr-id.yml + pr-commenter.yml — Welcome message

  • Trigger: pull_request (all branches)
  • Purpose: Posts the one-time welcome message on new PRs
  • Why two workflows: pr-id.yml runs in PR context (uploads PR number), pr-commenter.yml runs via workflow_run with write permissions

main-build.yml — Main branch build

  • Trigger: push to main, camel-4.18.x, camel-4.22.x
  • Steps: Same as PR build but without comment posting

sonar-build.yml + sonar-scan.yml — SonarCloud PR analysis

  • Status: Temporarily disabled (INFRA-27808 — SonarCloud quality gate adjustment pending)
  • Trigger: pull_request (main branch) → workflow_run on SonarBuild completion
  • Why two workflows: sonar-build.yml runs in PR context (builds with JaCoCo coverage on core modules, uploads compiled classes artifact), sonar-scan.yml runs via workflow_run with secrets access to run the Sonar scanner and post results
  • Coverage scope: Currently limited to core modules (camel-api, camel-core, etc.) and coverage aggregator. Component coverage planned for future integration with incremental-build.sh module detection

Other workflows

  • pr-labeler.yml — Auto-labels PRs based on changed files
  • pr-doc-validation.yml — Validates documentation changes
  • pr-cleanup-branches.yml — Cleans up merged PR branches
  • alternative-os-build-main.yml — Tests on non-Linux OSes
  • check-container-versions.yml — Checks test container version updates
  • generate-sbom-main.yml — Generates SBOM for releases
  • security-scan.yml — Security vulnerability scanning

Actions

incremental-build

The core test runner. Determines which modules to test using:

  1. File-path analysis: Maps changed files to Maven modules
  2. POM dependency analysis (dual detection):
    • Grep-based: For parent/pom.xml changes, detects property changes and finds modules that explicitly reference the affected properties via ${property} in their pom.xml files
    • Scalpel-based: Uses Maveniverse Scalpel (Maven extension) for effective POM model comparison — catches managed dependencies, plugin version changes, BOM imports, and transitive dependency impacts that the grep approach misses
  3. Extra modules: Additional modules passed via /component-test

Both detection methods run in parallel. Their results are merged (union), deduplicated, and tested. If Scalpel fails (build error, runtime error), the script falls back to grep-only with no regression.

The script also:

  • Detects tests disabled in CI (@DisabledIfSystemProperty(named = "ci.env.name"))
  • Applies an exclusion list for generated/meta modules
  • Checks for excluded modules with associated integration tests (via manual-it-mapping.txt) and advises contributors to run them manually
  • Reports recovered flaky tests (see below)
  • Generates a unified PR comment with all test information
  • Parses Maven reactor output from incremental-test.log and reports per-module elapsed time, total reactor duration, and the top 20 slowest modules (see reactor_timing.sh)

Unit tests for reactor timing parsing live in reactor_timing_test.sh.

Recovered flake reporting (collect-flakes.py)

Surefire retries failing tests: surefire.rerunFailingTestsCount defaults to 2 in the full profile of parent/pom.xml, and both CI systems pass it again explicitly. A test that fails and then passes within those attempts is a recovered flake. The build stays green and nothing appears in the console output, so without this step the retry is invisible.

collect-flakes.py runs on the always-path (a recovered flake means exit code 0, so it cannot live in the failure branch where parse_errors.sh runs). It walks **/target/{surefire,failsafe}-reports/TEST-*.xml and reports every <testcase> carrying <flakyFailure>/<flakyError> children. Tests with <rerunFailure>/<rerunError> failed every attempt and already fail the build, so they are deliberately excluded.

Two outputs:

  • A section appended to the PR comment and the job summary, naming the module, test, attempt count and first failure message. Nothing is emitted when no test was retried.
  • flakes.json, uploaded as flakes-java-<version> on PRs and flakes-main-java-<version> on main. Develocity's flaky-test data does not cover fork PRs (.mvn/develocity.xml publishes build scans only when authenticated), so this artifact is the only per-PR record.

Notes:

  • The section names its JDK (flake-label on the action, --label on the script, also recorded in flakes.json). The PR-comment artifact is uploaded with overwrite: true across the JDK matrix on the grounds that the content is identical between entries. Flake data is the one part that is not: if JDK 17 flakes and JDK 25 does not, whichever finishes last decides what the comment shows. The label means the reader can tell which entry a shown flake came from, and the per-JDK artifacts remain the complete record.

  • No time figure is reported. Surefire records no per-attempt timing, and <testcase time> reflects only the final successful attempt. Estimating cost from it would understate timeout-driven flakes, which are the common kind.

  • The script declares its dependencies inline via PEP 723 and must be run with uv run; plain python3 ignores the metadata block. uv is installed by the action.

  • XML is parsed with defusedxml, with forbid_dtd=True passed explicitly — the default only forbids entity declarations, which would let a bare <!DOCTYPE .. SYSTEM ..> through. A pre-parse byte scan for <!DOCTYPE is not sufficient either: it misses a UTF-16 document, where the marker is interleaved with NUL bytes. testdata/TEST-utf16-doctype-rejected.xml covers that case.

  • Failures are logged and skipped. This step must never be the reason a job fails.

Unit tests live in test_collect_flakes.py and run in pr-ci-scripts-validation.yml.

install-mvnd

Installs the Maven Daemon (mvnd) for faster builds.

install-packages

Installs system packages required for the build.

PR Labels

Label Effect
skip-tests Skip all tests
test-dependents Force testing dependent modules even if threshold exceeded

CI Environment

The CI sets -Dci.env.name=github.com via MVND_OPTS (in install-mvnd). Tests can use @DisabledIfSystemProperty(named = "ci.env.name") to skip flaky tests in CI. The test comment warns about these skipped tests.

POM Dependency Detection: Dual Approach

Grep-based detection (legacy)

The grep approach searches for ${property-name} references in module pom.xml files. It has known limitations:

  1. Managed dependencies without explicit <version> — Modules inheriting versions via <dependencyManagement> without declaring <version>${property}</version> are missed.
  2. Maven plugin version changes — Plugin version properties consumed in parent/pom.xml via <pluginManagement> are invisible to child modules.
  3. BOM imports — Modules using artifacts from a BOM are not linked to the BOM version property.
  4. Transitive dependency changes — Only direct property references are detected.
  5. Non-property version changes — Structural <dependencyManagement> edits without property substitution are not caught.

Scalpel-based detection (new)

Maveniverse Scalpel is a Maven core extension that compares effective POM models between the base branch and the PR. It resolves all 5 grep limitations by:

  • Reading old POM files from the merge-base commit (via JGit)
  • Comparing properties, managed dependencies, and managed plugins between old and new POMs
  • Resolving the full transitive dependency graph to find all affected modules
  • Detecting plugin version changes via project.getBuildPlugins() comparison

Scalpel runs in report mode (-Dscalpel.mode=report), writing a JSON report to target/scalpel-report.json without modifying the Maven reactor. The report includes affected modules with reasons (SOURCE_CHANGE, POM_CHANGE, TRANSITIVE_DEPENDENCY, MANAGED_PLUGIN).

Dual-detection strategy

Both methods run in parallel. Results are merged (union) before testing. This lets us:

  1. Validate Scalpel — Compare what each method detects across many PRs
  2. No regression — If Scalpel fails, grep results are still used
  3. Gradual migration — Once Scalpel is validated, grep can be removed

Scalpel is configured permanently in .mvn/extensions.xml. On developer machines it is a no-op (disabled via -Dscalpel.enabled=false in .mvn/maven.config). The CI script overrides this with -Dscalpel.enabled=true. The mvn validate with report mode adds ~60-90 seconds in CI.

Scalpel is only invoked when a subdirectory pom.xml is changed (e.g. parent/pom.xml, components/camel-kafka/pom.xml). Changes to the root pom.xml are excluded because it contains build-infrastructure config (license plugin, checkstyle, etc.) that does not affect module compilation or test behavior. Without this filter, Scalpel would report every module as affected since they all inherit from the root POM.

Scalpel features used for shadow comparison

  • Source-set-aware propagation: Distinguishes test-jar dependencies from regular dependencies. A module that depends only on another module's test-jar (e.g., camel-core's test-jar with test utilities) is propagated through the TEST source set, not the MAIN source set. This prevents a change to test utilities from triggering tests in all ~500 modules that depend on camel-core.
  • skipTestsForDownstreamModules: Allows specifying modules whose tests should be skipped when they appear as downstream dependents (mirrors the EXCLUSION_LIST in incremental-build.sh). This gives Scalpel an accurate picture of what skip-tests mode would actually test.

Shadow comparison

Scalpel runs in shadow mode: it observes what skip-tests mode would have done and reports it in a collapsible section of the PR comment, without affecting actual test execution. This allows the team to validate Scalpel's decisions across many PRs before switching to Scalpel-driven test execution.

The shadow comparison section shows:

  • How many modules Scalpel would test (direct + downstream)
  • How many downstream modules would have tests skipped (generated code, meta-modules)
  • Set differences: modules only Scalpel found vs modules only the current approach found
  • The full list of modules in each category

The comparison is apples-to-apples: the current approach's reactor is filtered through the EXCLUSION_LIST before comparing, so both sides exclude the same meta/generated modules (catalog, jbang, docs, etc.).

Configuration notes

The script overrides fullBuildTriggers to empty (-Dscalpel.fullBuildTriggers=) because Scalpel's default (.mvn/**) would trigger a full build whenever .mvn/extensions.xml itself changes (e.g., Dependabot bumping Scalpel).

The grep-based script fetches the PR diff via the GitHub REST API (unchanged). Scalpel uses local git history to compare effective POM models and detect source file changes — the CI workflow progressively deepens the shallow clone (50 → 200 → 1000 → full) until the merge-base is reachable. Scalpel runs for all PRs (not just POM changes) so the shadow comparison covers source-only changes too, building confidence for a future switch to Scalpel-driven builds. Scalpel disables its built-in JGit fetch (-Dscalpel.fetchBaseBranch=false) because the CI workflow already pre-fetches the base branch with native git (Scalpel uses JGit 5.x which has limited shallow clone support).

Manual Integration Test Advisories

Some modules are excluded from CI's -amd expansion (the EXCLUSION_LIST) because they are generated code, meta-modules, or expensive integration test suites. When a contributor changes one of these modules, CI cannot automatically test all downstream effects.

The file manual-it-mapping.txt (co-located with the incremental build script) maps source modules to their associated integration test suites. When a changed module has a mapping entry, CI posts an advisory in the PR comment:

You modified dsl/camel-jbang/camel-jbang-core. The related integration tests in dsl/camel-jbang/camel-jbang-it are excluded from CI. Consider running them manually:

mvn verify -f dsl/camel-jbang/camel-jbang-it -Djbang-it-test

To add new mappings, edit manual-it-mapping.txt using the format:

source-artifact-id:it-module-path:command

Multi-JDK Artifact Behavior

All non-experimental JDK matrix entries (17, 21) upload the CI comment artifact with overwrite: true. This ensures a comment is posted even if one JDK build fails. Since the comment content is identical across JDKs (same modules are tested regardless of JDK version), last writer wins.

Comment Markers

PR comments use HTML markers for upsert (create-or-update) behavior:

  • <!-- ci-tested-modules --> — Unified test summary comment