> ## Documentation Index
> Fetch the complete documentation index at: https://mcpjam-claude-project-secrets-implementation-e3rb5i.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# CI / CD

> Run MCP health checks, conformance suites, and evals in GitHub Actions, GitLab CI, and other CI environments

Run `mcpjam` in CI to catch MCP server regressions on every push. The examples below cover GitHub Actions and GitLab CI, but the same commands work in any CI environment.

## GitHub Actions

### Authentication

There are three ways to authenticate in CI, depending on your server setup.

#### Option 1: Headless OAuth login

Best when your server supports OAuth with auto-consent (no interactive login page). The workflow obtains a fresh access token on every run.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: OAuth login (headless)
        run: |
          set -euo pipefail
          npx -y @mcpjam/cli@latest oauth login \
            --url ${{ secrets.MCP_SERVER_URL }} \
            --protocol-version 2025-11-25 \
            --registration dcr \
            --auth-mode headless \
            --format json > /tmp/oauth-result.json
          TOKEN=$(jq -r '.credentials.accessToken // empty' /tmp/oauth-result.json)
          rm -f /tmp/oauth-result.json
          if [ -z "$TOKEN" ]; then
            echo "::error::OAuth login did not return an access token"
            exit 1
          fi
          echo "::add-mask::$TOKEN"
          echo "MCP_TOKEN=$TOKEN" >> "$GITHUB_ENV"

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json
```

#### Option 2: Refresh token

Best when you already have a refresh token from a previous `oauth login`. Refresh tokens are long-lived and safe to store as secrets. The CLI handles the token exchange automatically.

**Secrets needed:**

| Secret              | Description                                         |
| ------------------- | --------------------------------------------------- |
| `MCP_SERVER_URL`    | Your MCP server URL                                 |
| `MCP_REFRESH_TOKEN` | OAuth refresh token from a previous login           |
| `MCP_CLIENT_ID`     | OAuth client ID (required with refresh tokens)      |
| `MCP_CLIENT_SECRET` | OAuth client secret (if the client is confidential) |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: |
          npx -y @mcpjam/cli@latest server doctor \
            --url ${{ secrets.MCP_SERVER_URL }} \
            --refresh-token ${{ secrets.MCP_REFRESH_TOKEN }} \
            --client-id ${{ secrets.MCP_CLIENT_ID }} \
            --client-secret ${{ secrets.MCP_CLIENT_SECRET }} \
            --format json
```

<Tip>
  To get a refresh token, run `mcpjam oauth login` locally with `--format json` and grab `.credentials.refreshToken` from the output.
</Tip>

#### Option 3: Static API key

Best when your server uses a non-expiring API key instead of OAuth.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |
| `MCP_API_KEY`    | Static API key      |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --access-token ${{ secrets.MCP_API_KEY }} --format json
```

#### Option 4: No auth

Some servers don't require authentication at all.

**Secrets needed:**

| Secret           | Description         |
| ---------------- | ------------------- |
| `MCP_SERVER_URL` | Your MCP server URL |

```yaml theme={"theme":"css-variables"}
name: MCP Health Check

on:
  push:
    branches: [main]
  pull_request:

jobs:
  mcp-doctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Run doctor
        run: npx -y @mcpjam/cli@latest server doctor --url ${{ secrets.MCP_SERVER_URL }} --format json
```

### Tool surface diffing

Snapshot your tool surface before and after a deploy to catch breaking changes (renamed parameters, changed descriptions, removed tools).

```yaml theme={"theme":"css-variables"}
      - name: Snapshot before
        run: npx -y @mcpjam/cli@latest server export --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json > before.json

      # your deploy step here

      - name: Snapshot after
        run: npx -y @mcpjam/cli@latest server export --url ${{ secrets.MCP_SERVER_URL }} --access-token $MCP_TOKEN --format json > after.json

      - name: Diff
        run: diff <(jq -S . before.json) <(jq -S . after.json)
```

### OAuth conformance suite

Run the full registration x protocol version x auth mode matrix from a config file and output JUnit XML for test reporters.

```yaml theme={"theme":"css-variables"}
      - name: OAuth conformance
        run: |
          npx -y @mcpjam/cli@latest oauth conformance-suite \
            --config ./oauth-matrix.json \
            --reporter junit-xml > report.xml

      - name: Upload test report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: oauth-conformance
          path: report.xml
```

See [OAuth Conformance](/cli/oauth-conformance) for details on the config file format.

### Protocol conformance suite

Run a repeatable matrix of protocol check selections from a config file and publish JUnit XML.

```yaml theme={"theme":"css-variables"}
      - name: Protocol conformance
        run: |
          npx -y @mcpjam/cli@latest protocol conformance-suite \
            --config ./protocol-conformance.json \
            --reporter junit-xml > protocol-report.xml

      - name: Upload protocol report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: protocol-conformance
          path: protocol-report.xml
```

### MCP Apps conformance suite

Run the server-side MCP Apps surface checks from a config file and publish JUnit XML for CI dashboards.

```yaml theme={"theme":"css-variables"}
      - name: MCP Apps conformance
        run: |
          npx -y @mcpjam/cli@latest apps conformance-suite \
            --config ./apps-conformance.json \
            --reporter junit-xml > apps-report.xml

      - name: Upload apps report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: apps-conformance
          path: apps-report.xml
```

Single-run `protocol conformance`, `oauth conformance`, and `apps conformance` also accept `--reporter junit-xml` when you only need one target/check selection instead of a suite config file.

***

## GitLab CI

The same CLI commands work in GitLab CI. The examples below use GitLab CI/CD variables for secrets and `.gitlab-ci.yml` syntax.

### Authentication

#### Headless OAuth login

```yaml theme={"theme":"css-variables"}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
  script:
    - |
      npx -y @mcpjam/cli@latest oauth login \
        --url "$MCP_SERVER_URL" \
        --protocol-version 2025-11-25 \
        --registration dcr \
        --auth-mode headless \
        --format json > /tmp/oauth-result.json
      TOKEN=$(jq -r '.credentials.accessToken // empty' /tmp/oauth-result.json)
      rm -f /tmp/oauth-result.json
      if [ -z "$TOKEN" ]; then
        echo "OAuth login did not return an access token"
        exit 1
      fi
      export MCP_TOKEN="$TOKEN"
    - npx -y @mcpjam/cli@latest server doctor --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

#### Refresh token

```yaml theme={"theme":"css-variables"}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_REFRESH_TOKEN: $MCP_REFRESH_TOKEN
    MCP_CLIENT_ID: $MCP_CLIENT_ID
    MCP_CLIENT_SECRET: $MCP_CLIENT_SECRET
  script:
    - |
      npx -y @mcpjam/cli@latest server doctor \
        --url "$MCP_SERVER_URL" \
        --refresh-token "$MCP_REFRESH_TOKEN" \
        --client-id "$MCP_CLIENT_ID" \
        --client-secret "$MCP_CLIENT_SECRET" \
        --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

#### Static API key

```yaml theme={"theme":"css-variables"}
mcp-health-check:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_API_KEY: $MCP_API_KEY
  script:
    - npx -y @mcpjam/cli@latest server doctor --url "$MCP_SERVER_URL" --access-token "$MCP_API_KEY" --format json
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
```

### Tool surface diffing

Snapshot your tool surface before and after a deploy to catch breaking changes.

```yaml theme={"theme":"css-variables"}
mcp-tool-diff:
  image: node:20
  variables:
    MCP_SERVER_URL: $MCP_SERVER_URL
    MCP_TOKEN: $MCP_TOKEN
  script:
    - npx -y @mcpjam/cli@latest server export --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json > before.json
    # your deploy step here
    - npx -y @mcpjam/cli@latest server export --url "$MCP_SERVER_URL" --access-token "$MCP_TOKEN" --format json > after.json
    - jq -S . before.json > /tmp/before-sorted.json
    - jq -S . after.json > /tmp/after-sorted.json
    - diff /tmp/before-sorted.json /tmp/after-sorted.json
    - rm -f /tmp/before-sorted.json /tmp/after-sorted.json
```

### OAuth conformance suite

```yaml theme={"theme":"css-variables"}
mcp-oauth-conformance:
  image: node:20
  script:
    - |
      npx -y @mcpjam/cli@latest oauth conformance-suite \
        --config ./oauth-matrix.json \
        --reporter junit-xml > report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml
```

See [OAuth Conformance](/cli/oauth-conformance) for details on the config file format.

***

## Evals in CI

There are two ways to wire MCPJam evals into a pipeline: trigger a **hosted eval run** with the CLI, or run evals **locally with the SDK** and upload the results. Both authenticate with an MCPJam API key (`sk_…`) from **Settings → API keys**.

### Trigger a hosted eval suite

`mcpjam cloud eval run` starts an asynchronous run of a suite that lives in your MCPJam project. Without `--wait`, it prints a launch receipt and returns immediately. In CI, add `--wait` and `--out` to write a structured JSON report after every launched run reaches a terminal state.

**Secrets needed:**

| Secret           | Description             |
| ---------------- | ----------------------- |
| `MCPJAM_API_KEY` | MCPJam API key (`sk_…`) |

```yaml theme={"theme":"css-variables"}
      - name: Run hosted eval
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
        run: |
          npx -y @mcpjam/cli@latest cloud eval run \
            --suite "Nightly regression" \
            --project "My project" \
            --wait \
            --out eval-report.json \
            --format json > eval-result.json
          echo "Completed run $(jq -r '.runs[0].id' eval-result.json)"

      - name: Gate and write JUnit
        # Always run this step, even if the run above exited 1 on a failed
        # verdict: a default `if:` is `success()`, and skipping this step
        # would lose the gate's configurable policy and its JUnit report
        # exactly when the run failed. The job still ends up failed either
        # way — the run step above already exited nonzero.
        if: always()
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
        run: |
          RUN_ID=$(jq -r '.runs[0].id' eval-result.json)
          npx -y @mcpjam/cli@latest cloud eval gate \
            --run "$RUN_ID" \
            --project "My project" \
            --wait \
            --min-pass-rate-percent 100 \
            --reporter junit-xml \
            --out eval-report.xml

      - name: Upload eval reports
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: hosted-eval
          path: |
            eval-report.json
            eval-report.xml
```

In human format, `eval run` prints a `View:` line after the payload so you can open the run directly from the terminal:

```text theme={"theme":"css-variables"}
View: https://app.mcpjam.com/evals/suite/<suiteId>/runs/<runId>?project=<projectId>
```

This line is only emitted in human format — `--format json` output is unchanged, so scripts that parse the JSON stream are unaffected.

Use `--wait-timeout <ms>` to replace the 10-minute default. `--out` defaults to the structured JSON format; add `--reporter junit-xml` to write JUnit XML instead, or `--reporter html` for a self-contained HTML page (decision summary + failures only — traces, parity, and history are paid tiers not included here). When `--reporter` is present, the same report is also written to stdout.

<Warning>
  **`eval run --wait` sets a verdict-based exit code.** `0` pass, `1` a completed run's verdict failed (the ONLY condition that produces `1`), `2` usage error or an invalid suite file, `3` auth failed (no credential, or the platform rejected it, at launch or mid-wait), `4` a connection/setup failure this CLI itself observed before evaluation ran (or a local `--out` write failure), `5` no valid verdict — `inconclusive`, a null/unrecognized result, a run status of failed/cancelled/timed-out, or a wait that hit its deadline. A multi-target launch merges these worst-of across every waited run, in the order `1 > 3 > 4 > 5 > 0`.

  Retry guidance: `4` and `5` mean infrastructure, or an absence of observation — nothing here says the server is wrong, so retrying the CI job is reasonable. But a bare re-run is not automatically safe: `eval run --suite` only dedupes against an in-flight or already-completed launch when you pass a **stable** `--idempotency-key`, and exit `5` can mean the run is still running (a wait that hit its deadline) — without that key, a retry can start a second paid run alongside the first rather than resuming it. Pass `--idempotency-key` (or poll/resume the run id already in the receipt) before retrying on `4` or `5`. `3` means fix the credential first; it poisons every other observation in the same launch. Never retry blindly on `1` — that code is reserved for a run the platform actually graded as failed.

  Either `eval run --wait` or `eval gate` already fails the job on its own — the `eval gate` step above adds a configurable pass/fail *policy* (thresholds, per-scorer gates) and baseline comparison on top of the same verdict, so keep it when you want more than "did this run's own verdict pass".
</Warning>

<Note>
  `eval gate` sets a verdict-based exit code, and writes its report before doing so: `0` passed **or waived**, `1` an eval verdict failed, `2` usage error, `3` incomplete or non-gateable. Infrastructure conditions never map to `1`, so retrying on `3` is safe. This is a **different, four-code contract from `eval run --wait`** above — `gate`'s `3` means "incomplete", not the six-code scheme's `3` ("auth failed"), and the two are deliberately not unified (see the [CLI reference](/cli/reference#cloud-eval-gate) for why). `eval status` also prints a `View:` line in human format, identical to the one `eval run` prints.
</Note>

#### Waiving a gate

A run whose gate failed can be overridden by an authorized user until an expiry
they name, so a release is not blocked while a known regression is being fixed:

```bash theme={"theme":"css-variables"}
mcpjam cloud eval gate waive --run "$RUN_ID" --reason "hotfix ships today; tracked in ENG-4821" --expires-in 3d
```

`eval gate` then exits `0` and reports the outcome as `waived`. It is **not**
reported as a pass: the run keeps its failed result, the failing verdicts stay
in the report, and the waiver — who granted it, why, and until when — is named
in every artifact the command writes, including the JUnit XML your CI job
uploads (as a `<skipped>` element, so it neither fails the build nor renders as
a clean green row).

Only a real verdict failure is waivable. A cancelled run, a `--wait` timeout, or
a network failure still exits `3` with a waiver in place — those established
nothing, and a waiver granted for a regression is not consent to ship on an
infrastructure failure.

Waivers expire, and expiry is enforced on both sides: the platform republishes
the GitHub Check Run when the waiver lapses, and the CLI re-derives the expiry
itself rather than trusting the platform's answer. `mcpjam cloud eval gate
unwaive --run "$RUN_ID"` ends one early.

<Warning>
  The waiver reason is stored **unredacted** and readable by anyone who can see
  the suite, for as long as the suite exists. Never put secrets, tokens, or
  customer data in it.
</Warning>

#### Decision summary

`eval run --wait`, `eval status`, `eval gate` and `eval compare` all read one versioned object — the **run decision summary** — and every output format restates it. In `--format human` it is printed as a block; in `--format json` and in `--out`/`--reporter` artifacts it rides on the report as `decisionSummary`, verbatim.

```text theme={"theme":"css-variables"}
Decision summary: failed (verdict policy v2) — 2/3 case variants passed, 1 failed
  Why: a case did not meet its pass threshold
  Diagnostics: 2 non-passing of 9 trials examined (the complete set)
  Fetch order (c_orders, iteration 2) — failed
    First failed stage: Tool call — the call arguments did not match what the case expects
    Failure category: call arguments
    Expected tool calls: fetch_order
    Observed failure: server rejected arguments
    Evidence at Tool call: span ids span-call-2; reasons order_id must be a string
    Trace: /projects/prj_1/eval-runs/run_7/iterations/it_2/trace
    Next action: review the authored arguments against the tool input schema
  Setup abort (c_setup, iteration 5) — failed
    First failed stage: none was established — the run never reached the server's stages
    Failure category: setup
    Trace: /projects/prj_1/eval-runs/run_7/iterations/it_5/trace
    Next action: check the server connection and environment configuration
```

Four things about it are worth knowing before you script against it.

**The counts carry the population they count.** `measurementUnit` is `caseVariant` under verdict policy v2 — one case under one provider/model, with repetitions as *trials inside it* — and `trial` on a legacy percent-threshold run. A 3-case suite with 5 repetitions is legitimately "3" under one unit and "15" under the other, so a count quoted without its unit is not a fact.

**The summary explains the verdict; it never re-decides it.** Under policy v2 the run's own decision is the authority for the verdict, the rates, the validity phase and the per-case aggregation, and it is carried through on `decision`. The per-trial diagnostics sit *underneath* that: a case can pass with a failing trial in it, so tallying the diagnostics gives a different answer than the platform reached.

**`notEstablished` is not a failure.** It is a fourth verdict meaning no verdict exists at all — the run is unfinished, it stopped before finishing, or its decision could not be read. `undecided.reason` says which. It is also not `inconclusive`, which *is* a decision: the validity phase ran and withheld a verdict because the run did not measure the server well enough.

**A page of diagnostics says whether it is the whole story.** `diagnostics.complete` is true only when the listed trials are the run's entire non-passing set, and `scannedIterations` says how many were examined — so an empty list from a complete page ("nothing failed") is distinguishable from an empty list from a partial one ("we did not look").

Evidence is scoped to the claim it supports: for a measured failure the span ids, prompt indexes and reasons come from the first failed stage's row alone, and a setup abort or evaluator error keeps a stage-less pointer rather than naming a stage nothing established.

`eval status` prints the block only when a terminal run did not pass, and `eval gate` and `eval compare` write theirs to stderr alongside the gate report. `--format json` is unaffected in every case: it stays exactly one parseable document. If the summary cannot be fetched, the block is omitted rather than failing the command.

Hosted runs execute LLM iterations on the platform and consume your organization's credits or configured provider keys. See the [`cloud eval` command reference](/cli/reference#cloud-eval-commands) for the full surface, including `cloud eval judge` (request LLM-as-judge grading on a finished run), `cloud eval validate` (offline suite-file validation), `cloud eval export` (write a hosted suite to a local file), `cloud eval checks list/connect` (GitHub Checks integration), and more.

### Upload SDK eval results

If you instead run evals inside your own CI job with [`@mcpjam/sdk`](/sdk/concepts/running-evals) (`EvalTest` / `EvalSuite`), set `MCPJAM_API_KEY` and results upload automatically to the CI Evals dashboard (pass-rate trends, per-model breakdowns, and a full trace per iteration):

```yaml theme={"theme":"css-variables"}
      - name: Run SDK evals
        env:
          MCPJAM_API_KEY: ${{ secrets.MCPJAM_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: npx vitest run evals/
```

See [Save Results to MCPJam](/sdk/concepts/saving-results) for auto-save, the manual reporting APIs, CI metadata (branch, commit SHA, run URL), and artifact upload (JUnit XML, Jest/Vitest JSON).