Daily job-level failures across all monitored repos, stacked by category. Click a category in the legend to toggle it.
Top 15 (workflow, job pattern) pairs by median wall time across the selected window. Useful for identifying CI cost-optimization targets.
Jobs that have been failing for 3+ consecutive days with zero successful runs in the last 14 days. These are the silent CI-cost burners — fix them, exclude them from the matrix, or accept them as known-bad and add to a quarantine list.
multi_arch_release.yml run plus its dispatched downstreams
(test_artifacts, PyTorch wheels). Columns follow pipeline execution order across
12 lanes (ROCm Linux Builds → Linux Packaging → Publish → Tests → PyTorch Builds → PyTorch Tests → same for Windows).
Click ▸ on a column header to expand Math Libs / Test Artifacts / PyTorch Builds / PyTorch Tests into per-arch dots.
Click any row to open the full pipeline tree, or the ⬇ Downloads
button for install commands pinned to that exact build.
| Loading multi-arch release history… |
Lanes: ROCm Linux Builds · ROCm Linux Packaging · Linux Publish · Linux Tests · Linux PyTorch Builds · Linux PyTorch Tests · Linux JAX Builds · Linux JAX Tests · ROCm Windows Builds · ROCm Windows Packaging · Windows Publish · Windows Tests · Windows PyTorch Builds · Windows PyTorch Tests — each lane is shaded differently (cool blues/greens for Linux, warmer ambers/reds for Windows) and starts with a thicker primary-tinted border.
LINUX / WINDOWS tags above each column header indicate the OS.
Cell badges show passed / total jobs for that column on that ROCm version (multiple workflow runs for the same nightly are summed together).
Build jobs are routed by job-name path (Linux::release / Build Artifacts / {stage}); Test jobs by extracted Test {component} name; PyTorch Builds jobs by source workflow_id (multi_arch_release_*_pytorch_wheels.yml) with per-(py × torch) sub-cells; PyTorch Tests jobs (Test PyTorch | {arch}) split out into their own lane with per-(py × torch) cells and per-arch drill-down. JAX Builds jobs come from multi_arch_release_linux_jax_wheels.yml with per-(py × jax) sub-cells; JAX Tests jobs (Test JAX | {arch}) split out into their own lane so a failing JAX test no longer marks the Builds column red, with each py × jax cell broken down by test size (full today; more sizes surface automatically as CI adds them).
| Loading… |
|---|
| Loading CI HUD data... |
Pre-merge (presubmit) CI quality for open pull requests. Companion to CI HUD, which shows post-merge CI on the default branch.
| Loading… |
|---|
| Loading in-flight PR data... |
| Architecture | ||||||||
|---|---|---|---|---|---|---|---|---|
| Loading ASAN data... | ||||||||
| Architecture | ||||||||
|---|---|---|---|---|---|---|---|---|
| Loading TSAN data... | ||||||||
ROCm/rockrel's multi_arch_release.yml — click
Multi-Arch pipeline for that nightly's build tree, or source for the GitHub run.
gfx942 → MI300A + MI308X); the per-arch numbers
sum them and the table below breaks them back out.
Click a point to open that run in the
Nightly Tests tab.
weekly in ROCm/rocm-tests, which are too slow for the nightly cadence.
Same layout as Nightly Tests:
a row per GPU platform with Tests Run /
Pass /
Fail /
Error, and a ▸ caret to
drill into per-test results. Because these run roughly weekly, the window defaults to 90 days.
ROCm/rocm-systems.
Builds run on standard runners and are blocking.
Tests run on linux-mi455-gpu-rocm runners and are non-blocking.
Shows queue times, run durations, and success rates for both.
| Loading MI455 builds... |
| Loading MI455 tests... |
ROCm/rocm-systems.
Open bump PRs awaiting merge — updating submodule versions in TheRock. Click ▸ on a tracked-submodule row to see what upstream PRs that bump pulls in.
| PR | Submodule | Upstream span | Title | Author | Status | CI Checks | Age | Updated |
|---|---|---|---|---|---|---|---|---|
| Loading bump PR data... | ||||||||
Each bump-pr issue mapped to the bump PR that surfaced it, with an AI-synthesized root cause, commenter quotes, and the applied fix. Rows that share a Bump PR group together so you can see all issues caused by one bump. Scoped by the Date range below (kept in sync with the bar at the top of this tab).
| Bump PR | Bump merged | Commit merged | Test / Build | Component | Issue | Class | AI Summary | From Comments | Fix |
|---|---|---|---|---|---|---|---|---|---|
| Loading triage data… | |||||||||
Historical bump PRs (merged + closed + open) for tracked submodules, most recent first. Excludes Test: / Do not merge: PRs. Click any PR to open it on GitHub.
Current submodule versions from .gitmodules
| Submodule | Path | Current SHA | Pending Bump PR | Last Bumped |
|---|---|---|---|---|
| Loading submodule status... | ||||
Issues from ROCm/TheRock repository
| Issue ⇅ | Title ⇅ | Status ⇅ | Labels | Assignee ⇅ | Author ⇅ | Created ⇅ | Updated ⇅ |
|---|---|---|---|---|---|---|---|
| Loading issues from GitHub... | |||||||
| Loading… |
Recent failures grouped by job name + workflow + arch. Larger clusters mean a stage is broken across many runs — start here when triaging.
| Failures | Job pattern | Workflow | Architectures | Repository | Last seen | Action |
|---|---|---|---|---|---|---|
| Loading… | ||||||
Jobs where a re-run produced a different result than the original attempt — the test flapped pass↔fail without code changes.
| Job | Architecture | Attempt sequence | Repository | Last seen | Action |
|---|---|---|---|---|---|
| Loading… | |||||
fail → pass usually means a transient issue (resource, network); pass → fail means a regression slipped past the first run.
Paste two commit SHAs (7+ hex chars). The lambda will look up their timestamps and pull every PR merged in that window across the selected repos. Tip: copy SHAs from the CI HUD tab. Share with #release-notes?from=SHA1&to=SHA2&repos=ROCm/TheRock.
bug → Fixes
feature → Features
breaking → Breaking
feat: → Features
fix: → Fixes
chore: → Internal
[CI] → CI/Build
[docs] → Documentation
[bump] → Dependencies
Add … → Features
Fix … → Fixes
Refactor … → Refactor
| Author |
|---|
| Loading… |
pull_request_review webhook event. Adding it requires (1) updating the GitHub webhook subscription, (2) extending the lambda + ClickHouse schema, (3) backfilling. Worth doing if review responsiveness becomes a focus area.
Issues on this tab are scoped to the bump-pr label; Bump PR analytics use TheRock's standard title-based bump detection (so older bot bumps without the label still appear).
Click a row to filter the issue list below.
| Component | Open | Closed | Open now | Distribution |
|---|---|---|---|---|
| Loading… | ||||
| Total | 0 | 0 | 0 | |
Label-driven buckets: Code Regression, Test Infra, Test Flaky, Infra Machine, Infra Timeout, CICD, Test Timeout, and Miscellaneous (no category label). Click a row to filter the issue list.
| Classification | Open | Closed | Open now | Distribution |
|---|---|---|---|---|
| Loading… | ||||
| Total | 0 | 0 | 0 | |
bump-pr-revert PRs per component (TheRock + submodule repos, main/develop only). Narrowed by the selected period.
Each panel shows total merges + average per week, a scatter chart, a weekly breakdown table, and the full per-period PR list.
.gitmodules submodules). Pick one to focus.
Every PR labeled bump-pr-revert across TheRock + submodule repos (main/develop only), narrowed by the selected period.
| PR | Repo | Title | Component | Author | State | Date |
|---|---|---|---|---|---|---|
| Loading… | ||||||
Each dot is one bump PR — x-axis is when it was merged, y-axis is how many hours it sat open before merging.
Vertical dashed lines mark OSSCI GPU runner events from frontend/data/gpu-events.json (currently empty —
populate that file to overlay events). Use this to spot whether bump-PR delays line up with GPU removals.
| Issue | Title | Status | Component | Class | Triage | Bump PR(s) | Author | Updated |
|---|---|---|---|---|---|---|---|---|
| Loading… | ||||||||
project: / component: / test: labels first, then a priority-ordered title/body/label regex (see _classifyIssueComponent in lambdas/query-proxy/handlers-misc.js). An issue that matches nothing is bucketed as Other.
Category is driven entirely by the issue's GitHub labels — no AI, no title/body regex. First match wins: code-regression → test-infra → test-flaky → infra-machine → infra-timeout. Issues carrying none of these fall back to miscellaneous (see ISSUE_CATEGORY_LABEL_MAP in lambdas/query-proxy/index.js).
Scope: Issues breakdowns are gated by the bump-pr label (nightly triage bugs have their own Nightly Issue Insights tab). Merged Bump PRs are resolved to one of TheRock's 10 .gitmodules submodules (component bumps roll up to their parent submodule; third-party / sysdep version bumps and automation PRs are excluded) and always source from ROCm/TheRock. Reverted Bumps counts every PR carrying the bump-pr-revert label across all tracked repos (these reverts usually live in rocm-systems / rocm-libraries, not TheRock); the Reverts by Component breakdown groups them by ROCm component. All are narrowed by the selected period.
Scoped to GitHub issues labelled ci-nightly (TheRock) or ci:nightly (rockrel), across the monitored repos plus ROCm/rockrel.
Resolved from the issue's project: / component: / test: labels, then a title/body matcher. Click a row to filter the issue list below.
| Component | Open | Closed | Open now | Distribution |
|---|---|---|---|---|
| Loading… | ||||
Label-driven buckets. Most nightly bugs carry no category label yet, so Miscellaneous dominates until triagers add one. Click a row to filter the issue list.
| Classification | Open | Closed | Open now | Distribution |
|---|---|---|---|---|
| Loading… | ||||
Read from the issue's gfx* label, else its title, else the body — rolled up to the canonical family
(gfx942 → gfx94X-dcgpu). An issue naming several arches counts in each, so the
total exceeds the issue count. Click a row to filter the issue list.
| GPU Arch | Open | Closed | Open now | Distribution |
|---|---|---|---|---|
| Loading… | ||||
From the platform: Linux / platform: Windows label, else the title/body. An issue
affecting both counts in each. Click a row to filter the issue list.
| Platform | Open | Closed | Open now | Distribution |
|---|---|---|---|---|
| Loading… | ||||
| Issue | Title | Status | Component | Class | GPU Arch | Platform | Author | Updated |
|---|---|---|---|---|---|---|---|---|
| Loading… | ||||||||
ci-nightly (TheRock) or ci:nightly (rockrel), in the monitored repos plus ROCm/rockrel. An issue is in the period if it was created or updated inside it.
Component and Category use the same classifiers as the Bump PR Insights tab (_classifyIssueComponent / _classifyIssueType in lambdas/query-proxy/handlers-misc.js). Category is label-only, so an issue with no category label is miscellaneous.
GPU Arch and Platform are multi-valued: an issue reading "Linux+Windows gfx900/gfx906" is counted once in each bucket, so those tables' totals are issue-instances and exceed the issue count. Both fall back to Unspecified when nothing can be resolved.
Open now ignores the selected period — it's the current all-time backlog for that bucket, so a long-open bug with no recent activity still shows up.