Daily job-level failures across all monitored repos, stacked by category. Click a category in the legend to toggle it.
Top 15 (workflow, job pattern) pairs by median wall time across the selected window. Useful for identifying CI cost-optimization targets.
Jobs that have been failing for 3+ consecutive days with zero successful runs in the last 14 days. These are the silent CI-cost burners — fix them, exclude them from the matrix, or accept them as known-bad and add to a quarantine list.
multi_arch_release.yml run plus its dispatched downstreams
(test_artifacts, PyTorch wheels). Columns follow pipeline execution order across
12 lanes (ROCm Linux Builds → Linux Packaging → Publish → Tests → PyTorch Builds → PyTorch Tests → same for Windows).
Click ▸ on a column header to expand Math Libs / Test Artifacts / PyTorch Builds / PyTorch Tests into per-arch dots.
Click any row to open the full pipeline tree, or the ⬇ Downloads
button for install commands pinned to that exact build.
| Loading multi-arch release history… |
Lanes (12 total): ROCm Linux Builds · ROCm Linux Packaging · Linux Publish · Linux Tests · Linux PyTorch Builds · Linux PyTorch Tests · ROCm Windows Builds · ROCm Windows Packaging · Windows Publish · Windows Tests · Windows PyTorch Builds · Windows PyTorch Tests — each lane is shaded differently (cool blues/greens for Linux, warmer ambers/reds for Windows) and starts with a thicker primary-tinted border.
LINUX / WINDOWS tags above each column header indicate the OS.
Cell badges show passed / total jobs for that column on that ROCm version (multiple workflow runs for the same nightly are summed together).
Build jobs are routed by job-name path (Linux::release / Build Artifacts / {stage}); Test jobs by extracted Test {component} name; PyTorch Builds jobs by source workflow_id (multi_arch_release_*_pytorch_wheels.yml) with per-(py × torch) sub-cells; PyTorch Tests jobs (Test PyTorch | {arch}) split out into their own lane with per-(py × torch) cells and per-arch drill-down.
| Loading… |
|---|
| Loading CI HUD data... |
Pre-merge (presubmit) CI quality for open pull requests. Companion to CI HUD, which shows post-merge CI on the default branch.
| Loading… |
|---|
| Loading in-flight PR data... |
| Architecture | ||||||||
|---|---|---|---|---|---|---|---|---|
| Loading ASAN data... | ||||||||
| Architecture | ||||||||
|---|---|---|---|---|---|---|---|---|
| Loading TSAN data... | ||||||||
Open bump PRs awaiting merge — updating submodule versions in TheRock. Click ▸ on a tracked-submodule row to see what upstream PRs that bump pulls in.
| PR | Submodule | Upstream span | Title | Author | Status | CI Checks | Age | Updated |
|---|---|---|---|---|---|---|---|---|
| Loading bump PR data... | ||||||||
Each bump-pr issue mapped to the bump PR that surfaced it, with an AI-synthesized root cause, commenter quotes, and the applied fix. Rows that share a Bump PR group together so you can see all issues caused by one bump. Scoped by the Date range below (kept in sync with the bar at the top of this tab).
| Bump PR | Bump merged | Commit merged | Test / Build | Component | Issue | Class | AI Summary | From Comments | Fix |
|---|---|---|---|---|---|---|---|---|---|
| Loading triage data… | |||||||||
Historical bump PRs (merged + closed + open) for tracked submodules, most recent first. Excludes Test: / Do not merge: PRs. Click any PR to open it on GitHub.
Current submodule versions from .gitmodules
| Submodule | Path | Current SHA | Pending Bump PR | Last Bumped |
|---|---|---|---|---|
| Loading submodule status... | ||||
Issues from ROCm/TheRock repository
| Issue ⇅ | Title ⇅ | Status ⇅ | Labels | Assignee ⇅ | Author ⇅ | Created ⇅ | Updated ⇅ |
|---|---|---|---|---|---|---|---|
| Loading issues from GitHub... | |||||||
| Loading… |
Recent failures grouped by job name + workflow + arch. Larger clusters mean a stage is broken across many runs — start here when triaging.
| Failures | Job pattern | Workflow | Architectures | Repository | Last seen | Action |
|---|---|---|---|---|---|---|
| Loading… | ||||||
Jobs where a re-run produced a different result than the original attempt — the test flapped pass↔fail without code changes.
| Job | Architecture | Attempt sequence | Repository | Last seen | Action |
|---|---|---|---|---|---|
| Loading… | |||||
fail → pass usually means a transient issue (resource, network); pass → fail means a regression slipped past the first run.
Paste two commit SHAs (7+ hex chars). The lambda will look up their timestamps and pull every PR merged in that window across the selected repos. Tip: copy SHAs from the CI HUD tab. Share with #release-notes?from=SHA1&to=SHA2&repos=ROCm/TheRock.
bug → Fixes
feature → Features
breaking → Breaking
feat: → Features
fix: → Fixes
chore: → Internal
[CI] → CI/Build
[docs] → Documentation
[bump] → Dependencies
Add … → Features
Fix … → Fixes
Refactor … → Refactor
| Author |
|---|
| Loading… |
pull_request_review webhook event. Adding it requires (1) updating the GitHub webhook subscription, (2) extending the lambda + ClickHouse schema, (3) backfilling. Worth doing if review responsiveness becomes a focus area.
Issues on this tab are scoped to the bump-pr label; Bump PR analytics use TheRock's standard title-based bump detection (so older bot bumps without the label still appear).
Click a row to filter the issue list below.
| Component | Open | Closed | Total | Distribution |
|---|---|---|---|---|
| Loading… | ||||
Infra = CI / build / release-engineering plumbing. Code = product code bugs & feature requests.
| Classification | Open | Closed | Total | Distribution |
|---|---|---|---|---|
| Loading… | ||||
Bucketed by days since opened. Click a row to filter the issue list below to just that age bucket.
| Age | Open | Closed | Total | Distribution |
|---|---|---|---|---|
| Loading… | ||||
Each panel shows total merges + average per week, a scatter chart, a weekly breakdown table, and the full per-period PR list.
.gitmodules submodules). Pick one to focus.
Each dot is one bump PR — x-axis is when it was merged, y-axis is how many hours it sat open before merging.
Vertical dashed lines mark OSSCI GPU runner events from frontend/data/gpu-events.json (currently empty —
populate that file to overlay events). Use this to spot whether bump-PR delays line up with GPU removals.
| Issue | Title | Status | Component | Class | Triage | Bump PR(s) | Author | Updated |
|---|---|---|---|---|---|---|---|---|
| Loading… | ||||||||
ISSUE_COMPONENT_RULES in lambdas/query-proxy/index.js). An issue that matches no rule is bucketed as Other.
Classification uses the same approach (ISSUE_INFRA_RULES) plus an allow-list of label names (infra, ci, build, release, tooling, etc.). Issues that don't look like infra default to code.
Scope: Issues breakdowns include only items carrying the bump-pr GitHub label. Merged Bump PRs are resolved to one of TheRock's 10 .gitmodules submodules (component bumps roll up to their parent submodule; third-party / sysdep version bumps and automation PRs are excluded) and always source from ROCm/TheRock. Both are narrowed by the selected period.