Skip to content

perf(actions): match runner tasks in SQL via normalized job-label table#38282

Open
bircni wants to merge 2 commits into
go-gitea:mainfrom
bircni:perf/actions-runner-poll-db
Open

perf(actions): match runner tasks in SQL via normalized job-label table#38282
bircni wants to merge 2 commits into
go-gitea:mainfrom
bircni:perf/actions-runner-poll-db

Conversation

@bircni

@bircni bircni commented Jun 30, 2026

Copy link
Copy Markdown
Member

Split out of #38150, which is being broken into smaller, independently reviewable PRs. This one contains the database part: the schema and query changes that make runner task assignment scale with the waiting backlog. It is independent of the no-DB split (#38281) and can land in either order.

Runner task assignment previously loaded every waiting job and filtered labels in Go. This replaces that with an indexed SQL query so a poll stays ~O(1 row) regardless of backlog size:

  • Normalized action_run_job_label table — one row per RunsOn label, kept in sync on job insert/delete, so label matching happens in SQL instead of in memory.
  • Composite (status, updated) index on action_run_job so "oldest waiting job" is an index seek instead of a sort of the whole waiting backlog.
  • Rewritten CreateTaskForRunner around the SQL match; unpreparable jobs are failed so they leave the queue, and run/attempt status is re-aggregated on claim so run-level concurrency still sees the occupying run.
  • Migration v343 creates the table, adds the index, and backfills labels for assignable (waiting/blocked) jobs.

Replace the in-memory label scan in runner task assignment with an
indexed SQL query so a runner poll stays O(1 row) regardless of the
waiting backlog:

- Add a normalized action_run_job_label table (one row per RunsOn label)
  kept in sync on job insert/delete, so labels match in SQL instead of
  loading and filtering every waiting job in Go.
- Add a composite (status, updated) index so "oldest waiting job" is an
  index seek instead of a sort of the whole waiting backlog.
- Rewrite CreateTaskForRunner around the SQL match, failing unpreparable
  jobs so they leave the queue, and re-aggregate run/attempt status on
  claim so run-level concurrency still sees the occupying run.
- Migration v343 creates the table, adds the index, and backfills labels
  for assignable (waiting/blocked) jobs.

Assisted-by: Claude:claude-opus-4-8
@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Jun 30, 2026
@bircni bircni requested a review from Zettat123 June 30, 2026 14:33
This migration targets the 1.28 release, so move AddActionRunJobMatchingSchema
(migration 343) from the v1_27 package into a new v1_28 package and mark the
1.27 version boundary. The migration ID is unchanged.

Assisted-by: Claude:claude-opus-4-8

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces database-backed runner label matching and queue-pick optimizations for Gitea Actions task assignment. It normalizes runs-on labels into a dedicated table and rewrites task picking to select the oldest matchable waiting job in SQL, avoiding O(backlog) in-memory filtering and preventing head-of-line stalls from unpreparable jobs.

Changes:

  • Add normalized action_run_job_label table plus migration/backfill (v343) and add a composite (status, updated) index to speed up “oldest waiting job” selection.
  • Route ActionRunJob creation through a single insert path that also persists label rows, and add label cleanup on run/repo deletion paths to avoid orphaned label rows.
  • Rewrite CreateTaskForRunner into a transactional SQL-based picker with bounded skip/fail handling for unpreparable jobs, plus new/updated tests covering matching and failure behavior.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
services/repository/delete.go Deletes job-label rows before repo-scoped job deletion to avoid orphan labels.
services/actions/run.go Inserts jobs via InsertActionRunJob to keep label projection in sync.
services/actions/reusable_workflow.go Uses InsertActionRunJob for reusable-workflow child job inserts.
services/actions/rerun.go Uses InsertActionRunJob when cloning jobs for reruns to keep labels synced.
services/actions/cleanup.go Deletes job-label rows before deleting run jobs during run cleanup.
models/migrations/v1_28/v343.go Migration creating label table, adding composite index, and backfilling labels for waiting/blocked jobs.
models/migrations/v1_28/v343_test.go Migration test validating backfill behavior (including dedup).
models/migrations/v1_28/main_test.go TestMain bootstrap for v1_28 migration tests.
models/migrations/migrations.go Registers migration 343.
models/fixtures/action_run_job_label.yml Fixture file for the new label table (empty baseline).
models/actions/task.go Removes the old in-memory scanning picker implementation (moved to task_pick.go).
models/actions/task_test.go Adds tests validating SQL-based picking semantics and unpreparable-job handling.
models/actions/task_pick.go New transactional SQL-based CreateTaskForRunner and helpers.
models/actions/run_job.go Adds composite (status, updated) index tags used by the picker.
models/actions/run_job_label.go New model + insert/delete helpers for normalized runs-on labels and SQL match condition builder.
models/actions/run_job_label_test.go Tests for label matching contract and SQL condition equivalence.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +221 to +223
if fallbackErr != nil {
return err
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants